I experience a similar issue although in a different environment.
I have 160 voip based asterisk users (Koncept1020a and grandstream gpx2000) phones, a high speed lan (1gb) in well configured network and a fast linux box (asterisk uses less than 10% cpu on average) running fedora.
We use the TE110P pri interface cards on the telco interface and are configured as euro-isdn.
I use a slightly modified version of * 1.2 - no changes to any modules that handle the audio!
Changes include a voicemail type conf file for fax-to-email and modified app_rxfax to use it. A modified park call thread, to use retrydial instead of dial when returning to the original extension in park calls. Enhanced the manager channel status command to return a list of channels status by prefix for our own written soft operator switchboard.
Outbound audio sometimes deteriorates badly. Remote telco users can be heard clearly with no audio break-up by the asterisk voip user, but at the same time the remote part hears audio that is very choppy or breaks up. This does not happen on every call!
I have tried various codecs from g711 to gsm and all exhibit the problem.
I have ruled out our network as we run 600 call centre agents through our own home-grown intel dialogic based voip call centre system with excellent quality. The call centre agents soft-phone uses g723 codec.
Also the fact that I can hear the other party clearly at all times rules out bandwidth problem.
I have followed wiki and other sites audio problem resolution steps, including adjusting the rx and tx gains, I also as a last resort tried the different echo cancellations provided in * but I have not been able to resolve.
I suspect some incompatibility in the rtp frame sizes between the digium card and phones but I dont know how to verify this.