IAX2 Trunk call quality

I’m having the following issue on an IAX trunk b/w 2 * boxes.

Box A is running Asterisk 1.2
Box B is running Asterisk 1.4 with the Digium web GUI

I have an IAX trunk b/w the two using trunking and g729 codec.
Box B is connected to the internet with a Fractional T1 (14 channels).

My endpoints are Polycom 501 and 330.

Whenever I see a spike in incoming bandwidth on the T1 my call quality drops on Box B on the incoming audio only. The party on Box A can still hear me fine. It starts skipping and I get about every other word. I started recording calls to debug. At first I thought this was a bandwidth issue but when I playback one of the recorded calls that sounded horrible the recording is crystal clear.

Any with any ideas on what’s going on? I’m not seeing anything out of the ordinary at the CLI

Any suggestions would be appreciated.