Strange Polycom Jitter behavior

I am getting bad jitter with my polycom 501 and 601 phones. The thing that makes it strange is that the jitter only happens on incoming calls while using the handset or headset. If the caller is put on speaker phone the jitter goes away. Only my users experience the jitter, not the person on the other end. Any guidance with this would be greatly appreciated.

Software/hardware info,

Asterisk 1.4.19.2
FreePBX 2.4
Polycom 501, 601
Polycom Bootrom 4.0.0.0423
Polycom SIP 3.0.1.032