I have seen this problem before, and the only thing we put it down to was a CoDec mis-match, yeah i know what your thinking, but hey it is worth the shot having you look at it.
Check in your configuration files SIP or H.323 or IAX what ever your using and look at the allowed CoDec’s, then see which ones the Polycom one support, if it is over a local Network then i suggest you lock the Polycom phone in on the uLAW or aLAW CoDec and then set your Configuration file to only except this CoDec when making or receiving calls.
keep this in mind, receiving calls from the PSTN side will be either in aLAW or ulAW format, when it comes in Asterisk will have to Trans code it back to the CoDec your phone is using, this can cause one-way-audio and i have seen it happen a lot, restricting it to either of the LAW formats generally removes this problem, i would only recommend you use other CoDec is your phone does not support them the LAW form CoDec’s and some other lower rate CoDec.
However if you are Using these CoDec’s already, then i am not entirely sure what could be causing this problem, more information might be needed. Oh i should mention that Asterisk will not throw up an error on the CLI if there is a CoDec mis-match well not all the time.
What is a good test is too run the command 'sip show peers/friends" on the CLI and have a look at see what ports the Phone is using, what CoDec it is using versus the CoDec the TDM line is using, and then work it from there, sometime it is just the small things that can cause the major headaches