I am not familar with the ploycom; however, in the past when I have found everything working properly except for some audio not working, or one side of the audio not working it is usually related to the audio codec not being supported by one end.
G729 requires some special installation and licensing for asterisk to be able to encode/transcode with it…
If the polycom phone support just ulaw, try to disable g729 and see what happens.