Unable to hear person called but they hear us

This is a strange problem, I have a server that is not natted. It is a stand alone server, configured using freepbx 2.4.
I have a number of providers I use, some use iax and some use sip and it occurs on all of them.
When a call is placed to me or I place a call during a conversation I will find that I am not able to hear the user on the other end but they can hear me and this lasts for about 10 seconds.
All of my phones are configure to use ulaw, and nothing else, and canreinvite=no is set on all trunks and extensions.
This problem seemed to have started after I had upgraded to what came after 1.4.18. The bad thing is I can’t go back to this version because of dependencies. I use Debian packages in unstable.
I have been using this configuration for over a year now using the same P4 3GHZ machine. It has 1Gig of memory in the system and the load stays at about .07 to .011. This machine only gets about 10 calls an hour through it so it is not a overly utilized machine.

I will say that since I upgraded to 1.4.21.2 it has gotten better the length of time that this blackout occurs is much shorter now, instead of lasting for 20 or more seconds its down to 3 to 10 seconds.

I am really not sure where to look for a problem like this, being that it occurs sporadically. I does not matter what provider, even on enum calls, like free800 it occurs.
I have a very large sip.conf and friends, so if these things are needed please let me know and I can post them onto my website to be downloaded.

I would appreciate any directions as to where to look for this problem.

And this is not a natted server, remember I am directly connected and all of my sip phones or sip software have this problem. I need to do some better testing but it seems not to happen when a call is made internal.

All phones are internal private ip addresses ranges, except for the asterisk server itself. So this seems to direct me to look more in the area of the bridging code. I understand that Debian has used backported the 1.6 bridging code in there release. Whether or not these calls are utilizing that code or not I would not be able to say.

Again if there are any configs needed please let me know which ones and I will be happy to provide them.

Thanks

Are you saying that you have one-way audio for about 10 seconds and then it fixes itself?

Yes, this is true.

Also my bandwidth is not a problem that I can see. I monitor it from within and outside of my network. I check google.com and my network providers router with cacti. I never see long ping delays or over utilization. I had thought at first this could have been the problem.
I have a 15mbs down and 2mbs up service and this gets me about 14.5mbs up and 1.7mbs down. I have configured bandwidth management that has worked for a long time without giving me any problems.

Has anyone got an idea as to this problem?