Loud Echoing, on client end

I am running into a problem where there is a loud echoing on the client end of phone calls place from asterisk, this same problem also occurs when calling into the system.

The echoing is not heard by the employee (using the IP Phone) but rather from the person who called into asterisk or who was called by asterisk. The noise is really more than an echo, it is a very loud noise which I would almost describe as the noise a helicopter makes when hovering over you. It has been very difficult to trouble shoot as it only happens about a half of the time when you call into the system.

I am using a Digium 2400E card in the server and have a Dell 1.6 Quad Core Processor with 1 gig of ram. (I have tried replacing the Digium card)
The system is currently hosting about 5 external phone lines and about 6 IP phones. (Linksys SPA 942)

Let me know if you need any more information.