Asterisk Choppy sound

Hi, I am running Asterisk 1.8.3.1 with all the unnecessary modules striped out. I got only app_dial, chan_local, chan_sip and the basic required dialplan and resources. I am controlling my dialplan with AGI, A2Billing to be exact. Using no transcoding, strictly G723 and G729 (Testing only). My * configuration is also minimal and as per the requirement. The client who is sending SIP calls to this server is on Public network and have both fast connections.
Until recently, I am seeing calls being dropped after 10-15 seconds. I tried to enable recording, and figure out the voice i too much choppy. I have tried both with and without directmedia, directrtpsetup, canreinvite to evade media handling by * and to bypass but can’t figure out where the problem is.

Would like to hear some anyone how to find where the problem is and why the sound quality is too bad. The sever is i7 CPU with 12 GB RAM. No debug or verbose enabled for max. performance. Call quality starts dropping after 100+ calls.

Any help would be appreciated.

So, you are saying the first 100 calls are OK.
If so, it sounds like a CPU problem (of your server). Try a less “expensive” codec like G.711. Does it help ?

Is the internet connection of your server good enough to handle 100+ calls at G711?