Endpoint intermittent delay

I have a setup in a large house with nearly 60 extensions all together, and most of the time everything works well. However, I keep getting a report of a delay at one specific endpoint, and of course, it is the phone of the owner of the house. I have tried everything I can think of to resolve this delay, and now it is down to and ‘intermittent’ issue.

Running : Asterisk 11.6-cert1

my sip conf:
[general]
context=pri_from_als
allowoverlap=yes
externip=XX.XX.XX.XX
bindport=5060
nat=force_rport,comedia
srvlookup=no
videosupport=no
disallow=all
allow=ulaw ; force ulaw, to avoid any quality issues
musicclass=sip
language=en
sendrpid=yes
dtmfmode=rfc2833
rfc2833compensate=yes
canreinvite=no ; changed 10/4
directrtpsetup=no ; chanhed 10/4
t38pt_udptl = no ; Asterisk T.38 not officially supported
t38pt_rtp=no
t38pt_tcp=no
mwi_from=asterisk

[first_gateway]
type=peer
host=XX.XX.XX.XX ; First AT&T IP Border Element (IPBE) Address
qualify=2000 ; SIP Ping Frequency in msec.
context=pri_from_als

[second_gateway]
type=peer
host=XX.XX.XX.XX ; Second AT&T IP Border Element (IPBE) Address
qualify=2000 ;
;context=pri_from_als

;## Template for the phones -
house-phone
deny=0.0.0.0/0.0.0.0
permit=10.XX.XX.XX/255.255.255.0
secret=MY_SECRET
dtmfmode=rfc2833
relaxdtmf=yes
canreinvite=no
directmedia=outgoing ; added 10/6
context=internal
disallow=all
allow=ulaw ; force ulaw, to avoid any quality issues
host=dynamic
trustrpid=yes
sendrpid=no
type=friend
qualify=3000
qualifyfreq=60
transport=udp
encryption=no
callcounter=yes
faxdetect=no

0897C-2001
08282-2002

I am not familiar with the network side of things but I do know that the IP on the server is a NATed Ip on both the LAN side and the WAN side. Not sure why this is, and I am thinking this maybe part of the problem.

Is there anything anyone might suggest that I add to/take away from my configuration that may clear things up?

Thanks!

Slight change from above:
Actually, I have been told the internal LAN IP’s are routed, not NAT’ed. I don’t know if this makes any difference.