Long INVITE response time

I am running 1.4.16.2 and for whatever reason my box is getting the invite from the server, but it takes so long for it to respond to that invite that the server has already sent out another, then failed over to a second server(with a different ip address), and sent another invite, and the first server no longer wants to deal with the call and the call is dropped.

It only happens on some calls, not all of them. the logs report no errors other than:

[Apr 18 17:28:17] WARNING[3937]: res_config_pgsql.c:207 realtime_pgsql: Postgresql RealTime: Could not find any rows in table sip_users. [Apr 18 17:28:17] WARNING[3937]: res_config_pgsql.c:207 realtime_pgsql: Postgresql RealTime: Could not find any rows in table sip_users. [Apr 18 17:28:17] WARNING[3937]: res_config_pgsql.c:207 realtime_pgsql: Postgresql RealTime: Could not find any rows in table sip_users. [Apr 18 17:28:18] WARNING[3937]: res_config_pgsql.c:370 realtime_multi_pgsql: Postgresql RealTime: Could not find any rows in table sip_users. [Apr 18 17:28:18] WARNING[3937]: res_config_pgsql.c:370 realtime_multi_pgsql: Postgresql RealTime: Could not find any rows in table sip_users.

But I do not think that this is the cause? Maybe it is any ideas?