Realtime odbc and haproxy issues

I’m testing my setup with haproxy and MySQL and getting some weird behavior.

If I block the connection between my asterisk server and one of the database servers with iptables everything works as expected. Haproxy marks that server as down, fails over to the second server, everything works.

If I shut down MySQL on the primary server haproxy again marks that server down and fails over to the second one. Asterisk however doesn’t handle this well. It sees it can’t connect to the database and attempts a reconnect, which works. Within a few seconds though sip peers start showing unreachable and disappearing. I have trunks to my providers setup in sip.conf, these all show unreachable. While all this is happening I can connect to mysql from CLI so I know haproxy is working fine.

Once this starts happening the only way I can get things working again is to restart asterisk. I’ve tried sip reload and reloading the odbc module but that has no effect. I haven’t seen anything that looks important with core debug on, it shows the mysql drop and reconnect and all the peers registration attempts.

I’m running asterisk 11.12, haproxy 1.5.5, and mysql connector odbc-5.2.7.

Has anybody seen this behavior before?

Think I’ve got this one figured out. The second server was far away, ping times were 40ms. If I only had a few endpoints registered it handled things fine but with about 100 endpoints registered it would have problems.

I used a server nearby with 1ms pings and it handles things fine.