AMI drop connection at high load


#1

Hey guys. I’ve encountered the following problem while implementing my custom monitoring solution. While listening to Asterisk events through AMI at high load my client application (written using twisted-starpy) is getting disconnected from AMI. While observing tcpdump logs i’ve discovered that the server rejects my connection at this moment: IP 127.0.0.1.5038 > 127.0.0.1.58231: Flags [R]

Does anyone have any idea on how to solve this problem?
Thanks in advance. Any help would be greatly appreciated.


#2

What is in the Asterisk log?

Can you confirm that you try to keep the connection up long term?


#3

There are no any related entries in the Asterisk log (even with debug/verbose 999 option set). The problem doesn’t seem to be related with keepalive, actually connection is being dropped when the data exchange intensity raises up by a large margin (in terms of mbps/pps) so probably it is a buffer/socket issue.


#4

Hi Just_kgn, and anyone else with this problem. I just found this thread, and a workaround for the problem.

I have an asterisk server on my local network, wrote and tested some AMI code, which works fine here, but when running the same connection across a lower speed/higher latency WAN link, the connection drops repeatedly drops as you describe.
My workaround was to set up a connection proxy using the xinetd superserver ( helped by heronforge.net/redhat/node11.html ) (you could use inetd and nc if you had to, my testing with nc worked OK). The supseserver listens (on a different port), and passes traffic through to asterisk. the superserver i can only assume handles buffering or dropped packets more gracefully.

The procedure:

on the asterisk machine, add the following line to /etc/services

create /etc/xinetd.d/asteriskamifwd containing:

service asteriskamifwd { flags = REUSE socket_type = stream wait = no user = root redirect = localhost 5038 log_on_failure += USERID only_from = 127.0.0.1/24 10.0.0.1/24 }
set only_from to appropriate addresses (see the xinetd man pages) or scrub the line if you don’t care about restricting client addresses.

restart the superserver:

now connect to asterisk on port 5039 rather than 5038, and all should be better behaved.
You could instead create a similar config on a machine ‘close to’ the asterisk server (where the connection dropping isn’t a problem), and then connect via that machine.

seems to be working OK for me so far.

[color=#FF0000]WARNING:[/color] This bypasses some of the AMI security. Connections coming via the superserver would appear to asterisk as such. security settings in the ami config must allow connections from the machine running superserver (127.0.0.1 in the above example) method will allow all connections to the superserver through to asterisk, regardless of the remote IP address. Be sure to configure only_from correctly.

.

Anyone else have workaround, success/problems with this method, or better solutions?


#5

you could do the same thing the superserver that TrevorP suggested with a local twisted application. This would also allow you to handle security a little better as well.

You could even go as far as doing something like having the twisted/starpy interface local, have it timestamp each event, then stick that event in a queue. Using XMLRPC, you could have that system then process the queue, making XMLRPC calls to your application (or as many subscribers to the data) with the data from the queue. Because each event is timestamped as it’s received from asterisk, you can survive poor connectivity/low bandwidth/lag issues cleanly.

-Chris