Hey Everyone, reaching out to get some direction as its a little past my knowledge in asterisk now.
I really want to improve the high availability of our platform and have not been able to get the right advice.
Our current setup includes
2x asterisk front end servers
2x mysql database servers that do master-master replication.
asterisk 1 is connected to mysql 1
asterisk 2 is connected to mysql2
A vendor who recommended this setup says it is good and reliable, but im skeptical and think we should be able ot improve on this.
This provides some redundancy, but its certainly not high availability.
peer connections are done via dns-srv so connections are split across the front end servers.
Now my major issue is, if we have an issue with a database server (i.e. just recently had one fill up its disk unexpectedly) and asterisk cannot connect to the DB, it renders the extensions hitting that server failing to work. Cause asterisk is still running and the port is opening its trying to connect.
The causes issues and frustrations.
I am trying to work out how we can set things up to prevent an issue like this from happening. Is it possible to have the asterisk server have a primary and secondary MYSQL to connect to so if the first one fails to connect it can connect to the second?
Or any advice you have on how to make this more effective?
Wouldn’t that be the task of the database? I think MySQL has InnoDB clusters based on the Paxos consensus algorithm, but there are other choices. I am not using MySQL myself, but what I am using is basically dealing with a single DB from the perspective of Asterisk. Otherwise it would be a mess.
Well its mysql cluster of 2 servers, master-master replication.
But the limiting factor is that asterisk servers are connecting to the individual database servers. So if one is down, it does not look at the other, which effectively takes that asterisk server offline. But I want it to talk to the second one as failover so it doesnt have that problem.
You could read the mongodb docs about replication. The downside is, there’s no official mongodb interface.
You could still monitor the state of the dbs from outside asterisk and reload/restart asterisk if necessary with a working connection. I think you still have to worry about the state of the current sessions, but that depends on what you can tolerate.
What does “best practice” mean anyway? I’d rather have replica sets, separate APIs that are also responsible for the more complicated stuff, and multiple sources and sinks. I’d rather maintain half a dozen identical database servers with Ansible than have one or another database on each server. It doesn’t matter with one or two boxes, but it does with about 100.
Wouldnt best practice more be having single application servers instead of all in one?
Im wondering if doing something like haproxy would be the best option. That way the asterisk servers look at a single IP, and haproxy deals with loadbalance and failover etc?
Anyone have experience using haproxy with asterisk?
I have no experience with haproxy, but I think it could be used for the front ends (asterisk, other asterisk apps). You’d still need to have a common (realtime) database, or you might lose the state of registrations, etc when haproxy switches. And here the suggestion was to use a db that comes with replication support.
The reason for replicating onto the Asterisk machine is that, otherwise Asterisk becomes unusable if the database machine is down, and sluggish if there are problems accessing it.
Next month we are going to test a setup with haproxy on the mysql side so the asterisk servers connect to a single IP an haproxy manages the load balance and failover of mysql and see how that goes
Well I always try to think in “points of failure”
if you have that customer can send calls to 2 asterisk servers, then you have a 0.5 PoF 1/2 for asterisk
when you add Realtime you add an extra PoF factor, if each asterisk is only connecting to one database server, then for the database you have a value of 1 PoF, and the hole platform 0.5 + 1 = 1.5 PoF
to lower the PoF of the database you either have to configure asterisk to connect to multiple database (ODBC), use haproxy or something similar
but remember that using haproxy is also a PoF and you have add in the chance that a system will fail, as haproxy is not without its problem (false positive resulting in a failover)
so when you start to have both haproxy, sip proxy, asterisk server, database server, SAN, Virtual Machine’s, multiple hosting center, A/B power, multiple BGP connections, parallel switches
the complexity start to build up
lets just say I have experienced more downtime to power generator test that actual power failure :-/
and we are currently working on how to remove realtime as scaling is becoming a problem
I tried setting the second asterisk connection to the secondary DB server. But when 1 server goes down it all stops working, so it must not be right. I am guessing there is a different way to configure odbc failover.
odbc.ini has
asterisk1
asterisk2
Which I changed for one to point to mysql1 and 1 to point to mysql2 but this doesnt seem to be the right approach.
This ain’t failover. Failover is something the database product has and is taken care of by the ODBC driver of not. I think there are some Oracle products that have that feature.
Failover means that one or more servers can fail, and the failed servers will automatically synchronize again later. In addition, there are questions about what is or is not a successful write (or read) attempt. Sometimes you want the data to be written on the majority of the servers to be considered successful.
I myself like to use MongoDB when easy replication matters and the appropriate jargon uses terms like “read/write concerns” for this.
Configured for main DB server as priority and secondary only as a backup if the primary does not connect.
Configured asterisk to point to 127.0.0.1:3306 (haproxy).
Works a charm. If DB server goes down asterisk isnt skipping a beat. (Obviously still potential for issues like replication etc depending on the problem) but has solved the major hole of asterisk not working on 1 frontend if it cant connect to its DB, so a great step forward.
