I am desiging an asterisk system to be used in an outbound call center. Initially there will be 80-90 agents, with plans to grow to 120 agents.
All of our agents are in the same building, and the plan is to use a softphone (SJPhone?).
Is it realistic to expect a single server, Dell PowerEdge 1800 Dual Xeon (dual-core) 3.0GHz and 4GB ram with the TE411P to handle up to 96 users? (24*4 T1’s). The TE411P is new from Digium, just arrived Friday, so it should have the latest firmware.
The OS is CentOS 4.2.
I am also wondering if I should use the x86_64 or i686 builds…I have not found a suitable way to stress test this.
Any suggestions as far as codecs, etc would be helpful. What scares me is that we plan on 90-95% utilization of our T1 lines.
I appreciate any and all help.
Jeremy Crawford
I think the biggest concern would be the IRQ interrupts eating CPU time, but that may not even be a concern.
I’d go with an uncompressed codec. Ulaw or similar to keep the cpu utilization down. I don’t think you have much to worry about as far as bandwidth is concerned, even with 200+ clients. On your average 10/100 network anyway.
Personally, for this application I’d a dual setup with the second box acting as a failover using some tricks from the linux HA project.
note: I’ve never done something that large, so take my words for what they are worth. I’d love to hear input from people who have done this as well.
I’ll save you some time. You won’t be able to have that many outbound agents very reliably with a single server. Both the GnuDialer and VICIDIAL projects(the two GPL Asterisk-based outbound dialers) are designed to work with multiple servers for large call center setups. The most we’ve ever successfully had on a single server was 60 agents and that was at a low dial rate with no recording/transcoding/monitoring.
Buy two or three cheaper servers and split your lines across them and you should be fine. Our 120 seat VICIDIAL inbound/outbound installation has a total of 7 Asterisk servers and 16 T1s. The calls are load-balanced across all of the servers and it all works like it’s a single dialer. Our servers are just P4 3.2GHz processors with 2GB RAM.
Thanks for the info. Is there a accurate way to test this stress point before I have the agents show up at the desk and start making the calls? I would like to know that even with 2 servers that it can handle the load. I will have a few days with the t1’s active before the agents show up for work, and it would be nice to show that all is well prior to then.
It is very hard to test real capacity, You cannot accurately measure things like audio quality on each agent phone or delay in calls going to agents/being hung up without having actual agents sitting there giving you feedback.
What call center software are you using?
Most of the software we are using for scripting, dialog, etc is custom, but we are installing astguiclient for recording, etc.
Since you will be installing astGUIclient you will have the ability to use it’s system performance logging that tracks things like CPU usage, load average, RAM usage and live Asterisk channels for each Asterisk server you have it installed on. All of the system performance data is stored in a database and can give you a very good idea of what your system load is proportional to how many Asterisk channels are live. It even can generate graphs of system performance stats(you do have to activate the performance logging in the AST_update.pl script)
What will your highest dial ratio be for your outbound dialing(number of lines per agent)?
Our current call center is all analog/pots, so the agent to line ratio will be 1:1, which should make this task much simpler.
One thing that has me confused is Asterisk Business Edition…it claims up to 240 concurrent calls on a single server. Out of the box it is 120 calls per concurrent server. Since all of my traffic is going to be sip to zap, and the zap channels are all the TE411P (hardware echo cancelation), I would think I have the “ideal” configuration.
I am going to be using ULAW for the client side, since we are on a switched 10/100/1000 network, so again, I think this would be ideal for lower cpu usage.
Does the business edition have anything different than the main stream? I prefer to stay on the opensource,but if there is a difference, I would use the business edition.
The Asterisk Business Edition (BIZ-Asterisk) is supposedly just code that has been more thoroughly tested than the Open Source version(OSS-Asterisk) so that it can be guaranteed. As such it does not necessarily have all of the latest features or patches applied to it. BIZ-Asterisk is not supposed to be optimized in any way over OSS-Asterisk.
As for your specific setup How will your calls be going to agents? Will you be using Asterisk Agents/Queues or meetme rooms or direct dial where the agents receives a new call every time and must hangup in between?
BIZ-Asterisk is capacity tested to 240 concurrent calls with regular calls, not agent/queues calls and not meetme calls. You must also remember that there is no recording or monitoring going on in these capacty tests, all of those things reduce your capacity and use up CPU time.
[quote=“mflorell”]As for your specific setup How will your calls be going to agents? Will you be using Asterisk Agents/Queues or meetme rooms or direct dial where the agents receives a new call every time and must hangup in between?
One more question, since you are using POTS/analog for agents and T1s for your trunks, where does the SIP fit in?
[/quote]
Our current call center software, developed in house, is just a web interface with a survey. The agent currently is picking up the pots phone, dialing the number, and peforming the survey.
Our “new” call center (using asterisk) will not have any pots at all. All new T1 lines are being installed, and going direct to the asterisk box. The agents will be using softphones. Other than that, our day 1 experience will be to use the same method of dialing…ie: dial manually, listen to the ring, etc.
Our future plan is to integrate our app with astguiclient/vicdial. Instead of a meetme plan, we would have it first ring the agent, once the agent connects, it would then dial the contact.
What I was trying to say about our agent to phone ratio being 1:1, is that there will be no warm-transfers, or multiple “external” lines per agent. Each agent will have a single external call. Asterisk will see this as a SIP->ZAP.
There will be a few users that will be doing ZAP-BARGE, etc, but this is expected to be limited to a max of 4.
If you are doing just regular calls you may be able to do up to 92 agents on a single server(you will only have 92 voice channels on your 4 T1s anyway).
If you want to grow past that you will run into problems. You would need to add another T1 card to the server which in my experience will give you problems.
The real issue with Asterisk and load is not necessarily the load average, but the load spikes that happen several times a day on a loaded system. On every system we’ve run(even without VICIDIAL) if the system is loaded beyond a certian point Asterisk will cause load spikes several times a day and those times are when everything bad happens(choppy audio on VOIP channels, delays in call transfers or dialing, interrupted recordings)
I would strongly recommend building your application to work on multiple servers. That will guarantee that you can grow without having to rewrite your applications and will allow for less than 100% downtime if one of your servers crashes.
As for VICIDIAL, it is possible to use VICIDIAL as a one-dial-at-a-time method with agents hearing ringing, etc. It’s just manual dial mode and it works very well. It also has the capability to launch a custom web page with customer details attached (that could be your survey web page) upon connection of the call. It does use meetme, and because of that, more server CPU resources, but that allows the agent to never have to hangup their phone in between calls as well as having integrated recording statistics, monitoring, etc…
In manual dial mode you could be able to handle 70 or so VICIDIAL agents on the server you specified earlier. You should consider using VICIDIAL instead of spending the time to develop an entire interface and backend yourself.
Thanks for all the info…that helps a lot.
If I were to move the mysql, apache, and php stuff to another box, and connect it to the asterisk via jumbo framed gigabit, would provide a sufficent amount of relief to be worth the effort?
I had planned on this design in order to move the voicemail and recorded calls to another box, so that the playback via web would not cause any additional load. This second box would make sense for the additon of a 2 port T1 interface card to handle the additional users, rather than loading the first server with 2 t1 cards.
Does this sound reasonable?
You’re welcome, glad I could help.
I was assuming already that you were not going to have those(Php,Apache,MySQL) on the Asterisk server. The have to be on a separate server for an installation of your size.
Gigabit is a waste for Asterisk installations with less than 300 phones, you’ll never get anywhere close to saturating it. At one point we had 120 SIP phones connecting to Asterisk servers and all the traffic was going through one 100Mb FD ethernet port on an old HP 2424M Procurve switch. We never even got up to 50% utilization on that port.
DB/Web doesn’t need it either, you’re not moving anything that big, ULAW VOIP only takes at most 82kbs with headers resulting in a theoretical 1200+ conversations over 100Mbs-capable connection.
I would setup two identical Asterisk servers, both with quad T1 cards and split the load between them, then setup a separate Web/DB server and a separate Web/archive server for your voice recordings/voicemail/backups and web/DB failover.
That is 4 machines and would offer a measure of coverage if any one of them failed, they don’t need to be Xeon-class servers either.
Let me know how your project progresses.
Great…Thanks for all your help. I will keep you posted.