Dramatic performance loss after upgrading 1.8 -> 13.13

alex.svetkin · February 13, 2017, 5:55pm

Hello everyone!

We are operating legacy Asterisk version 1.8 and decided to move to version 13. But after the upgrade we noticed a very significant performance lost, something like 3-4 time more CPU consumption and as a result calls are stuttering etc.
About the system:

bare metal, Intel XEON E5-2620 12 cores, 32G RAM.
100-200 calls made per second
we use chan_sip
we don’t use cdr, cel, conferencing, voicemail, websockets, lua scripting, call parking, IVR
we do use directmedia=no, queues, mixmonitor, ODBC, UserEvents, AGI intensively
There is no signs of high disk/network load, just CPU and “linux load average”.
I tried to carefully edit configs when moving to 13, changing as little as possible just to make it work on 13. So it’s IMHO not likely that I’ve turned some heavy new feature on…

Logs after the startup look fine for me:

[2017-02-10 18:07:17] Asterisk 13.13.1 built by root on a x86_64 running Linux on 2017-02-09 15:01:55 UTC
[2017-02-10 18:07:17] NOTICE[6671] cdr.c: CDR logging disabled.
[2017-02-10 18:07:21] NOTICE[6671] loader.c: 98 modules will be loaded.
[2017-02-10 18:07:21] NOTICE[6671] res_odbc.c: Adding ENV var: ODBCSYSINI=/etc
[2017-02-10 18:07:21] NOTICE[6671] res_odbc.c: Adding ENV var: ODBCINI=/etc/odbc.ini
[2017-02-10 18:07:21] NOTICE[6671] res_odbc.c: Registered ODBC class 'mysql-transition-info' dsn->[MySQL-asterisk]
[2017-02-10 18:07:21] NOTICE[6671] res_odbc.c: res_odbc loaded.
[2017-02-10 18:07:21] ERROR[6671] pbx_lua.c: Error loading extensions.lua: cannot open '/etc/asterisk/extensions.lua' for reading: No such file or directory
[2017-02-10 18:07:21] WARNING[6671] res_config_mysql.c: MySQL realtime: no requirements setting found, using 'warn' as default.
[2017-02-10 18:07:21] WARNING[6671] res_musiconhold.c: Cannot open dir /var/lib/asterisk/moh/autodial2/ or dir does not exist
[2017-02-10 18:07:21] WARNING[6671] res_musiconhold.c: Cannot open dir /var/lib/asterisk/moh/.nomusic_reserved or dir does not exist
[2017-02-10 18:07:21] VERBOSE[6671] chan_sip.c: SIP channel loading...
[2017-02-10 18:07:21] WARNING[6671] chan_sip.c: !!! PLEASE NOTE: Setting 'nat' for a peer/user that differs from the  global setting can make
[2017-02-10 18:07:21] WARNING[6671] chan_sip.c: !!! the name of that peer/user discoverable by an attacker. Replies for non-existent peers/users
[2017-02-10 18:07:21] WARNING[6671] chan_sip.c: !!! will be sent to a different port than replies for an existing peer/user. If at all possible,
[2017-02-10 18:07:21] WARNING[6671] chan_sip.c: !!! use the global 'nat' setting and do not set 'nat' per peer/user.
[2017-02-10 18:07:21] WARNING[6671] chan_sip.c: !!! (config category='MTT-out-2' global force_rport='No' peer/user force_rport='Yes')
[2017-02-10 18:07:21] WARNING[6671] chan_sip.c: !!! PLEASE NOTE: Setting 'nat' for a peer/user that differs from the  global setting can make
[2017-02-10 18:07:21] WARNING[6671] chan_sip.c: !!! the name of that peer/user discoverable by an attacker. Replies for non-existent peers/users
[2017-02-10 18:07:21] WARNING[6671] chan_sip.c: !!! will be sent to a different port than replies for an existing peer/user. If at all possible,
[2017-02-10 18:07:21] WARNING[6671] chan_sip.c: !!! use the global 'nat' setting and do not set 'nat' per peer/user.
[2017-02-10 18:07:21] WARNING[6671] chan_sip.c: !!! (config category='rt' global force_rport='No' peer/user force_rport='Yes')
[2017-02-10 18:07:21] NOTICE[6671] iax2/provision.c: No IAX provisioning configuration found, IAX provisioning disabled.
[2017-02-10 18:07:21] WARNING[6671] res_snmp.c: Could not load res_snmp.conf
[2017-02-10 18:07:21] NOTICE[6671] app_queue.c: Queue members successfully reloaded from database.
[2017-02-10 18:07:21] VERBOSE[6671] asterisk.c: Asterisk Ready.

But there are some suspicious entries. I’ve disabled CDR completely and don’t get/set CDR() vars, but sometimes get in the logs:

[2017-02-13 06:44:13] ERROR[6689] cdr.c: Unable to find CDR for channel Local/dial@dialer-000d4fcb;1
....
[2017-02-13 06:44:05] ERROR[6689] cdr.c: Unable to find CDR for channel SIP/xxx-000e5770

We don’t use crypto functions, but still I get:

[2017-02-13 08:43:56] WARNING[6735][C-001083f7] /usr/local/src/asterisk-13.13.1/include/asterisk/crypto.h: AES encryption disabled. Install OpenSSL.

These messages are repeating many times.

I understand that my questing is vague, but maybe anyone could give a hint in the right direction?
Have anyone accomplished an upgrade 1.x -> 13 sucessefully?

johnkiniston · February 13, 2017, 6:10pm

Did you follow the notes in the UPGRADE files from 1.8 to 11, 11 to 12, and 12 to 13?

Are you noloading modules you don’t need in your modules.conf ?

alex.svetkin · February 13, 2017, 6:18pm

Hi johnkiniston! Thank you for the reply.

Yes, I’ve followed upgrading instructions.

I don’t use SIP_CAUSE (noted as performance slowdown)
build with BUILD_NATIVE

I’ve removed all modules from 1.8 before “make install” and disabled all not used modules on “make menuselect”. But I’ll doublecheck that.

johnkiniston · February 13, 2017, 7:38pm

You have errors that make me think that you haven’t disabled the modules you thought you did.

For example:

[2017-02-10 18:07:21] ERROR[6671] pbx_lua.c: Error loading extensions.lua: cannot open ‘/etc/asterisk/extensions.lua’ for reading: No such file or directory
[2017-02-10 18:07:21] WARNING[6671] res_snmp.c: Could not load res_snmp.conf

I’d recommend looking at your modules.conf, Either disable the autoloader and hand load the modules you need or use the autoloader and noload the modules you don’t need.

david551 · February 13, 2017, 10:41pm

Modules that are loaded but not used should not consume many resources. There will be a certain amount of standing data, which might be on the heap and too fragmented to effectively page out, but most of the code will be deleted from memory, if memory is short, and only held in the image of the shared object file on disk. CPU consumption should be negligible.