On Saturday morning, I enabled the TLS transport in my PJSIP configuration and turned on SIPS for our Polycom phones and our provider trunks this weekend. I wanted to test TLS while we have a lower weekend call volume. Things are working, aside from some non-fatal errors. However, I am concerned that memory usage is gradually increasing since this change. Here is a graph from Observium showing memory usage the last 15 days:
I have never seen Asterisk consume memory like this.
I tried to investigate what was filling most of the memory, and it appears to be thousands of copies of the same CA certificates. I have less than 30 contacts online this weekend. Should I be concerned? Or should the dirty anonymous private pages eventually be released?
Find the PID…
# ps auxww | grep [/]usr/sbin/asterisk root 20633 1.0 17.0 5182868 1003084 ? S<l Jan29 24:01 /usr/sbin/asterisk -C /var/taskeasy/gap/etc/asterisk.conf -p -f -g -U root -G asterisk
/proc/20633/smaps to see where the memory usage is:
# ~/bin/smaps-diag.pl 20633 [heap]: private - [clean] 29.3 M [dirty] shared - [clean] - [dirty] [mmap]: private 1.1 M [clean] 916.3 M [dirty] shared 1.4 M [clean] - [dirty] [stack]: private - [clean] 84.0 k [dirty] shared - [clean] - [dirty] [vvar]: private - [clean] - [dirty] shared - [clean] - [dirty]
Sort maps by dirty pages, get the largest ones:
# pmap -x 20633 | sort -n -k 4 | tail 00007fafe4000000 65524 65524 65524 rw--- [ anon ] 00007fafd0000000 65532 65532 65532 rw--- [ anon ] 00007fafd4000000 65536 65536 65536 rw--- [ anon ] 00007fafec000000 65536 65536 65536 rw--- [ anon ] 00007fb0d0000000 65536 65536 65536 rw--- [ anon ] 00007fafb8000000 85872 85872 85872 rw--- [ anon ] 00007fafd8000000 131056 131056 131056 rw--- [ anon ] 00007fafc8000000 131060 131060 131060 rw--- [ anon ] 00007fafc0000000 131068 131068 131068 rw--- [ anon ] total kB 5182868 1003844 972964
Find the page address range for the largest:
# cat /proc/20633/smaps | grep -B 9 'Private_Dirty:[[:space:]]\+131068' 7fafc0000000-7fafc7fff000 rw-p 00000000 00:00 0 Size: 131068 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Rss: 131068 kB Pss: 131068 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 131068 kB
Dump that memory:
# gdb -p 20633 <<EOF dump binary memory /var/spool/asterisk/tmp/dirty-shm 0x7fafc0000000 0x7fafc7fff000 EOF
Chop out binary data:
strings /var/spool/asterisk/tmp/dirty-shm > /var/spool/asterisk/tmp/dirty-shm-strings
105M of the 128M is plain text:
# ls -alh /var/spool/asterisk/tmp/dirty-shm* -rw-r--r-- 1 root root 128M Jan 31 11:14 /var/spool/asterisk/tmp/dirty-shm -rw-r--r-- 1 root root 105M Jan 31 11:15 /var/spool/asterisk/tmp/dirty-shm-strings
I looked at the plain text, and noticed that had SIP protocol text and a huge amount of CA certificates. I was curious to see how many times the same certificate repeated in the same memory, and they number in the tens of thousands:
# openssl crl2pkcs7 -nocrl -certfile /var/spool/asterisk/tmp/dirty-shm-strings | openssl pkcs7 -print_certs -text -noout | grep Subject: | sort | uniq -c 772 Subject: C=GB, ST=Greater Manchester, L=Salford, O=Comodo CA Limited, CN=AAA Certificate Services 773 Subject: C=GB, ST=Greater Manchester, L=Salford, O=Sectigo Limited, CN=Sectigo RSA Domain Validation Secure Server CA 1 Subject: CN=0004F2173F38 4761 Subject: CN=Polycom Equipment Issuing CA 1 15537 Subject: CN=Polycom Equipment Issuing CA 2 20294 Subject: CN=Polycom Equipment Policy CA 771 Subject: CN=*.taskeasy.com 1289 Subject: CN=us-west-or.sip.flowroute.com 1281 Subject: CN=us-west-wa.sip.flowroute.com 2447 Subject: C=US, O=DigiCert Inc, OU=www.digicert.com, CN=Thawte RSA CA 2018 2572 Subject: C=US, O=Let's Encrypt, CN=R3 1070 Subject: C=US, ST=California, L=San Francisco, O=Twilio, Inc., CN=*.pstn.twilio.com 773 Subject: C=US, ST=New Jersey, L=Jersey City, O=The USERTRUST Network, CN=USERTrust RSA Certification Authority 1 Subject: O=Polycom Inc., CN=0004F22DECCA 1 Subject: O=Polycom Inc., CN=0004F258F269
All of that for just 28 contacts.
# asterisk -rx 'pjsip show contacts' | grep ^Objects Objects found: 28
In the time I wrote this post, it has continued to increase by 67M:
# ~/bin/smaps-diag.pl 20633 [heap]: private - [clean] 29.3 M [dirty] shared - [clean] - [dirty] [mmap]: private 1.1 M [clean] 983.9 M [dirty] shared 1.4 M [clean] - [dirty] [stack]: private - [clean] 84.0 k [dirty] shared - [clean] - [dirty] [vvar]: private - [clean] - [dirty] shared - [clean] - [dirty]