On Saturday morning, I enabled the TLS transport in my PJSIP configuration and turned on SIPS for our Polycom phones and our provider trunks this weekend. I wanted to test TLS while we have a lower weekend call volume. Things are working, aside from some non-fatal errors. However, I am concerned that memory usage is gradually increasing since this change. Here is a graph from Observium showing memory usage the last 15 days:
I have never seen Asterisk consume memory like this.
I tried to investigate what was filling most of the memory, and it appears to be thousands of copies of the same CA certificates. I have less than 30 contacts online this weekend. Should I be concerned? Or should the dirty anonymous private pages eventually be released?
Find the PID…
# ps auxww | grep [/]usr/sbin/asterisk
root 20633 1.0 17.0 5182868 1003084 ? S<l Jan29 24:01 /usr/sbin/asterisk -C /var/taskeasy/gap/etc/asterisk.conf -p -f -g -U root -G asterisk
Summarize /proc/20633/smaps to see where the memory usage is:
# ~/bin/smaps-diag.pl 20633
[heap]:
private - [clean] 29.3 M [dirty]
shared - [clean] - [dirty]
[mmap]:
private 1.1 M [clean] 916.3 M [dirty]
shared 1.4 M [clean] - [dirty]
[stack]:
private - [clean] 84.0 k [dirty]
shared - [clean] - [dirty]
[vvar]:
private - [clean] - [dirty]
shared - [clean] - [dirty]
Sort maps by dirty pages, get the largest ones:
# pmap -x 20633 | sort -n -k 4 | tail
00007fafe4000000 65524 65524 65524 rw--- [ anon ]
00007fafd0000000 65532 65532 65532 rw--- [ anon ]
00007fafd4000000 65536 65536 65536 rw--- [ anon ]
00007fafec000000 65536 65536 65536 rw--- [ anon ]
00007fb0d0000000 65536 65536 65536 rw--- [ anon ]
00007fafb8000000 85872 85872 85872 rw--- [ anon ]
00007fafd8000000 131056 131056 131056 rw--- [ anon ]
00007fafc8000000 131060 131060 131060 rw--- [ anon ]
00007fafc0000000 131068 131068 131068 rw--- [ anon ]
total kB 5182868 1003844 972964
Find the page address range for the largest:
# cat /proc/20633/smaps | grep -B 9 'Private_Dirty:[[:space:]]\+131068'
7fafc0000000-7fafc7fff000 rw-p 00000000 00:00 0
Size: 131068 kB
KernelPageSize: 4 kB
MMUPageSize: 4 kB
Rss: 131068 kB
Pss: 131068 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 131068 kB
Dump that memory:
# gdb -p 20633 <<EOF
dump binary memory /var/spool/asterisk/tmp/dirty-shm 0x7fafc0000000 0x7fafc7fff000
EOF
Chop out binary data:
strings /var/spool/asterisk/tmp/dirty-shm > /var/spool/asterisk/tmp/dirty-shm-strings
105M of the 128M is plain text:
# ls -alh /var/spool/asterisk/tmp/dirty-shm*
-rw-r--r-- 1 root root 128M Jan 31 11:14 /var/spool/asterisk/tmp/dirty-shm
-rw-r--r-- 1 root root 105M Jan 31 11:15 /var/spool/asterisk/tmp/dirty-shm-strings
I looked at the plain text, and noticed that had SIP protocol text and a huge amount of CA certificates. I was curious to see how many times the same certificate repeated in the same memory, and they number in the tens of thousands:
# openssl crl2pkcs7 -nocrl -certfile /var/spool/asterisk/tmp/dirty-shm-strings | openssl pkcs7 -print_certs -text -noout | grep Subject: | sort | uniq -c
772 Subject: C=GB, ST=Greater Manchester, L=Salford, O=Comodo CA Limited, CN=AAA Certificate Services
773 Subject: C=GB, ST=Greater Manchester, L=Salford, O=Sectigo Limited, CN=Sectigo RSA Domain Validation Secure Server CA
1 Subject: CN=0004F2173F38
4761 Subject: CN=Polycom Equipment Issuing CA 1
15537 Subject: CN=Polycom Equipment Issuing CA 2
20294 Subject: CN=Polycom Equipment Policy CA
771 Subject: CN=*.taskeasy.com
1289 Subject: CN=us-west-or.sip.flowroute.com
1281 Subject: CN=us-west-wa.sip.flowroute.com
2447 Subject: C=US, O=DigiCert Inc, OU=www.digicert.com, CN=Thawte RSA CA 2018
2572 Subject: C=US, O=Let's Encrypt, CN=R3
1070 Subject: C=US, ST=California, L=San Francisco, O=Twilio, Inc., CN=*.pstn.twilio.com
773 Subject: C=US, ST=New Jersey, L=Jersey City, O=The USERTRUST Network, CN=USERTrust RSA Certification Authority
1 Subject: O=Polycom Inc., CN=0004F22DECCA
1 Subject: O=Polycom Inc., CN=0004F258F269
All of that for just 28 contacts.
# asterisk -rx 'pjsip show contacts' | grep ^Objects
Objects found: 28
In the time I wrote this post, it has continued to increase by 67M:
# ~/bin/smaps-diag.pl 20633
[heap]:
private - [clean] 29.3 M [dirty]
shared - [clean] - [dirty]
[mmap]:
private 1.1 M [clean] 983.9 M [dirty]
shared 1.4 M [clean] - [dirty]
[stack]:
private - [clean] 84.0 k [dirty]
shared - [clean] - [dirty]
[vvar]:
private - [clean] - [dirty]
shared - [clean] - [dirty]
