Too many open files


#1

Hi All,

    I am getting too many open file error while using the Background application on the asterisk CLI. 

I am using Asterisk 1.6 version, dahdi, libpri packages.

Please find the error in detail

[May 17 14:23:13] WARNING[14751]: file.c:936 ast_streamfile: Unable to open ROM-senti-Hot-Pompt_H_17 (format 0x48 (alaw|slin)): Too many open files
[May 17 14:23:13] WARNING[14751]: pbx.c:7871 pbx_builtin_background: ast_streamfile failed on DAHDI/136-1 for ROM-senti-Hot-Pompt_H_17

Please let me know how to solve this issue. Is there a fix to this issue or should i try with a latest Asterisk version.

Awaiting a reply.

Thanks
Karthik


#2

What is the result of running

ulimit -n

at a Unix shell prompt?

Also what is your Asterisk version; you need all four components of the version number for 1.6.* versions.

What operating system are you using?


#3

Hi David,

the result of ulimit -n is 1024.

OS: CentOS 5.4

Please find the version details->

Asterisk-1.6.0.6
dahdi-linux-2.1.0.4
Dahdi-tools-2.1.0.2
libpri-1.4.10

Thanks
Karthik


#4

Firstly, how many concurrent calls do you have - you may really have exceeded 1024 file descriptors? (CentOS does allow you to set ulimit -n.)

There are two file descriptor leak related fixes in 1.6.0.27 that are not in 1.6.0.6, although one of them is just about diagnostics and the other is about repairing collateral damage that had not yet happened in your version:

r183028 | jpeeler | 2009-03-18 16:18:27 -0500 (Wed, 18 Mar 2009) | 4 lines

Add some code removed by mistake from commit 182722 that works around a file
descriptor leak in versions of PWLib prior to 1.12.0.

r187300 | tilghman | 2009-04-08 23:31:38 -0500 (Wed, 08 Apr 2009) | 3 lines

Add debugging mode for diagnosing file descriptor leaks.
(Related to issue #14625)

Your version is still very old, though.

Note 1.6.0 and 1.6.1 are closed for new bugs.


#5

We are using Asterisk 1.6.2 and also having this file descriptor leak issue.

The number of open file descriptors keep on increasing and Asterisk has to be restarted after every few days.

After few days, there are a lot of open file descriptors even if there are no calls running.

I have tried different Asterisk 1.6.x and also different Asterisk 1.8.x versions, but this issue is in them also.

Has anyone else also experienced similar problem?

Does anyone know of an Asterisk version that does not have this problem?

Thanks.


#6

I do not have such a problem on 1.8.7 on a server that handles 12,000/day. Are you doing anything with AGI that may be behaving badly? Are there an excessive number of processes when you run out of file handles?


#7

I have not tried 1.8.7 but did try 1.8.4 but the problem was there in it.

Excessive processes are not running when we run out of file handles.

How do I troubleshoot if anything may be there in AGI that may be causing this?

Thanks.


#8

The number of open files allowed is per session. So if you need to set the ulimit during rc startup somewhere. Lets say you stop asterisk in a ssh session. If it’s restarted double check that you have set the ulimit and leave an open ssh session because when you close that session it will reset the ulimit. A common practice is to open a “screen” session and set the ulimit -n 65000 then simply close the ssh window thus leaving the screen session on. Later at night you can restart and have rc set the ulimit there.


#9

Check if any limit is set in /etc/security/limits.conf. also if you have set asterisk to start automatically at startup then it might be using init level limit for open file descriptors. try stopping asterisk and then starting again but problem may arise when you restart server. or you can try following command to set limit at init level. ulimit -n 65536


#10

Changing the max number of pen file handles is fine and the recommendations that were offered are all valid. I feel however, that changing the limits may only be masking the problem.

Since you have noticed that the number of open handles remains high, even when there are no calls being processed, I really think something is not closing files properly.

If you do now know if you are running any AGI programs, it may take a bit more detective work, but it can be done.

Do we know what files are open when the problem is occurring?
Find the PID of the asterisk process using ‘ps’. Then list all the files open by that process with the command:

Is there a specific file that is open a whole lot of times?
If so, what is opening the files? Is it dialpaln in extensions.conf or some child process?
If not, it may not even be Asterisk and you may be hitting a system wide limit, not a per process limit.

If you can locate the file that is open a lot, the next step is determining what process is doing it.
Even if the open is listed with lsof as owned by the asterisk server, it may in fact be a child process, probably an AGI script. A quick check in /etc/asterisk will tell you if you call AGI().

cd /etc/asterisk
grep AGI exten*

Maybe these will give you a clue where to look further.

Dale


#11

With lsof -p asterisk_pid , I see that there are a lot of open udp ports.

So, it seems to be a problem of lingering UDP ports that do not close.

What could be the solution?

Thanks.


#12

What is ‘a lot of open udp ports’?
Can you post the output here for review?
Please post the output of ‘netstat -anp’. (you can hide any personal Internet facing IP addresses if you like)
I may be good to see the output of a ‘ps -ef’ as well.


#13

By open ports, I mean ports that are in use.

Here are the outputs when no call is running and Asterisk 1.6.2.0 has been running for around 3 days and has handled around 1800 calls in 3 days.

Here is the output of lsof -p asterisk_pid:

COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
asterisk 25873 root cwd DIR 8,2 4096 2 /
asterisk 25873 root rtd DIR 8,2 4096 2 /
asterisk 25873 root txt REG 8,2 17732933 2245622 /usr/sbin/asterisk
asterisk 25873 root mem REG 8,2 325830 2343340 /usr/lib/asterisk/modules/res_agi.so
asterisk 25873 root mem REG 8,2 243055 2343347 /usr/lib/asterisk/modules/res_musiconhold.so
asterisk 25873 root mem REG 8,2 164815 2343224 /usr/lib/asterisk/modules/app_exec.so
asterisk 25873 root mem REG 8,2 1247808 50499 /usr/lib/libxml2.so.2.6.26
asterisk 25873 root mem REG 8,2 160985 2343268 /usr/lib/asterisk/modules/app_while.so
asterisk 25873 root mem REG 8,2 156491 2343249 /usr/lib/asterisk/modules/app_sayunixtime.so
asterisk 25873 root mem REG 8,2 165048 2343313 /usr/lib/asterisk/modules/func_dialgroup.so
asterisk 25873 root mem REG 8,2 615850 2343195 /usr/lib/asterisk/modules/chan_oss.so
asterisk 25873 root mem REG 8,2 166089 2343298 /usr/lib/asterisk/modules/bridge_builtin_features.so
asterisk 25873 root mem REG 8,2 166242 2343258 /usr/lib/asterisk/modules/app_talkdetect.so
asterisk 25873 root mem REG 8,2 142563 2343289 /usr/lib/asterisk/modules/format_siren7.so
asterisk 25873 root mem REG 8,2 166755 2343245 /usr/lib/asterisk/modules/app_read.so
asterisk 25873 root mem REG 8,2 156673 2343309 /usr/lib/asterisk/modules/func_config.so
asterisk 25873 root mem REG 8,2 162055 2343203 /usr/lib/asterisk/modules/pbx_loopback.so
asterisk 25873 root mem REG 8,2 170463 2343248 /usr/lib/asterisk/modules/app_record.so
asterisk 25873 root mem REG 8,2 156297 2343333 /usr/lib/asterisk/modules/func_timeout.so
asterisk 25873 root mem REG 8,2 159284 2343336 /usr/lib/asterisk/modules/func_vmcount.so
asterisk 25873 root mem REG 8,2 155192 2343233 /usr/lib/asterisk/modules/app_milliwatt.so
asterisk 25873 root mem REG 8,2 152807 2343263 /usr/lib/asterisk/modules/app_verbose.so
asterisk 25873 root mem REG 8,2 143642 2343284 /usr/lib/asterisk/modules/format_h264.so
asterisk 25873 root mem REG 8,2 50839 2343456 /usr/lib/asterisk/modules/format_mp3.so
asterisk 25873 root mem REG 8,2 178861 2343321 /usr/lib/asterisk/modules/func_lock.so
asterisk 25873 root mem REG 8,2 155166 2343296 /usr/lib/asterisk/modules/cdr_custom.so
asterisk 25873 root mem REG 8,2 157561 2343282 /usr/lib/asterisk/modules/format_gsm.so
asterisk 25873 root mem REG 8,2 144487 2343270 /usr/lib/asterisk/modules/codec_adpcm.so
asterisk 25873 root mem REG 8,2 143685 2343283 /usr/lib/asterisk/modules/format_h263.so
asterisk 25873 root mem REG 8,2 159887 2343254 /usr/lib/asterisk/modules/app_softhangup.so
asterisk 25873 root mem REG 8,2 164691 2343216 /usr/lib/asterisk/modules/app_db.so
asterisk 25873 root mem REG 8,2 159892 2343200 /usr/lib/asterisk/modules/pbx_ael.so
asterisk 25873 root mem REG 8,2 142487 2343292 /usr/lib/asterisk/modules/format_vox.so
asterisk 25873 root mem REG 8,2 169685 2343237 /usr/lib/asterisk/modules/app_mp3.so
asterisk 25873 root mem REG 8,2 147671 2343210 /usr/lib/asterisk/modules/app_cdr.so
asterisk 25873 root mem REG 8,2 161653 2343316 /usr/lib/asterisk/modules/func_env.so
asterisk 25873 root mem REG 8,2 142572 2343291 /usr/lib/asterisk/modules/format_sln.so
asterisk 25873 root mem REG 8,2 158141 2343212 /usr/lib/asterisk/modules/app_channelredirect.so
asterisk 25873 root mem REG 8,2 169835 2343208 /usr/lib/asterisk/modules/app_amd.so
asterisk 25873 root mem REG 8,2 169187 2343299 /usr/lib/asterisk/modules/bridge_multiplexed.so
asterisk 25873 root mem REG 8,2 143659 2343285 /usr/lib/asterisk/modules/format_ilbc.so
asterisk 25873 root mem REG 8,2 154123 2343262 /usr/lib/asterisk/modules/app_userevent.so
asterisk 25873 root mem REG 8,2 155883 2343302 /usr/lib/asterisk/modules/func_aes.so
asterisk 25873 root mem REG 8,2 149235 2343305 /usr/lib/asterisk/modules/func_blacklist.so
asterisk 25873 root mem REG 8,2 151309 2343293 /usr/lib/asterisk/modules/format_wav.so
asterisk 25873 root mem REG 8,2 166927 2343240 /usr/lib/asterisk/modules/app_parkandannounce.so
asterisk 25873 root mem REG 8,2 154913 2343337 /usr/lib/asterisk/modules/func_volume.so
asterisk 25873 root mem REG 8,2 154608 2343261 /usr/lib/asterisk/modules/app_url.so
asterisk 25873 root mem REG 8,2 166520 2343310 /usr/lib/asterisk/modules/func_cut.so
asterisk 25873 root mem REG 8,2 161610 2343266 /usr/lib/asterisk/modules/app_waitforsilence.so
asterisk 25873 root mem REG 8,2 153313 2343314 /usr/lib/asterisk/modules/func_dialplan.so
asterisk 25873 root mem REG 8,2 151775 2343329 /usr/lib/asterisk/modules/func_shell.so
asterisk 25873 root mem REG 8,2 138517 2343272 /usr/lib/asterisk/modules/codec_a_mu.so
asterisk 25873 root mem REG 8,2 153223 2343335 /usr/lib/asterisk/modules/func_version.so
asterisk 25873 root mem REG 8,2 162799 2343318 /usr/lib/asterisk/modules/func_global.so
asterisk 25873 root mem REG 8,2 159609 2343211 /usr/lib/asterisk/modules/app_chanisavail.so
asterisk 25873 root mem REG 8,2 140883 2343286 /usr/lib/asterisk/modules/format_jpeg.so
asterisk 25873 root mem REG 8,2 165533 2343218 /usr/lib/asterisk/modules/app_dictate.so
asterisk 25873 root mem REG 8,2 149161 2343231 /usr/lib/asterisk/modules/app_image.so
asterisk 25873 root mem REG 8,2 141720 2343341 /usr/lib/asterisk/modules/res_clialiases.so
asterisk 25873 root mem REG 8,2 143679 2343281 /usr/lib/asterisk/modules/format_g729.so
asterisk 25873 root mem REG 8,2 27736 358322 /lib/libcrypt-2.5.so
asterisk 25873 root mem REG 8,2 175682 2343327 /usr/lib/asterisk/modules/func_realtime.so
asterisk 25873 root mem REG 8,2 101404 4840 /lib/libnsl-2.5.so
asterisk 25873 root mem REG 8,2 6467 2343455 /usr/lib/asterisk/modules/app_saycountpl.so
asterisk 25873 root mem REG 8,2 148177 2343280 /usr/lib/asterisk/modules/format_g726.so
asterisk 25873 root mem REG 8,2 151802 2343260 /usr/lib/asterisk/modules/app_transfer.so
asterisk 25873 root mem REG 8,2 249642 2343206 /usr/lib/asterisk/modules/app_adsiprog.so
asterisk 25873 root mem REG 8,2 749051 2343276 /usr/lib/asterisk/modules/codec_ilbc.so
asterisk 25873 root mem REG 8,2 153496 2343300 /usr/lib/asterisk/modules/bridge_simple.so
asterisk 25873 root mem REG 8,2 339403 2343194 /usr/lib/asterisk/modules/chan_mgcp.so
asterisk 25873 root mem REG 8,2 278532 2343190 /usr/lib/asterisk/modules/chan_agent.so
asterisk 25873 root mem REG 8,2 142614 2343290 /usr/lib/asterisk/modules/format_sln16.so
asterisk 25873 root mem REG 8,2 170521 2343204 /usr/lib/asterisk/modules/pbx_realtime.so
asterisk 25873 root mem REG 8,2 149595 2343328 /usr/lib/asterisk/modules/func_sha1.so
asterisk 25873 root mem REG 8,2 161565 2343342 /usr/lib/asterisk/modules/res_clioriginate.so
asterisk 25873 root mem REG 8,2 160829 2343236 /usr/lib/asterisk/modules/app_morsecode.so
asterisk 25873 root mem REG 8,2 140075 2343278 /usr/lib/asterisk/modules/codec_ulaw.so
asterisk 25873 root mem REG 8,2 190917 2343255 /usr/lib/asterisk/modules/app_speech_utils.so
asterisk 25873 root mem REG 8,2 167627 2343219 /usr/lib/asterisk/modules/app_directed_pickup.so
asterisk 25873 root mem REG 8,2 170669 2343308 /usr/lib/asterisk/modules/func_channel.so
asterisk 25873 root mem REG 8,2 172769 2343191 /usr/lib/asterisk/modules/chan_bridge.so
asterisk 25873 root mem REG 8,2 153822 2343320 /usr/lib/asterisk/modules/func_iconv.so
asterisk 25873 root mem REG 8,2 188323 2343232 /usr/lib/asterisk/modules/app_macro.so
asterisk 25873 root mem REG 8,2 238510 2343253 /usr/lib/asterisk/modules/app_sms.so
asterisk 25873 root mem REG 8,2 185724 2343214 /usr/lib/asterisk/modules/app_confbridge.so
asterisk 25873 root mem REG 8,2 160593 2343228 /usr/lib/asterisk/modules/app_forkcdr.so
asterisk 25873 root mem REG 8,2 223349 2343196 /usr/lib/asterisk/modules/chan_phone.so
asterisk 25873 root mem REG 8,2 156860 2343251 /usr/lib/asterisk/modules/app_sendtext.so
asterisk 25873 root mem REG 8,2 172712 2343294 /usr/lib/asterisk/modules/format_wav_gsm.so
asterisk 25873 root mem REG 8,2 180396 2343241 /usr/lib/asterisk/modules/app_playback.so
asterisk 25873 root mem REG 8,2 56109 2343458 /usr/lib/asterisk/modules/res_config_mysql.so
asterisk 25873 root mem REG 8,2 152182 2343334 /usr/lib/asterisk/modules/func_uri.so
asterisk 25873 root mem REG 8,2 154889 2343250 /usr/lib/asterisk/modules/app_senddtmf.so
asterisk 25873 root mem REG 8,2 182698 2343256 /usr/lib/asterisk/modules/app_stack.so
asterisk 25873 root mem REG 8,2 157244 2343303 /usr/lib/asterisk/modules/func_audiohookinherit.so
asterisk 25873 root mem REG 8,2 156365 2343450 /usr/lib/asterisk/modules/res_realtime.so
asterisk 25873 root mem REG 8,2 149243 2343324 /usr/lib/asterisk/modules/func_md5.so
asterisk 25873 root mem REG 8,2 142908 2343279 /usr/lib/asterisk/modules/format_g723.so
asterisk 25873 root mem REG 8,2 168400 2343238 /usr/lib/asterisk/modules/app_nbscat.so
asterisk 25873 root mem REG 8,2 530545 2343275 /usr/lib/asterisk/modules/codec_gsm.so
asterisk 25873 root mem REG 8,2 375746 2343199 /usr/lib/asterisk/modules/chan_unistim.so
asterisk 25873 root mem REG 8,2 165554 2343323 /usr/lib/asterisk/modules/func_math.so
asterisk 25873 root mem REG 8,2 157586 2343215 /usr/lib/asterisk/modules/app_controlplayback.so
asterisk 25873 root mem REG 8,2 157607 2343297 /usr/lib/asterisk/modules/cdr_manager.so
asterisk 25873 root mem REG 8,2 178400 2343205 /usr/lib/asterisk/modules/pbx_spool.so
asterisk 25873 root mem REG 8,2 142782 2343288 /usr/lib/asterisk/modules/format_siren14.so
asterisk 25873 root mem REG 8,2 548719 2343264 /usr/lib/asterisk/modules/app_voicemail.so
asterisk 25873 root mem REG 8,2 153791 2343223 /usr/lib/asterisk/modules/app_echo.so
asterisk 25873 root mem REG 8,2 170710 2343315 /usr/lib/asterisk/modules/func_enum.so
asterisk 25873 root mem REG 8,2 165480 2343306 /usr/lib/asterisk/modules/func_callerid.so
asterisk 25873 root mem REG 8,2 162183 2343239 /usr/lib/asterisk/modules/app_originate.so
asterisk 25873 root mem REG 8,2 157984 2343252 /usr/lib/asterisk/modules/app_setcallerid.so
asterisk 25873 root mem REG 8,2 193648 2343346 /usr/lib/asterisk/modules/res_monitor.so
asterisk 25873 root mem REG 8,2 155583 2343452 /usr/lib/asterisk/modules/res_speech.so
asterisk 25873 root mem REG 8,2 163219 2343243 /usr/lib/asterisk/modules/app_privacy.so
asterisk 25873 root mem REG 8,2 184108 2343221 /usr/lib/asterisk/modules/app_disa.so
asterisk 25873 root mem REG 8,2 158575 2343269 /usr/lib/asterisk/modules/app_zapateller.so
asterisk 25873 root mem REG 8,2 317221 2343234 /usr/lib/asterisk/modules/app_minivm.so
asterisk 25873 root mem REG 8,2 419519 2343277 /usr/lib/asterisk/modules/codec_lpc10.so
asterisk 25873 root mem REG 8,2 25976 2343457 /usr/lib/asterisk/modules/cdr_addon_mysql.so
asterisk 25873 root mem REG 8,2 165894 2343322 /usr/lib/asterisk/modules/func_logic.so
asterisk 25873 root mem REG 8,2 200138 2343193 /usr/lib/asterisk/modules/chan_local.so
asterisk 25873 root mem REG 8,2 155488 2343307 /usr/lib/asterisk/modules/func_cdr.so
asterisk 25873 root mem REG 8,2 205641 2343213 /usr/lib/asterisk/modules/app_chanspy.so
asterisk 25873 root mem REG 8,2 152047 2343242 /usr/lib/asterisk/modules/app_playtones.so
asterisk 25873 root mem REG 8,2 149204 2343325 /usr/lib/asterisk/modules/func_module.so
asterisk 25873 root mem REG 8,2 149124 2343453 /usr/lib/asterisk/modules/res_timing_pthread.so
asterisk 25873 root mem REG 8,2 155032 2343330 /usr/lib/asterisk/modules/func_sprintf.so
asterisk 25873 root mem REG 8,2 158449 2343222 /usr/lib/asterisk/modules/app_dumpchan.so
asterisk 25873 root mem REG 8,2 152395 2343287 /usr/lib/asterisk/modules/format_pcm.so
asterisk 25873 root mem REG 8,2 188752 2343331 /usr/lib/asterisk/modules/func_strings.so
asterisk 25873 root mem REG 8,2 201949 2343207 /usr/lib/asterisk/modules/app_alarmreceiver.so
asterisk 25873 root mem REG 8,2 157213 2343295 /usr/lib/asterisk/modules/cdr_csv.so
asterisk 25873 root mem REG 8,2 151857 2343332 /usr/lib/asterisk/modules/func_sysinfo.so
asterisk 25873 root mem REG 8,2 159484 2343311 /usr/lib/asterisk/modules/func_db.so
asterisk 25873 root mem REG 8,2 535415 2343344 /usr/lib/asterisk/modules/res_crypto.so
asterisk 25873 root mem REG 8,2 21501 2343454 /usr/lib/asterisk/modules/app_addon_sql_mysql.so
asterisk 25873 root mem REG 8,2 140097 2343271 /usr/lib/asterisk/modules/codec_alaw.so
asterisk 25873 root mem REG 8,2 166028 2343301 /usr/lib/asterisk/modules/bridge_softmix.so
asterisk 25873 root mem REG 8,2 125736 358314 /lib/ld-2.5.so
asterisk 25873 root mem REG 8,2 1589908 358315 /lib/libc-2.5.so
asterisk 25873 root mem REG 8,2 208352 292899 /lib/libm-2.5.so
asterisk 25873 root mem REG 8,2 16428 358316 /lib/libdl-2.5.so
asterisk 25873 root mem REG 8,2 125644 5708 /lib/libpthread-2.5.so
asterisk 25873 root mem REG 8,2 13084 5778 /lib/libtermcap.so.2.0.8
asterisk 25873 root mem REG 8,2 183610 2343235 /usr/lib/asterisk/modules/app_mixmonitor.so
asterisk 25873 root mem REG 8,2 7748 292908 /lib/libcom_err.so.2.1
asterisk 25873 root mem REG 8,2 155225 2343257 /usr/lib/asterisk/modules/app_system.so
asterisk 25873 root mem REG 8,2 75028 54300 /usr/lib/libz.so.1.2.3
asterisk 25873 root mem REG 8,2 242880 292904 /lib/libsepol.so.1
asterisk 25873 root mem REG 8,2 93508 292905 /lib/libselinux.so.1
asterisk 25873 root mem REG 8,2 286427 2343217 /usr/lib/asterisk/modules/app_dial.so
asterisk 25873 root mem REG 8,2 21788 4613 /lib/libnss_dns-2.5.so
asterisk 25873 root mem REG 8,2 141837 2343345 /usr/lib/asterisk/modules/res_limit.so
asterisk 25873 root mem REG 8,2 491552 2343198 /usr/lib/asterisk/modules/chan_skinny.so
asterisk 25873 root mem REG 8,2 229793 2343273 /usr/lib/asterisk/modules/codec_g722.so
asterisk 25873 root mem REG 8,2 549289 2343202 /usr/lib/asterisk/modules/pbx_dundi.so
asterisk 25873 root mem REG 8,2 257411 2343227 /usr/lib/asterisk/modules/app_followme.so
asterisk 25873 root mem REG 8,2 46548 292900 /lib/libgcc_s-4.1.2-20070626.so.1
asterisk 25873 root mem REG 8,2 221007 2343338 /usr/lib/asterisk/modules/res_adsi.so
asterisk 25873 root mem REG 8,2 76400 5597 /lib/libresolv-2.5.so
asterisk 25873 root mem REG 8,2 286748 158 /lib/libssl.so.0.9.8e
asterisk 25873 root mem REG 8,2 257895 2343201 /usr/lib/asterisk/modules/pbx_config.so
asterisk 25873 root mem REG 8,2 2049304 2343197 /usr/lib/asterisk/modules/chan_sip.so
asterisk 25873 root mem REG 8,2 842543 2343339 /usr/lib/asterisk/modules/res_ael_share.so
asterisk 25873 root mem REG 8,2 155819 2343265 /usr/lib/asterisk/modules/app_waitforring.so
asterisk 25873 root mem REG 8,2 161223 2343229 /usr/lib/asterisk/modules/app_getcpeid.so
asterisk 25873 root mem REG 8,2 150639 2343304 /usr/lib/asterisk/modules/func_base64.so
asterisk 25873 root mem REG 8,2 157304 57260 /usr/lib/libk5crypto.so.3.1
asterisk 25873 root mem REG 8,2 599540 57261 /usr/lib/libkrb5.so.3.3
asterisk 25873 root mem REG 8,2 33680 57259 /usr/lib/libkrb5support.so.0.1
asterisk 25873 root mem REG 8,2 190712 57262 /usr/lib/libgssapi_krb5.so.2.2
asterisk 25873 root mem REG 8,2 619055 2343449 /usr/lib/asterisk/modules/res_phoneprov.so
asterisk 25873 root mem REG 8,2 241562 2343451 /usr/lib/asterisk/modules/res_smdi.so
asterisk 25873 root mem REG 8,2 7880 5790 /lib/libkeyutils-1.2.so
asterisk 25873 root mem REG 8,2 602347 2343225 /usr/lib/asterisk/modules/app_externalivr.so
asterisk 25873 root mem REG 8,2 191931 2343220 /usr/lib/asterisk/modules/app_directory.so
asterisk 25873 root mem REG 8,2 150447 2343267 /usr/lib/asterisk/modules/app_waituntil.so
asterisk 25873 root mem REG 8,2 152289 2343326 /usr/lib/asterisk/modules/func_rand.so
asterisk 25873 root mem REG 8,2 46680 4615 /lib/libnss_files-2.5.so
asterisk 25873 root mem REG 8,2 158246 2343319 /usr/lib/asterisk/modules/func_groupcount.so
asterisk 25873 root mem REG 8,2 168945 2343246 /usr/lib/asterisk/modules/app_readexten.so
asterisk 25873 root mem REG 8,2 183573 2343226 /usr/lib/asterisk/modules/app_festival.so
asterisk 25873 root mem REG 8,2 168488 2343230 /usr/lib/asterisk/modules/app_ices.so
asterisk 25873 root mem REG 8,2 164371 2343274 /usr/lib/asterisk/modules/codec_g726.so
asterisk 25873 root mem REG 8,2 160838 2343247 /usr/lib/asterisk/modules/app_readfile.so
asterisk 25873 root mem REG 8,2 169872 2343259 /usr/lib/asterisk/modules/app_test.so
asterisk 25873 root mem REG 8,2 153353 2343317 /usr/lib/asterisk/modules/func_extstate.so
asterisk 25873 root mem REG 8,2 151301 2343343 /usr/lib/asterisk/modules/res_convert.so
asterisk 25873 root mem REG 8,2 165793 2343312 /usr/lib/asterisk/modules/func_devstate.so
asterisk 25873 root mem REG 8,2 165776 2343209 /usr/lib/asterisk/modules/app_authenticate.so
asterisk 25873 root mem REG 8,2 1444188 53514 /usr/lib/mysql/libmysqlclient.so.15.0.0
asterisk 25873 root mem REG 8,2 1296900 157 /lib/libcrypto.so.0.9.8e
asterisk 25873 root mem REG 8,2 1170828 2343192 /usr/lib/asterisk/modules/chan_iax2.so
asterisk 25873 root DEL REG 8,3 2545930 /var/tmp/iaxy.bin-2116089569
asterisk 25873 root 0u CHR 1,3 1314 /dev/null
asterisk 25873 root 1u CHR 1,3 1314 /dev/null
asterisk 25873 root 2u CHR 1,3 1314 /dev/null
asterisk 25873 root 3r CHR 1,9 1723 /dev/urandom
asterisk 25873 root 4u unix 0xf67de740 308012748 /var/run/asterisk.ctl
asterisk 25873 root 5w REG 8,3 3935 65284 /var/log/asterisk/messages
asterisk 25873 root 6w REG 8,3 0 98379 /var/log/asterisk/event_log
asterisk 25873 root 7w REG 8,3 45670 98378 /var/log/asterisk/queue_log
asterisk 25873 root 8u IPv4 308012772 TCP *:5038 (LISTEN)
asterisk 25873 root 9r DIR 0,9 0 369 inotify
asterisk 25873 root 10u REG 8,3 40960 328264 /var/lib/asterisk/astdb
asterisk 25873 root 11u IPv4 308012775 UDP *:4520
asterisk 25873 root 12u IPv4 308012776 TCP *:sieve (LISTEN)
asterisk 25873 root 14u IPv4 308012778 UDP *:commplex-main
asterisk 25873 root 15r FIFO 0,6 308012779 pipe
asterisk 25873 root 16w FIFO 0,6 308012779 pipe
asterisk 25873 root 17u IPv4 308012780 UDP *:iax
asterisk 25873 root 18u IPv4 308012782 UDP *:mgcp-callagent
asterisk 25873 root 19u IPv4 308012784 UDP *:sip
asterisk 25873 root 21u IPv4 308012785 TCP A.A.A:sip-tls (LISTEN)
asterisk 25873 root 22u unix 0xf5e78200 308012786 socket
asterisk 25873 root 23r FIFO 0,6 308012790 pipe
asterisk 25873 root 24w FIFO 0,6 308012790 pipe
asterisk 25873 root 25u IPv4 308703045 UDP *:54444
asterisk 25873 root 26u IPv4 308703046 UDP *:54445
asterisk 25873 root 27u IPv4 308703047 UDP *:4486
asterisk 25873 root 39u IPv4 308719542 UDP *:14952
asterisk 25873 root 41w IPv4 308719543 UDP *:14953
asterisk 25873 root 42r IPv4 308719544 UDP *:4978
asterisk 25873 root 43u IPv4 309602362 UDP *:42500
asterisk 25873 root 44u IPv4 309602363 UDP *:42501
asterisk 25873 root 45w IPv4 309602364 UDP *:4185
asterisk 25873 root 46u IPv4 308779409 UDP *:14942
asterisk 25873 root 47r IPv4 308779410 UDP *:14943
asterisk 25873 root 48w IPv4 308779411 UDP *:4797
asterisk 25873 root 52u REG 8,3 39402 2545930 /var/tmp/iaxy.bin-2116089569 (deleted)
asterisk 25873 root 58u IPv4 308707947 UDP *:33668
asterisk 25873 root 61u IPv4 308202346 UDP *:16234
asterisk 25873 root 62u IPv4 308202347 UDP *:16235
asterisk 25873 root 63u IPv4 308202348 UDP *:4158
asterisk 25873 root 64u IPv4 308202834 UDP *:19814
asterisk 25873 root 65u IPv4 308202835 UDP *:19815
asterisk 25873 root 66u IPv4 308202836 UDP *:4085
asterisk 25873 root 67u IPv4 308707948 UDP *:33669
asterisk 25873 root 68u IPv4 308707949 UDP *:4923
asterisk 25873 root 72u IPv4 309444248 UDP *:5846
asterisk 25873 root 75u IPv4 309366741 UDP *:45618
asterisk 25873 root 76u IPv4 309366742 UDP *:45619
asterisk 25873 root 79u IPv4 309366743 UDP *:4081
asterisk 25873 root 86u IPv4 309539906 UDP *:apc-5454
asterisk 25873 root 88u IPv4 308864275 UDP *:37038
asterisk 25873 root 89u IPv4 308864276 UDP *:37039
asterisk 25873 root 90u IPv4 308864277 UDP *:4273
asterisk 25873 root 95u IPv4 309539907 UDP *:apc-5455
asterisk 25873 root 96u IPv4 309539908 UDP *:4072
asterisk 25873 root 104u IPv4 309444249 UDP *:5847
asterisk 25873 root 105u IPv4 309444250 UDP *:4607
asterisk 25873 root 107u IPv4 309443468 UDP *:44456
asterisk 25873 root 108u IPv4 309443469 UDP *:44457
asterisk 25873 root 148u IPv4 309443470 UDP *:4339
asterisk 25873 root 174u IPv4 309519940 UDP *:19770
asterisk 25873 root 175u IPv4 309519941 UDP *:19771
asterisk 25873 root 176u IPv4 309519942 UDP *:4117

Here is the output of netstat -anp :

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 x.x.x.x:5061 0.0.0.0:* LISTEN 25873/asterisk
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 2946/mysqld
tcp 0 0 0.0.0.0:5038 0.0.0.0:* LISTEN 25873/asterisk
tcp 0 0 0.0.0.0:2000 0.0.0.0:* LISTEN 25873/asterisk
tcp 0 0 0.0.0.0:10000 0.0.0.0:* LISTEN 2584/perl
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 2498/sendmail: acce
tcp 0 0 :::80 :::* LISTEN 2322/httpd
tcp 0 0 :::22 :::* LISTEN 2080/sshd
tcp 0 0 :::443 :::* LISTEN 2322/httpd
tcp 0 0 ::ffff:x.x.x.x:80 ::ffff:x.x.x.x:50792 TIME_WAIT -
tcp 0 0 ::ffff:x.x.x.x:80 ::ffff:x.x.x.x:9083 TIME_WAIT -
tcp 0 0 ::ffff:x.x.x.x:80 ::ffff:x.x.x.x:50784 TIME_WAIT -
tcp 0 0 ::ffff:x.x.x.x:80 ::ffff:x.x.x.x:50791 TIME_WAIT -
tcp 0 0 ::ffff:x.x.x.x:80 ::ffff:x.x.x.x:50790 TIME_WAIT -
tcp 0 0 ::ffff:x.x.x.x:80 ::ffff:x.x.x.x:5492 TIME_WAIT -
tcp 0 52 ::ffff:x.x.x.x:22 ::ffff:x.x.x.x:49255 ESTABLISHED 3463/0
tcp 0 0 ::ffff:x.x.x.x:80 ::ffff:x.x.x.x:50783 TIME_WAIT -
tcp 0 0 ::ffff:x.x.x.x:80 ::ffff:x.x.x.x:1176 TIME_WAIT -
tcp 0 0 ::ffff:x.x.x.x:80 ::ffff:x.x.x.x:6188 TIME_WAIT -
tcp 0 0 ::ffff:x.x.x.x:80 ::ffff:x.x.x.x:1173 TIME_WAIT -
tcp 0 0 ::ffff:x.x.x.x:80 ::ffff:x.x.x.x:26400 TIME_WAIT -
tcp 0 0 ::ffff:x.x.x.x:80 ::ffff:x.x.x.x:18394 TIME_WAIT -
tcp 0 0 ::ffff:x.x.x.x:80 ::ffff:x.x.x.x:5086 TIME_WAIT -
tcp 0 0 ::ffff:x.x.x.x:80 ::ffff:x.x.x.x:5789 TIME_WAIT -
tcp 0 0 ::ffff:x.x.x.x:80 ::ffff:x.x.x.x:2435 TIME_WAIT -
udp 0 0 0.0.0.0:42500 0.0.0.0:* 25873/asterisk
udp 0 0 0.0.0.0:33668 0.0.0.0:* 25873/asterisk
udp 0 0 0.0.0.0:42501 0.0.0.0:* 25873/asterisk
udp 0 0 0.0.0.0:33669 0.0.0.0:* 25873/asterisk
udp 0 0 0.0.0.0:4486 0.0.0.0:* 25873/asterisk
udp 0 0 0.0.0.0:5000 0.0.0.0:* 25873/asterisk
udp 0 0 0.0.0.0:10000 0.0.0.0:* 2584/perl
udp 0 0 0.0.0.0:4117 0.0.0.0:* 25873/asterisk
udp 0 0 0.0.0.0:2727 0.0.0.0:* 25873/asterisk
udp 0 0 0.0.0.0:44456 0.0.0.0:* 25873/asterisk
udp 0 0 0.0.0.0:4520 0.0.0.0:* 25873/asterisk
udp 428 0 0.0.0.0:44457 0.0.0.0:* 25873/asterisk
udp 0 0 0.0.0.0:54444 0.0.0.0:* 25873/asterisk
udp 0 0 0.0.0.0:54445 0.0.0.0:* 25873/asterisk
udp 428 0 0.0.0.0:37038 0.0.0.0:* 25873/asterisk
udp 0 0 0.0.0.0:37039 0.0.0.0:* 25873/asterisk
udp 0 0 0.0.0.0:4273 0.0.0.0:* 25873/asterisk
udp 0 0 0.0.0.0:45618 0.0.0.0:* 25873/asterisk
udp 428 0 0.0.0.0:45619 0.0.0.0:* 25873/asterisk
udp 428 0 0.0.0.0:19770 0.0.0.0:* 25873/asterisk
udp 856 0 0.0.0.0:19771 0.0.0.0:* 25873/asterisk
udp 0 0 0.0.0.0:4923 0.0.0.0:* 25873/asterisk
udp 0 0 0.0.0.0:4797 0.0.0.0:* 25873/asterisk
udp 0 0 0.0.0.0:4158 0.0.0.0:* 25873/asterisk
udp 0 0 0.0.0.0:5060 0.0.0.0:* 25873/asterisk
udp 428 0 0.0.0.0:5454 0.0.0.0:* 25873/asterisk
udp 428 0 0.0.0.0:5455 0.0.0.0:* 25873/asterisk
udp 0 0 0.0.0.0:5846 0.0.0.0:* 25873/asterisk
udp 428 0 0.0.0.0:5847 0.0.0.0:* 25873/asterisk
udp 0 0 0.0.0.0:4185 0.0.0.0:* 25873/asterisk
udp 0 0 0.0.0.0:4569 0.0.0.0:* 25873/asterisk
udp 428 0 0.0.0.0:14942 0.0.0.0:* 25873/asterisk
udp 428 0 0.0.0.0:14943 0.0.0.0:* 25873/asterisk
udp 0 0 0.0.0.0:19814 0.0.0.0:* 25873/asterisk
udp 0 0 0.0.0.0:19815 0.0.0.0:* 25873/asterisk
udp 0 0 0.0.0.0:4072 0.0.0.0:* 25873/asterisk
udp 428 0 0.0.0.0:14952 0.0.0.0:* 25873/asterisk
udp 0 0 0.0.0.0:14953 0.0.0.0:* 25873/asterisk
udp 0 0 0.0.0.0:16234 0.0.0.0:* 25873/asterisk
udp 428 0 0.0.0.0:16235 0.0.0.0:* 25873/asterisk
udp 0 0 0.0.0.0:4081 0.0.0.0:* 25873/asterisk
udp 0 0 0.0.0.0:4978 0.0.0.0:* 25873/asterisk
udp 0 0 0.0.0.0:4339 0.0.0.0:* 25873/asterisk
udp 0 0 0.0.0.0:4085 0.0.0.0:* 25873/asterisk
udp 0 0 x.x.x.x:123 0.0.0.0:* 2115/ntpd
udp 0 0 127.0.0.1:123 0.0.0.0:* 2115/ntpd
udp 0 0 0.0.0.0:123 0.0.0.0:* 2115/ntpd
udp 0 0 0.0.0.0:4607 0.0.0.0:* 25873/asterisk
udp 0 0 ::1:123 :::* 2115/ntpd
udp 0 0 :::123 :::* 2115/ntpd
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node PID/Program name Path
unix 2 [ ACC ] STREAM LISTENING 4841 1885/python /var/run/audit_events
unix 2 [ ACC ] STREAM LISTENING 208573828 2946/mysqld /var/lib/mysql/mysql.sock
unix 2 [ ] DGRAM 1329 493/udevd @/org/kernel/udev/udevd
unix 2 [ ACC ] STREAM LISTENING 5595 2257/hald @/var/run/hald/dbus-WktfCRh0R6
unix 2 [ ACC ] STREAM LISTENING 5490 2213/xfs /tmp/.font-unix/fs7100
unix 2 [ ACC ] STREAM LISTENING 4960 1961/dbus-daemon /var/run/dbus/system_bus_socket
unix 2 [ ACC ] STREAM LISTENING 5149 2052/acpid /var/run/acpid.socket
unix 2 [ ACC ] STREAM LISTENING 308012748 25873/asterisk /var/run/asterisk.ctl
unix 2 [ ] DGRAM 5603 2257/hald @/org/freedesktop/hal/udev_event
unix 2 [ ACC ] STREAM LISTENING 5594 2257/hald @/var/run/hald/dbus-uSrPjl5Njj
unix 2 [ ACC ] STREAM LISTENING 5584 2167/gpm /dev/gpmctl
unix 15 [ ] DGRAM 4836 1903/syslogd /dev/log
unix 3 [ ] STREAM CONNECTED 309815149 2946/mysqld /var/lib/mysql/mysql.sock
unix 3 [ ] STREAM CONNECTED 309815148 17601/httpd
unix 3 [ ] STREAM CONNECTED 309815084 2946/mysqld /var/lib/mysql/mysql.sock
unix 3 [ ] STREAM CONNECTED 309815083 23240/httpd
unix 2 [ ] DGRAM 309815077 3463/0
unix 3 [ ] STREAM CONNECTED 309815023 2946/mysqld /var/lib/mysql/mysql.sock
unix 3 [ ] STREAM CONNECTED 309815022 17966/httpd
unix 3 [ ] STREAM CONNECTED 309814990 2946/mysqld /var/lib/mysql/mysql.sock
unix 3 [ ] STREAM CONNECTED 309814989 11255/httpd
unix 3 [ ] STREAM CONNECTED 309814961 2946/mysqld /var/lib/mysql/mysql.sock
unix 3 [ ] STREAM CONNECTED 309814960 26732/httpd
unix 3 [ ] STREAM CONNECTED 309814941 2946/mysqld /var/lib/mysql/mysql.sock
unix 3 [ ] STREAM CONNECTED 309814940 29669/httpd
unix 3 [ ] STREAM CONNECTED 309814927 2946/mysqld /var/lib/mysql/mysql.sock
unix 3 [ ] STREAM CONNECTED 309814926 17963/httpd
unix 3 [ ] STREAM CONNECTED 309814919 2946/mysqld /var/lib/mysql/mysql.sock
unix 3 [ ] STREAM CONNECTED 309814918 26880/httpd
unix 3 [ ] STREAM CONNECTED 309814901 2946/mysqld /var/lib/mysql/mysql.sock
unix 3 [ ] STREAM CONNECTED 309814900 26875/httpd
unix 3 [ ] STREAM CONNECTED 308012787 2946/mysqld /var/lib/mysql/mysql.sock
unix 3 [ ] STREAM CONNECTED 308012786 25873/asterisk
unix 2 [ ] DGRAM 136310 1883/auditd
unix 2 [ ] DGRAM 7204 2584/perl
unix 2 [ ] DGRAM 7059 2506/clientmqueue
unix 2 [ ] DGRAM 7030 2498/sendmail: acce
unix 3 [ ] STREAM CONNECTED 6673 1961/dbus-daemon /var/run/dbus/system_bus_socket
unix 3 [ ] STREAM CONNECTED 6672 2257/hald
unix 3 [ ] STREAM CONNECTED 6485 2052/acpid /var/run/acpid.socket
unix 3 [ ] STREAM CONNECTED 6484 2264/acpid.socket
unix 3 [ ] STREAM CONNECTED 6477 2257/hald @/var/run/hald/dbus-uSrPjl5Njj
unix 3 [ ] STREAM CONNECTED 6475 2264/acpid.socket
unix 3 [ ] STREAM CONNECTED 5598 2257/hald @/var/run/hald/dbus-WktfCRh0R6
unix 3 [ ] STREAM CONNECTED 5597 2258/hald-runner
unix 2 [ ] DGRAM 5414 2184/crond
unix 2 [ ] DGRAM 5378 2167/gpm
unix 2 [ ] DGRAM 5290 2115/ntpd
unix 2 [ ] DGRAM 5257 2096/xinetd
unix 2 [ ] DGRAM 5098 2028/automount
unix 2 [ ] DGRAM 5024 2001/hidd
unix 3 [ ] STREAM CONNECTED 4963 1961/dbus-daemon
unix 3 [ ] STREAM CONNECTED 4962 1961/dbus-daemon
unix 2 [ ] DGRAM 4849 1906/klogd
unix 2 [ ] DGRAM 4840 1885/python
unix 3 [ ] STREAM CONNECTED 4798 1883/auditd
unix 3 [ ] STREAM CONNECTED 4797 1885/python

Here is the output of ps -ef :

UID PID PPID C STIME TTY TIME CMD
root 1 0 0 2010 ? 00:02:02 init [3]
root 2 1 0 2010 ? 00:00:05 [migration/0]
root 3 1 0 2010 ? 00:00:00 [ksoftirqd/0]
root 4 1 0 2010 ? 00:00:00 [watchdog/0]
root 5 1 0 2010 ? 00:00:02 [migration/1]
root 6 1 0 2010 ? 00:00:00 [ksoftirqd/1]
root 7 1 0 2010 ? 00:00:00 [watchdog/1]
root 8 1 0 2010 ? 00:00:01 [migration/2]
root 9 1 0 2010 ? 00:00:00 [ksoftirqd/2]
root 10 1 0 2010 ? 00:00:00 [watchdog/2]
root 11 1 0 2010 ? 00:00:03 [migration/3]
root 12 1 0 2010 ? 00:00:00 [ksoftirqd/3]
root 13 1 0 2010 ? 00:00:00 [watchdog/3]
root 14 1 0 2010 ? 00:00:00 [events/0]
root 15 1 0 2010 ? 00:00:00 [events/1]
root 16 1 0 2010 ? 00:00:00 [events/2]
root 17 1 0 2010 ? 00:00:00 [events/3]
root 18 1 0 2010 ? 00:00:00 [khelper]
root 19 1 0 2010 ? 00:00:00 [kthread]
root 25 19 0 2010 ? 00:00:00 [kblockd/0]
root 26 19 0 2010 ? 00:00:09 [kblockd/1]
root 27 19 0 2010 ? 00:00:00 [kblockd/2]
root 28 19 0 2010 ? 00:00:03 [kblockd/3]
root 29 19 0 2010 ? 00:00:00 [kacpid]
root 130 19 0 2010 ? 00:00:00 [cqueue/0]
root 131 19 0 2010 ? 00:00:00 [cqueue/1]
root 132 19 0 2010 ? 00:00:00 [cqueue/2]
root 133 19 0 2010 ? 00:00:00 [cqueue/3]
root 136 19 0 2010 ? 00:00:00 [khubd]
root 138 19 0 2010 ? 00:00:00 [kseriod]
root 214 19 0 2010 ? 00:00:03 [kswapd0]
root 215 19 0 2010 ? 00:00:00 [aio/0]
root 216 19 0 2010 ? 00:00:00 [aio/1]
root 217 19 0 2010 ? 00:00:00 [aio/2]
root 218 19 0 2010 ? 00:00:00 [aio/3]
root 375 19 0 2010 ? 00:00:00 [kpsmoused]
root 420 19 0 2010 ? 00:00:00 [ata/0]
root 421 19 0 2010 ? 00:00:00 [ata/1]
root 422 19 0 2010 ? 00:00:00 [ata/2]
root 423 19 0 2010 ? 00:00:00 [ata/3]
root 424 19 0 2010 ? 00:00:00 [ata_aux]
root 430 19 0 2010 ? 00:00:00 [scsi_eh_0]
root 431 19 0 2010 ? 00:00:00 [scsi_eh_1]
root 432 19 0 2010 ? 00:05:10 [kjournald]
root 459 19 0 2010 ? 00:00:27 [kauditd]
root 493 1 0 2010 ? 00:00:00 /sbin/udevd -d
root 1356 19 0 2010 ? 00:00:00 [kjournald]
root 1358 19 0 2010 ? 00:00:00 [kjournald]
root 1360 19 0 2010 ? 03:32:58 [kjournald]
root 1883 1 0 2010 ? 00:04:42 auditd
root 1885 1883 0 2010 ? 00:06:44 python /sbin/audispd
root 1903 1 0 2010 ? 00:48:15 syslogd -m 0
root 1906 1 0 2010 ? 00:00:00 klogd -x
root 1922 1 0 2010 ? 00:00:00 irqbalance
dbus 1961 1 0 2010 ? 00:00:00 dbus-daemon --system
root 2001 1 0 2010 ? 00:00:00 /usr/bin/hidd --server
root 2028 1 0 2010 ? 00:02:06 automount
root 2052 1 0 2010 ? 00:00:00 /usr/sbin/acpid
root 2080 1 0 2010 ? 00:08:49 /usr/sbin/sshd
root 2096 1 0 2010 ? 00:00:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
ntp 2115 1 0 2010 ? 00:00:00 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
root 2167 1 0 2010 ? 00:00:00 gpm -m /dev/input/mice -t exps2
root 2184 1 0 2010 ? 00:00:05 crond
xfs 2213 1 0 2010 ? 00:00:00 xfs -droppriv -daemon
root 2242 1 0 2010 ? 00:00:00 /usr/sbin/atd
68 2257 1 0 2010 ? 00:00:00 hald
root 2258 2257 0 2010 ? 00:00:00 hald-runner
68 2264 2258 0 2010 ? 00:00:00 hald-addon-acpi: listening on acpid socket /var/run/acpid.socket
root 2322 1 0 2010 ? 00:00:40 /usr/sbin/httpd
root 2498 1 0 2010 ? 00:00:52 sendmail: accepting connections
smmsp 2506 1 0 2010 ? 00:00:00 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue
root 2571 1 0 2010 ? 00:00:02 /usr/sbin/smartd -q never
root 2584 1 0 2010 ? 00:00:00 /usr/bin/perl /usr/libexec/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 2588 1 0 2010 tty1 00:00:00 /sbin/mingetty tty1
root 2589 1 0 2010 tty2 00:00:00 /sbin/mingetty tty2
root 2590 1 0 2010 tty3 00:00:00 /sbin/mingetty tty3
root 2595 1 0 2010 tty4 00:00:00 /sbin/mingetty tty4
root 2597 1 0 2010 tty5 00:00:00 /sbin/mingetty tty5
root 2599 1 0 2010 tty6 00:00:00 /sbin/mingetty tty6
root 2910 1 0 Jul02 ? 00:00:00 /bin/sh /usr/bin/mysqld_safe --defaults-file=/etc/my.cnf --pid-file=/var/run/mysqld/mysqld.pid --log-error=/var/log/mysqld.log
mysql 2946 2910 3 Jul02 ? 5-14:04:45 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-locking --socket=/var/lib/mysql/mysql.sock
root 3463 2080 0 05:54 ? 00:00:00 sshd: root@pts/0
root 3467 3463 0 05:54 pts/0 00:00:00 -bash
root 3496 3467 0 05:54 pts/0 00:00:00 screen
root 3497 3496 0 05:54 ? 00:00:00 SCREEN
root 3498 3497 0 05:54 pts/1 00:00:00 /bin/bash
root 3915 3498 0 06:03 pts/1 00:00:00 ps -ef
root 10643 19 0 Dec12 ? 00:00:00 [pdflush]
apache 11255 2322 0 Dec25 ? 00:00:01 /usr/sbin/httpd
apache 17601 2322 0 Dec25 ? 00:00:00 /usr/sbin/httpd
apache 17963 2322 0 Dec25 ? 00:00:01 /usr/sbin/httpd
apache 17966 2322 0 Dec25 ? 00:00:00 /usr/sbin/httpd
apache 17967 2322 0 Dec25 ? 00:00:00 /usr/sbin/httpd
apache 23240 2322 0 Dec25 ? 00:00:02 /usr/sbin/httpd
root 24245 1 0 Dec15 ? 00:01:42 /usr/bin/perl -w /var/lib/asterisk/agi-bin/nslib/nscallback/nscallbackd
root 24646 19 0 Dec15 ? 00:00:00 [pdflush]
root 25873 1 0 Dec23 ? 00:15:39 /usr/sbin/asterisk
apache 26732 2322 0 Dec25 ? 00:00:02 /usr/sbin/httpd
apache 26875 2322 0 Dec25 ? 00:00:02 /usr/sbin/httpd
apache 26880 2322 0 Dec25 ? 00:00:02 /usr/sbin/httpd
apache 27805 2322 0 Dec25 ? 00:00:02 /usr/sbin/httpd
apache 29669 2322 0 Dec25 ? 00:00:01 /usr/sbin/httpd

Thanks.


#14

I agree that it appears there are some possibly hung calls.

A couple of more questions.

Do you know what the script, /var/lib/asterisk/agi-bin/nslib/nscallback/nscallbackd, does?

A the time when you are seeing the problem, what is the output of the following Asterisk CLI command?

Dale


#15

Thanks for your reply.

The script /var/lib/asterisk/agi-bin/nslib/nscallback/nscallbackd runs an infinite loop, which connects to Asterisk manager and sends an Originate command and then disconnects from the Asterisk Manager.

The output of core show channels verbose is (when there are 3 active calls):

centosCLI> core show channels verbose
Channel Context Extension Prio State Application Data CallerID Duration Accountcode BridgedTo
SIP/x.x.x.x-00 xx 00111111 1 Up AppDial (Outgoing Line) 00111111 00:00:15 xxx SIP/xxx-00001a26
SIP/xxx-00001a26 xx 00111111 1 Up Dial SIP/x.x.x.x/0000011 00111111 00:00:16 xxx SIP/x.x.x.x-00
SIP/x.x.x.x-00 xx 00111111 1 Down AppDial (Outgoing Line) 00111111 00:00:26 xxx (None)
SIP/xxx-00001a24 xx 00111111 1 Ring Dial SIP/x.x.x.x/00111 00111111 00:00:27 xxx (None)
SIP/x.x.x.x-00 xx 00111111 1 Up AppDial (Outgoing Line) 00111111 00:02:09 xxx SIP/xxx-00001a22
SIP/xxx-00001a22 xx 00111111 1 Up Dial SIP/x.x.x.x/0000011 00111111 00:02:10 xxx SIP/x.x.x.x-00
6 active channels
3 active calls
3972 calls processed

Also, sometimes the following error is being received:

ERROR[4104] utils.c: fwrite() returned error: Broken pipe

Thanks.


#16

I have a feeling this is related to the nscallbackd script.
Is this function used a lot? Is it possible that in some cases it is leaving some connections open?


#17

Thanks for your reply.

I do not know if it is leaving some connections open.

But, yes, it is used a lot.

nscallbackd script is a Perl script. It spawns a child process and connects to Asterisk Manager and makes a call and disconnects.

Should I post it here for review?


#18

If you would like to post it, I can take a look.


#19

Following is the nscallbackd script, I have deleted the SQL queries.

#!/usr/bin/perl -w

use strict;
use DBI;
use POSIX qw(setsid);
use vars qw($dbh);

use lib ‘/var/lib/asterisk/agi-bin/nslib/nscallback/lib’;
use Asterisk::Manager;
use Config::IniFiles;

$|++;

######################### GLOBALS ########################
my $path_daemon = “/var/lib/asterisk/agi-bin/nslib/nscallback/”;
my $daemon = “CallBack-Daemon”;

######################### READ THE CONFIG FILE ########################
my $conf_file = “/etc/asterisk/billing.conf”;

Read the config file

my $cfg = Config::IniFiles->new( -file => $conf_file );

if (not defined $cfg) {
print “Failed to parse $conf_file: \n”;
foreach(@Config::IniFiles::errors) {
print “Error: $_\n” ;
}
exit(1);
}

######################### DB PARAMETER ########################
my $dbname = $cfg->val(‘database’, ‘dbname’);
my $dbhost = $cfg->val(‘database’, ‘hostname’);
my $dbport = $cfg->val(‘database’, ‘port’);
my $login = $cfg->val(‘database’, ‘user’);
my $pwd = $cfg->val(‘database’, ‘password’);
my $dbtype = $cfg->val(‘database’, ‘dbtype’);

######################### LOG FILE ###########################
my $log_file = “/var/lib/asterisk/agi-bin/nslib/log/nscallback.log”;

######################### ASTERISK MANAGER ########################
my $manager_host = $cfg->val(‘global’, ‘manager_host’);
my $manager_username = $cfg->val(‘global’, ‘manager_username’);
my $manager_secret = $cfg->val(‘global’, ‘manager_secret’);

###################### BEHAVIOUR PARAMETER ########################
my $pass_nb = 0;

my $trace = ‘/tmp/pgtrace.out’;
my ($conn, $result, $i);
my ($status, $timestamp);

my $DEBUG = 0; # set this to 1 for traces

my @childs; # define child in the daemon

if (($path_daemon eq “”) or ($daemon eq “”))
{
print “Error in Config File check it!!\n”;
exit(0);
}

SET SIGNAL HANDLER

$SIG{‘TERM’} = ‘INT_handler’;
my $set_exit = 0;

TEST MANAGER PARAMETER

my $astman = new Asterisk::Manager;

$astman->user($manager_username);

$astman->secret($manager_secret);

$astman->host($manager_host);

$status = $astman->connect;

my @res_manager = $astman->sendcommand( Action => ‘Command’, Command => ‘show channels’);

print @res_manager;

exit(0);

&daemonize($path_daemon);

open(LOGFILE, “>>”.$log_file) || warn “Could not open >> $log_file.\n”;
$timestamp = &longfmt();
print LOGFILE “$timestamp”;
print LOGFILE “[DAEMON START FORK]\n”;
close LOGFILE;

our infinite loop

while(1) {

# check if there're changes
my $res_func_callback = &func_callback_processing();

# wait for 5 secondes
sleep(10);

foreach (@childs) {
	waitpid($_, 0);
}
foreach (@childs) {
	my $childpid = $_;
	# print "CHILD = $childpid \n";
}
@childs = ();
foreach (@childs) {
	my $childpid = $_;
	# print "AFTER EMPTY CHILD = $childpid \n";
}

if ($set_exit) { exit(0); }

}

Subroutine that daemonize this proccess

sub daemonize {
# chdir ‘/home/areski/bin’ or die “Can’t chdir to /: $!”;
chdir $_[0] or die “Can’t chdir to /: $!”;
open STDIN, ‘/dev/null’ or die “Can’t read /dev/null: $!”;
open STDOUT, ‘>>/dev/null’ or die “Can’t write to /dev/null: $!”;
open STDERR, ‘>>/dev/null’ or die “Can’t write to /dev/null: $!”;
defined(my $pid = fork) or die “Can’t fork: $!”;
exit if $pid;
setsid or die “Can’t start a new session: $!”;
umask 0;
}

This Subroutine check if there new emails to import

sub func_callback_processing {

my ($direlem,$LINES);
my ($dbh, $sth, $sth2 , $sth3, $res);
my $SQL="";
my $QUERY="";
my @server_manager;
my @row;
my @row2;
my ($line, $count, $id, $uniqueid, $status, $server_ip, $num_attempt, $channel, $exten, $context, $priority, $application, $data, $timeout, $callerid, $variable, $account, $async, $actionid, $id_server_group);
my $timestamp;
my $pid;
    system ("cd /etc/emailcallback && php email.php &");


$pass_nb++;
open(LOGFILE, ">>".$log_file)
	|| warn "Could not open >> $log_file.\n";

if ($dbtype eq "mysql")
{
	$dbh ||= DBI->connect("dbi:mysql:$dbname:$dbhost", "$login", "$pwd");		
}else{
	$dbh ||= DBI->connect("dbi:Pg:dbname=$dbname;host=$dbhost;port=$dbport", "$login", "$pwd");
	#$dbh ||= DBI->connect("dbi:Pg:dbname=$dbname;host=$dbhost;port=$dbport", "$login");
}

if (!$dbh) {
	$timestamp = &longfmt();
	print LOGFILE "[$timestamp]";
	print LOGFILE "ERR: Couldn't open connection: ".$DBI::errstr."\n";
	# die "ERR: Couldn't open connection: ".$DBI::errstr."\n";
	sleep (10);
	return 1;
}

if ($dbtype eq "mysql")
{
	$SQL = "SELECT aaa";
} else {
	$SQL = "SELECT aaa";
}

$timestamp = &longfmt();
print LOGFILE "$timestamp : TRY TO FIND PENDING CALLBACK";

$sth = $dbh->prepare($SQL);
$sth->execute();

print LOGFILE "\n ** Number of PENDING callbacks ".$sth->rows." ** and context= " . $row[12] . "\n";

while ( @row = $sth->fetchrow ) {
	
			
	## UPDATE THE CALLBACK BEFORE FORK
	$QUERY = "aaa";
	print LOGFILE $QUERY."\n";
	$sth3 = $dbh->prepare($QUERY);
	$sth3->execute();
	
	if ($pid = fork){
		# PARENT -> PID = CHILD PID
		print LOGFILE "\n WE ARE IN THE PARENT PROCESS -> CHILD PID = $pid -> RECORD ID = $id \n";
		push(@childs, $pid); # Add Child pid in the array
		sleep(1);
		next;
	} elsif ($pid == 0) {
		# PID = 0
		# CHILD
		print LOGFILE "\n WE ARE IN THE CHILD PROCESS\n";
	} else {
		$QUERY = "aaa";
		print LOGFILE $QUERY."\n";
		$sth3 = $dbh->prepare($QUERY);
		$sth3 -> execute();
		$sth3 -> finish;
		die "couldnt fork: $!\n";
	}
	
	# Save existing handler.
	 my $saved_warn_handler = $SIG{__WARN__};
	# Suppress warnings.
	$SIG{__WARN__} = sub {};
	
	# OPEN THE DB CONNECTION FOR THE CHILD
	my $dbh_child = $dbh->clone();
	
	# Restore saved handler.
	$SIG{__WARN__} = $saved_warn_handler;
	
	$dbh->{InactiveDestroy} = 1;
	undef $dbh;
	
	
	# my $dbh_child ||= DBI->connect("dbi:Pg:dbname=$dbname;host=$pghost;port=$pgport", "$login");
	
	@server_manager = &get_server($dbh_child, $id_server_group);		
	
	if (!(@server_manager)){
		print LOGFILE " server_manager not find for the id_server_group= $id_server_group \n";
		$QUERY = "aaa";
		print LOGFILE "\n".$QUERY."\n";
		$sth2 = $dbh_child->prepare($QUERY);

		$sth2 -> execute();
		$sth2 -> finish;
		#next;
		exit(0);
	}
	$timestamp = &longfmt();
	print LOGFILE "$timestamp\n";
	
	$astman->user($server_manager[3]);
	$astman->secret($server_manager[4]);
	$astman->host($server_manager[1]);		
	
	$status = $astman->connect;
	if ( defined $status ){
		$timestamp = &longfmt();
		print LOGFILE "[$timestamp]\n";
		print LOGFILE " ::>>> $status \n";

		
		# Action: Originate
		# Parameters: Channel, Exten, Context, Priority, Timeout, CallerID, Variable, Account, Application, Data
		my @res_manager = $astman->sendcommand( Action => 'Originate',
                                   Channel => $channel,
                                   Exten => $exten,
                                   Context => $context,
                                   Priority => $priority,
                                   Timeout => $timeout,
                                   CallerID => $callerid,
                                   Variable => $variable,
                                   Account => $account,
                                   Application => $application,
                                   Data => $data );
			#	Uniqueid => $uniqueid );
# added uniqueid
				
		$timestamp = &longfmt();
		print LOGFILE "[$timestamp]";
		print LOGFILE $res_manager[0] . ' - ' . $res_manager[1] . ' - ' . $res_manager[2] . ' - ' . $res_manager[3] . "\n";
		
		if (is_integer($num_attempt)){  $num_attempt++;
		}else{$num_attempt = 0;}
		
		if ( $res_manager[1] =~ /^Success/ ){
			print LOGFILE "[$timestamp]\n";
			print LOGFILE "\n ## SUCCESS CALLBACK -> ($channel, $exten, $context, $priority)\n";
			$QUERY = "aaa";
			print LOGFILE $QUERY;
			$sth2 = $dbh_child->prepare($QUERY);
			$sth2->execute();
			
			$QUERY = $SQL = "aaa";
			print LOGFILE "[$timestamp]\n";
			$sth2 = $dbh_child->prepare($QUERY);
			$sth2->execute();
			$sth2->finish;

} else {

			print LOGFILE "[$timestamp]\n";
			print LOGFILE "\n ## CALLBACK FAIL \n";
			$QUERY = "aaa";
			print LOGFILE $QUERY;			
			$sth2 = $dbh_child->prepare($QUERY);
			$sth2->execute();
			
			$QUERY = $SQL = "aaa";
			print LOGFILE "[$timestamp]\n";
			print LOGFILE $QUERY;
			$sth2 = $dbh_child->prepare($QUERY);
			$sth2->execute();
			$sth2->finish;
	
	}	
		$astman->disconnect;	
	}else{
		$timestamp = &longfmt();
		print LOGFILE "[$timestamp]\n";
		print LOGFILE "## ERROR Cannot connect to the asterisk manager! \n Please check the manager configuration...\n";
		
		## UPDATE THE CALLBACK AFTER FORK
		$QUERY = "aaa";
		print LOGFILE $QUERY."\n";
		$sth2 = $dbh_child->prepare($QUERY);
		$sth2->execute();
		$sth2->finish;
	}
	
	$timestamp = &longfmt();
	print LOGFILE "[$timestamp]\n";
	print LOGFILE " [END SELECT] \n";
	
	# CLOSE THE DB CONNECTION OF THE CHILD
	$dbh_child->disconnect;
	exit(0);
} # end while fetchrow


$sth->finish;
$dbh->disconnect;

# print LOGFILE "   ++++ \n";
close LOGFILE;

return 1;

}

Get the less recently used manager server

Return an array with the selected row

sub get_server{
my ($dbh) = shift(@);
my ($id_server_group) = shift(@
);
my $sth;
my $SQL="";
my @row;

if ($id_server_group > 0){
	$SQL = "aaa";
}else{
	$SQL = "aaa";
}
print LOGFILE " SQL =".$SQL." \n";

$sth = $dbh->prepare($SQL);
$sth->execute();
@row = $sth->fetchrow;

return @row;

}

Perl trim function to remove whitespace from the start and end of the string

sub trim($) {
my $string = shift;
$string =~ s/^\s+//;
$string =~ s/\s+$//;
return $string;
}

check if a variable is an integer

sub is_integer {
$_[0] =~ /^[±]?\d+$/
}

sub insert0 {
my ($date) = shift;
if ($date < 10) {
return “0$date”;
}
return $date;
}

Long format: Custom datestring for the logfile

sub longfmt {
my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$iddst) = localtime(time);
my $datestring;

$year += 1900;
$mon++;
$mon  = insert0($mon);
$mday = insert0($mday);
$min  = insert0($min);
$sec  = insert0($sec);
$datestring = "\n[$year-$mon-$mday $hour:$min:$sec]";
return($datestring);

}

sub INT_handler {
$set_exit = 1;
print LOGFILE “\n *** Recieved signal to exit *** \n”;
}