Asterisk doesn't reply to INVITE


Server has the following configuration:
4x Xeon 1.59GHz
RHEL 5.3 2.6.18-128.el5PAE

We do voicemail with Asterisk for 20k users. Sometimes, when Asterisk receive an INVITE (3 times according to the timer) he doesn’t reply to them while he is still alive and responsive for other requests.
Some one would have an idea on limitation with a module implies in the answer process?
Thanks in advance for any advice.