SIP to SIP garbled at beginning of call

Have snom 190 phones across VPN going through a rh 8.0 server (3.0 Ghz Semtron w/512 RAM), very low traffic on VPN and server. Most SIP to SIP calls have a 10-15 sec garbled transmission, using ulaw or g729. Server is dedicated and is in runlevel 3. After 10-15 seconds call is fine. All zaptel calls are fine using SIP phones. Only SIP to SIP is issue. Any ideas would greatly appreciated.