Multiple Gateways - Only one works

I have an Asterisk server on a network with 2 internet gateways using a 3rd party sip provider for calls. For some reason, I can make and receive calls using one of the gateways, but if I change the default gateway to the other, I can no longer receive calls though I can still make them. Any one else have this issue?