DTMF cuts off audio

I’m using Asterisk 14.6.0. There is a problem when calling to the outside world via a SIP provider. If the callee presses a key on their keyboard, the audio from the callee to the caller is cut off but not the other way around. This happens both on incoming and outgoing calls.
The user for the SIP provider has dtmfmode=info and all the other Asterisk users have dtmfmode=rfc2833. The Asterisk box is behind a NAT