I managed to solve it.
The issue occurred when I tried to use the Opus or G722 codecs. I switched to the ulaw codec, and the problem stopped occurring.
I don’t know how to explain why it sometimes worked when using Opus or G722, and sometimes it caused issues.
I would be grateful if someone could provide an explanation for it.