How to decode Opus Codec

I have found where all RTP packets are captured on asterisk source code on the file /usr/src/asterisk-18.17.0/res/res_rtp_asterisk.c

On the method ast_rtp_read next to the methot that logs rtp message this is the buffer &rtp->rawdata[AST_FRIENDLY_OFFSET] that contains the payload. I have no idea why asterisk skips the first bytes of the buffer with AST_FRIENDLY_OFFSET but if I skip those number of bytes I get the payload. I also know the size from the variable ‘res’. If I analyze/capture those packets I am able to play and audio when using ulaw format. I just have to remove the first 12 bytes of each packet. In other words I remove the RTP header (12 bytes long) and when I join all the packets together I end up with a raw ulaw file. I can then convert that file to a wav with this command:

ffmpeg -f mulaw -ar 8000 -i input.sln -acodec pcm_s16le -ar 8000 -ac 1 output.wav

Anyways why is it that when I do the same technique when the phones are using the Opus codec I am not able to listen to that audio? The first difference I encounter when using opus codec is that each packet varies in size. When I remove the RTP header from all those RTP packets I am not able to conver that file to an opus file. Can someone explain me what am I doing wrong? I have also done my research and I only grab the RTP packets that have the same id (the last 4 bytes). I also sort them in order by looking at the sequence number. Since I am making local tests they come in order. But even after I filter for all the packets that have the same RTP id and also make sure they are in the correct order I am not able to convert it to a valid audio file.