I have always added jitter buffers in Asterisk where I can and its never really been a problem until Asterisk 13 where the /j Local Channel option adds quite a bit of delay.
After reevaluating the architecture of the system, I really cant see the need for implementing a jitter buffer in Asterisk unless Asterisk terminates the media e.g. conference, announcements etc. For all other call scenarios, whether native or non native bridging, the end devices would provide the jitter buffering anyway.
I have therefore only added the JITTERBUFFER(adaptive)=default function in the modules that terminate the media.
Does this seem a reasonable architectural decision? Have I missed something?