Why would I want to disable the IAX jitter buffer?

Is there ever a disadvantage to using the jitter buffer?

I think it adds a tiny bit of latency when turned on.