Your codecs are ncompatible. One of the conditions for direct media is that compatible codecs must have been negotiated.
canreinvite is deprecated in favour of directmedia.
; is a comment delimiter. You must list every possible peer separately or use allowguest. In fact you may be using allowguest already, but not have directmedia in the general section.
One of your clients must actually be a server. They are peers not clients in SIP terms.
You haven’t specified a version of Asterisk, but there is a bug in older versions that prevents multi-stage direct meida re-invites.
Settings in features.conf and extensions.conf can conflict with directmedia, but you haven’t included those files.