RTP streams between H323 and SIP

I have a h323 phone (OpenPhone) and SIP Phone. I make a call between those 2 phones.
I set the “canreinvite=yes” for SIP phone.The RTP stream for the h323 phone still go through Asterisk.
Anyone can give me a hand to fix this problem?
I want the RTP streams to go from endpoint to endpointwithout going through Asterisk.