The Trying does not use the Contact. SIP responses are sent to the source IP address and port from which the INVITE came.
The Contact will be used, however, for subsequent in-dialog SIP requests. There is no ability to override that or use the From, because that’s not how SIP works.