Secondary IP not working in asterisk

Hi all I am using asterisk for sip calls .

On my asterisk machine i have nic .

eth0 - with ip 10.120.49.33
eth0:0 - with ip 10.120.49.50

i want to use both the ips above for my load balancer test where eth0 and eth0:0 will act as 2 server .

But the problem is when sip request goes on eth0:0 ip the reply is coming back from eth0 ip . I cant use bindaddr in sip.conf as it will bind the ip to only one server and i wont be able to check the load balancer .

Any suggestion on above scenario ?