When I call using xlite with asterisk pbx, I hear the other person’s voice but he didn’t hear my voice i.e. there is no sound at other end.
X-Lite works with Asterisk, so you obviously haven’t told us the important piece of information.
One way audio is typically due to firewall or NAT problems.
Please Asterisk Support for support questions.
Do we have to solve the firewall and NAT issue at caller machine or at callee machine.
Also how can we solve the firewall or NAT issue?
Yes.
By understanding what the firewall or NAT is doing wrong in your case.