For chan_sip, it is struct sip_pvt that contains the information used. There will be something similar for chan_pjsip, however I assume that it is decomposed in a similar way to the way in which the conf file is structured.
However, Asterisk doesn’t store phone numbers, in this context. It identifies peers by IP address or user name. The user name doesn’t have to be a phone number, and some of the security advice is that it should not be.
extensions.conf is where phone numbers are identified for outgoing calls, and that is processed by pbx.c. There need be no phone number associated with an incoming call.
For outgoing calls, the peer may also use user name to identify a local endpoint, in which case IP address and port will not uniquely identify an endpoint (which may or may not correspond to a phone number).
The above is very much a simplification, only applies to SIP, and is really outside the scope of this forum, as it is a developer, not an end user question. There are mailing lists and IRC channels for developers.
I’d suggest that, if you cannot answer the question by looking at the code, you don’t know enough about Asterisk to be trying to answer it.
For many channel technologies, the concept of an IP address is completely foreign.