I suppose it would also depend on the method of RTP delivery.
If you do not pass all RTP packets through the Asterisk server, but rather directly from peer to peer, (SIP phone to SIP phone) the system could support a vast number of callers. Using re-invites would allow this.
This would limit the actual RTP traffic load on the Asterisk server to any calls on hold (for music on hold), calls that go to voicemail, or calls leaving the local system to traverse the PSTN.
Even then, the system could be broken up into multiple servers. The voicemail server, for example, could be a server all it’s own. The same could be true for a PSTN gateway.
About the only thing Asterisk would have to do then would be to maintain the registrations and process the requests for any service based on the extensions.conf file’s configuration. SER could help with the registrations (in large systems), leaving Asterisk to just process the invite/re-invite requests. Basically, Asterisk as sort of a “DNS” for VOIP traffic.
This configuration would be fairly easy for any server to maintain…