The parts of the dialplan that can be abused should be in contexts that are neither default nor associated with a peer that can be accessed externally.
For SIP, allowguest should be disabled, unless you need it, and otherwise the default context should be as above.
For SIP, peers should be authenticated using strong passwords, and, ideally, unpredictable device names.
For SIP, peers that can only be authenticated by user and password, should have address range restrictions.
For VoIP in general, tools like fail2ban and be used to dynamically add firewall rules after a failed attack.