Is restarting Asterisk for security reasons a big deal?

I am sorry if I am posting on the wrong forum. I am trying to assess just how problematic is it for people to keep up with security updates for Asterisk, and if restarting the server is problematic.
We can patch asterisk for security vulnerabilities on the fly, without restart. Yet, I am not convinced it is something people running Asterisk want or interested in.
Any feedback would be greatly appreciated.