Running Asterisk on an machine with X enabled? How? Why Not?

I have seen several articles about running a small asterisk installation on a virtual machine under either Linux or Windows. I also read many dire warnings about never running X-windows on an asterisk server.
This seems to be inconsistent. Surely if I can run an X server and an asterisk server on two virtual machines, the same hardware should be able to run the same software on a physical machine with a single operating system? My requirement is for a system which is only handling two lines.
So I tried a quick test by apt-get install asterisk on an Ubuntu desktop machine. Sure enough, the voice quality sometimes sputters as if packets were being dropped. I noticed that asterisk is running at normal interactive priority (nice 0). When I manually pushed up the priority of asterisk (I tried nice -8) then the call quality seemed to clear up.
Is there not some way to convince asterisk that it should always run at a real-time priority level?

Vernon Cole
an old programmer finally learning asterisk