Fully self-hosted voice AI — the whole STT→LLM→TTS stack on your own Nvidia GPU (works with Asterisk)

Asterisk AI
1

Hi all,

I built and just open-sourced a self-hosted, real-time voice AI stack (STT → LLM → TTS) that answers SIP calls and holds a natural conversation, running entirely on your own box — no cloud, no per-minute billing, audio never leaves your infra. It’s free, no signup, and ships with the Asterisk bridge. Full disclosure: it’s my project and I’d love your feedback.

Asterisk side: the bridge uses ARI + external-media (RTP, g711 μ-law) via a Stasis app. Validated with a live PJSIP call. Barge-in works.

 same => n,Stasis(asterisk_to_openai_rt)

Drop-in: the server speaks the OpenAI Realtime API protocol. If you already integrated cloud OpenAI Realtime with Asterisk, just point it at your own box — change the URL, done.

Concurrency (sub-second TTFA, from the docs):

  • RTX 4090 → runs the whole stack — validated: fits in ~21.5/24 GB. Tight VRAM → best for dev / small setups.
  • DGX Spark desktop (~$4K) → ~20 concurrent calls (~200 lines with 10:1 overbooking)
  • H100 / H200 → ~75 concurrent · scales ~linearly with more GPUs

Why the NVIDIA GPU is a feature, not a tax: local inference = ~0.1 s first-audio, consistent (no cloud jitter/rate-limits); one-time card vs per-minute forever; audio stays in your rack; no kill-switch if a provider deprecates a model or cuts access. Runs on one FP8-capable GPU (Ada/Hopper/Blackwell). The 8B model needs only ~12 GB VRAM — validated on a single RTX 4090 (whole stack fits in ~21.5/24 GB, live voice TTFA 0.17–0.59 s), so a consumer card is plenty for dev/small setups.

Stack: NVIDIA NeMo (STT/TTS) + Qwen3-8B FP8 on vLLM. Bilingual ES/EN.

Repo: GitHub - infinitocloud/nemo-rt-community: Real-time, sub-second, bilingual (ES/EN) voice AI that runs entirely on your own NVIDIA GPU. One command, Apache-2.0. Truly on-premise — no cloud, no per-minute fees, no kill-switch. · GitHub (Asterisk bridge under integrations/asterisk/)

If you run it, I’d love concurrency/latency reports on other GPUs — there’s a [hardware] issue template in the repo for that. Happy to answer anything on the ARI/external-media side right here in the thread.