Build · Advanced · One to two weeks
Build an AI voice agent with open-source software
A phone or web voice assistant that listens, thinks and talks back, on your own hardware.
Instead of paying for Vapi, Otter.ai, ChatGPT Plus or ElevenLabs.
What you'll build
A voice assistant you can call or talk to in the browser: it hears what you say, works out a reply with a language model, and answers in a natural voice. It's the kind of agent services like Vapi and ElevenLabs charge per minute for. Built from open-source parts, it can run on your own machine with no per-minute bill, which makes it practical for a receptionist that books appointments, a practice partner for a language, or a hands-free helper in a workshop.
Blueprint
LiveKit Agents manages the live call. Each turn, whisper.cpp turns speech into text, Ollama writes the reply and GPT-SoVITS speaks it.
- You Your voice over a call or browser LiveKit Agents
- LiveKit Agents Audio of what you said whisper.cpp
- whisper.cpp The words as text Ollama
- Ollama The reply as text GPT-SoVITS
- GPT-SoVITS The reply as speech LiveKit Agents
The repos
- 01 · Run the call LiveKit Agents
Python framework for real-time voice agents as server-side participants in LiveKit WebRTC rooms, with STT/LLM/TTS plugins.
- Setup
- Easy; pip install livekit-agents with provider extras, then set LiveKit URL and API key/secret source
- Hardware
- Framework requirements not stated; the turn-detector model runs on CPU and needs under 500 MB RAM source
- Paid extras
- README example uses LiveKit Inference (part of LiveKit Cloud, billed by usage) or provider keys for Deepgram, OpenAI and Cartesia. LiveKit server can be self-hosted and docs cover a local Ollama LLM, but no local STT or TTS plugin is documented in the sources checked
- 02 · Hear (speech to text) whisper.cpp
Dependency-free C/C++ port of OpenAI's Whisper speech recognition, running on CPU or GPU across desktop, mobile and the browser.
- Setup
- Hard; build from source with CMake, then download a ggml model source
- Hardware
- CPU fine; memory from ~273 MB (tiny) to ~3.9 GB (large) source
- Paid extras
- None required
- 03 · Think Ollama
Local model runner that downloads and serves open LLMs (Gemma, Qwen, DeepSeek, gpt-oss and more) with a CLI and local API.
- Setup
- Easy; one-line install script or installer, then ollama run source
- Hardware
- Apple M-series or x86 (CPU only) on Mac; 4GB for Windows binary; RAM not stated source
- Paid extras
- None for local use; Ollama Cloud plans optional (Pro $20/month)
- 04 · Speak (text to speech) GPT-SoVITS
Few-shot voice cloning and TTS WebUI; 5-second zero-shot or 1-minute fine-tune, five languages, MIT code and weights.
- Setup
- Easy on Windows (integrated package); conda plus install script on Linux and macOS source
- Hardware
- No minimum RAM/VRAM stated. Stated speed (v2 ProPlus): RTF 0.028 on RTX 4060 Ti, 0.014 on RTX 4090, 0.526 on an M4 CPU. Tested on CUDA, Apple silicon and CPU. source
- Paid extras
- None required
Step by step
Install commands change, so each step links to the repo's own docs for the exact commands.
-
Get each piece working alone
Before wiring anything, check each part on its own: transcribe a recording with whisper.cpp, chat with a model in Ollama, and generate a sentence of speech with GPT-SoVITS. Note how long each takes on your machine.
-
Set up LiveKit Agents
Install LiveKit Agents. It needs a LiveKit server: run the open-source server yourself rather than the paid cloud used in the README example.
Uses LiveKit Agents · its docs
-
Plug in your own speech and model
In your agent, replace the default speech-to-text, language model and text-to-speech with your local whisper.cpp, Ollama and GPT-SoVITS endpoints.
Uses LiveKit Agents · its docs
-
Give it a job and limits
Write a short system prompt saying what the agent does and must never do (no payments, no medical advice). Test with interruptions, background noise and people talking over it.
Watch out
LiveKit's README example uses its paid LiveKit Cloud; self-host the server instead. Its turn-detector models have their own LiveKit licence. A GPU helps a lot.
Take it further
- Connect a phone number through a SIP provider so people can ring it.
- Let it book appointments by calling your Cal.diy API.
- Clone your own voice with GPT-SoVITS, with your consent only.
Questions
Can this run on a laptop?
Yes for testing, but replies will be slow without a GPU. For real calls, a machine with a decent graphics card makes the difference.
Is voice cloning legal?
Cloning your own voice is fine. Cloning anyone else's needs their clear permission, and some places have laws about it.
How is this different from Vapi?
Vapi hosts the whole pipeline and bills per minute. Here you run each part, so there's no per-minute bill, but you look after it.
Every repo here is open source, still maintained and checked against its own documentation. How we check →