Newsletter image

Subscribe to the Newsletter

Join 10k+ people to get notified about new posts, news and tips.

Do not worry we don't spam!

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use

Search

GDPR Compliance

We use cookies to ensure you get the best experience on our website. By continuing to use our site, you accept our use of cookies, Privacy Policy, and Terms of Service.

NVIDIA - Conversational AI, Text-to-Speech, Speech Recognition, Edge AI

Build a Local Voice Assistant on Nemotron 3.5 Bolt E4B

NVIDIA built Nemotron 3.5 Bolt E4B for local voice assistants, but it cannot hear or speak. The 4 GB is the LLM alone: adding Whisper and Piper takes a real loop to 4.4 GB, and the DFlash drafter costs more memory than the ears do.

License OpenMDW 1.1
License OpenMDW 1.1
TL;DR
  • Bolt E4B is text only. A voice assistant is three processes, and NVIDIA's 4 GB budget covers just the middle one.
  • The DFlash drafter (737 MB) costs more memory than whisper.cpp base.en (388 MB), so latency and transcription accuracy compete for the same space.
  • Bolt tops the Brevity Benchmark at 82.84, which matters more in a voice loop than tool calling, where it trails Qwen3.5 4B.
System Requirements
RAMAbout 4.4 GB resident for a minimal loop (base.en ears), about 5.2 GB with accurate ears and the drafter
GPUAny CPU runs it; NVIDIA reports sub-second replies on an RTX 5070 and about 6 s to first token on a Jetson Orin Nano 8GB
VRAMBolt 3.5 to 5.0 GB by cache depth, DFlash drafter 737 MB, whisper.cpp small.en 852 MB, Piper 60 MB

NVIDIA built Nemotron 3.5 Bolt E4B for local voice assistants and said so on the model card: "fast local voice/device assistants, interactive personas, and embedded/edge interfaces." It cannot hear and it cannot speak. It is a text model, and the 4 GB in its name is the budget for that text model alone. A voice assistant is three processes, not one, and the other two have to come out of the same RAM.

That is not a complaint, because NVIDIA designed it this way on purpose and the model card says so: it targets "low-memory, low-TTFT operation" so an agent can "respond quickly while preserving headroom for applications or adjacent model pipelines." The adjacent model pipeline is your speech stack. We covered the model itself, its Gemma 4 provenance and its licence in the Bolt E4B spotlight; this is the build, and the arithmetic nobody publishes with the headline.

Three processes, not one

A local voice loop is speech to text, then text to text, then text to speech. Bolt is the middle box only. The two on either side are separate models with separate weights, and on a device with a fixed memory budget they compete with the thing you actually came for.

The standard open-source answer for the outer two is whisper.cpp for the ears and Piper for the voice. Both are small, both run on CPU, and both are MIT licensed. That makes the arithmetic tractable, which is the first useful thing to do before you buy a single board computer on the strength of a headline number.

The real memory budget

Every figure below is from the publishers' own documentation: NVIDIA's model card, the whisper.cpp README's memory table, and the file sizes in the Piper voice repository. Nothing here was measured by us.

ComponentRoleOn diskResident
Nemotron 3.5 Bolt E4B, Q4_K_MThe assistant6.18 GB3.5 to 5.0 GB
DFlash drafter, BF16 GGUFOptional, cuts latency736.81 MBabout 0.74 GB
whisper.cpp base.enEars, budget142 MiBabout 388 MB
whisper.cpp small.enEars, accurate466 MiBabout 852 MB
Piper en_US-lessac-mediumVoice60.3 MiBabout 60 MB

So a minimal loop is roughly 3.95 GB, using base.en and no drafter. A comfortable one with accurate transcription and speculative decoding is about 5.2 GB. Neither is 4 GB, and the gap is not rounding: it is a third process you did not budget for.

Two numbers in that table deserve a second look. The Q4_K_M file is 6.18 GB on disk but only 3.5 GB resident at a shallow cache, because a GGUF is memory mapped and the weights that matter are the ones you touch. And the resident figure climbs to 5.0 GB as the KV cache deepens, which in a voice assistant means a long conversation quietly eats the headroom your speech stack is living in.

The drafter costs more than the ears

NVIDIA ships a companion DFlash drafter for speculative decoding, and for a voice assistant it is the most interesting file in the whole release. It is five Qwen3-style decoder layers, 360,480,000 parameters, in a single BF16 GGUF of 736,812,096 bytes. It carries no token embeddings and no output head, borrowing the target model's at runtime, which is why it is as small as it is.

It proposes up to seven candidate tokens per step and the target model verifies them. For conversation that is close to the ideal case: short, predictable, high-agreement replies are exactly what a drafter guesses well. NVIDIA's own stated purpose is "lower-latency local serving."

Here is the trade nobody puts on a slide. The drafter is 737 MB, while Whisper small.en needs about 852 MB and base.en about 388 MB. On a tight 8 GB device you are choosing between hearing people accurately and answering them quickly, and the two options cost almost the same. That is the real design decision here, and it only becomes visible once you stop treating the LLM as the whole system.

Brevity is the voice benchmark nobody quotes

Benchmark tables for small models lead with tool calling and reasoning. For a voice assistant the ranking is different, because every token the model emits is a token the TTS has to render and the user has to sit through. A model that answers correctly in forty words beats one that answers correctly in two hundred, and no reasoning benchmark captures that.

These are NVIDIA's published figures at Q4_K_M, the quantization you would actually deploy.

BenchmarkWhy it matters hereBolt E4BGemma 4 E4BQwen3.5 4B
Brevity BenchmarkShorter answers, shorter waits82.8478.4975.17
MultiChallengeHolding a multi-turn conversation55.6039.3652.23
RoleBenchStaying in character as a persona71.2160.5152.35
IHEvalObeying the system prompt over the user82.4066.6162.87
BFCL-v3Tool calling, if it must act67.3165.0370.56
Tau2-BenchAgentic task completion52.3238.7261.74

The pattern is clean and it decides your architecture. Bolt wins the conversational axes by wide margins and loses the agentic ones to Qwen3.5 4B. If your assistant answers questions, holds a persona and controls a few local devices, Bolt is the better pick and brevity alone justifies it. If it is really an agent that happens to have a microphone, the tool-calling gap is the number that will hurt, and our spotlight argued exactly that case.

Where the latency actually goes

Time to first token is the figure NVIDIA optimised and the one everyone quotes, but it is only the middle third of what the user experiences. The perceived delay is the sum of four things: finishing the transcription after the user stops talking, the model's time to first token, generating enough tokens to start speaking, and the TTS producing its first audio chunk.

That last point is the one people get wrong. You do not wait for the full reply before speaking. Stream the model's output into Piper sentence by sentence, start playback on the first complete sentence, and the user hears a response while the model is still writing. Done properly, the model's total generation time stops mattering and only its time to first sentence does, which is where a brevity-tuned model and a drafter both pay off twice.

For orientation on the middle third, NVIDIA reports sub-second responses on an RTX 5070 and roughly six seconds to first token on a Jetson Orin Nano 8GB. Those are vendor figures at batch size 1, not ours, and the Jetson number is the one to plan around if you are building something that lives on a shelf. Six seconds is past the point where a person assumes the device did not hear them, which is an argument for a cheap acknowledgement sound the moment transcription ends, and for watching what the newer edge inference silicon does to that figure.

Pulling the three pieces

Everything below runs on CPU if it has to, and all three projects ship prebuilt. The model is the only large download.

mkdir -p ~/voice/models && cd ~/voice/models

# 1. the assistant (6.18 GB)
curl -L -o bolt-e4b-q4km.gguf \
  https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB/resolve/main/Nemotron-3.5-Bolt-E4B-Q4_K_M.gguf

# 2. the ears (466 MiB; swap small.en for base.en to save ~460 MB resident)
curl -L -o ggml-small.en.bin \
  https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-small.en.bin

# 3. the voice (60 MiB, plus its json config)
curl -L -O https://huggingface.co/rhasspy/piper-voices/resolve/main/en/en_US/lessac/medium/en_US-lessac-medium.onnx
curl -L -O https://huggingface.co/rhasspy/piper-voices/resolve/main/en/en_US/lessac/medium/en_US-lessac-medium.onnx.json

du -sh ~/voice/models

Run the model as a server rather than a one-shot process, so the weights stay resident between utterances. Loading 6 GB from disk on every question is the single biggest self-inflicted latency problem in a hobby voice assistant. If your box has no CUDA, a GGUF this size is still workable on Vulkan or plain CPU, and the Colibri 2.1 write-up covers what each backend costs you.

llama-server -m ~/voice/models/bolt-e4b-q4km.gguf \
  --host 127.0.0.1 --port 8080 \
  -c 8192 \
  --temp 1.0 --top-p 0.95

Note the sampling values. NVIDIA recommends temperature 1.0 and top_p 0.95 for this model in both thinking modes, which is higher than the temperature most people reach for by habit. The context is set to 8192 rather than the model's 64K maximum on purpose: a deeper cache is resident memory you are taking away from Whisper.

Turning on the drafter

Speculative decoding in llama.cpp needs the draft model passed alongside the target. The flags are -md for the draft model and --draft-max for how many tokens it may propose, and seven matches what the DFlash checkpoint is built to offer.

curl -L -o dflash-draft.gguf \
  https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB-DFlash/resolve/main/Nemotron-3.5-Bolt-E4B-DFlash-BF16.gguf

llama-server -m ~/voice/models/bolt-e4b-q4km.gguf \
  -md ~/voice/models/dflash-draft.gguf \
  --draft-max 7 --draft-min 1 \
  --host 127.0.0.1 --port 8080 \
  -c 8192 --temp 1.0 --top-p 0.95

Speculative decoding speeds up generation, not prefill, so it shortens the gap between words rather than the gap before the first one. In a streamed voice loop that still helps, because it gets you to the end of the first sentence sooner. Budget the extra 737 MB before you enable it, and drop to base.en if that is what it takes.

Choosing your ears

The Whisper size you pick is the second real decision, and it is a different trade from the drafter. Transcription errors do not degrade gracefully in a voice assistant: a misheard word does not produce a slightly worse answer, it produces a confident answer to a question nobody asked.

ModelDiskResidentUse it when
tiny.en75 MiBabout 273 MBWake words and short fixed commands only
base.en142 MiBabout 388 MBQuiet room, close microphone, tight budget
small.en466 MiBabout 852 MBThe sensible default for open speech
medium1.5 GiBabout 2.1 GBOnly if you have the headroom to spare

The English-only variants are the right call for an English assistant: same size, better accuracy than the multilingual weights at that size. Going above small.en on an 8 GB device means giving up either the drafter or the conversation depth, and medium at 2.1 GB resident competes directly with the model itself.

Three components, three licences

The stack is permissive but not uniformly so, and the model is the restrictive part. whisper.cpp and Piper are both MIT. Bolt E4B is under OpenMDW 1.1, which is narrower than the Apache 2.0 of the Gemma 4 model it was post-trained from, and that narrowing is the thing we flagged in the spotlight. Read it before you ship a product rather than after, because the licence on the middle box governs the assistant even though the outer two are free of conditions.

What will bite you

The KV cache grows with conversation length, so an assistant that is comfortable at turn three can be swapping at turn thirty. Cap the context deliberately and summarise or drop old turns rather than letting it grow into the memory your speech stack needs.

Bolt's chat template rejects stray tool responses: a tool role message that does not immediately follow an assistant message with tool_calls is refused. NVIDIA documents an asynchronous pattern where you acknowledge the call and report the result as pending. If your assistant controls anything that takes real time, such as a kettle or a garage door, you will hit this on day one.

The 64K context is half of Gemma 4 E4B's 128K. For a voice assistant that is irrelevant, because nobody dictates 64,000 tokens, but it matters if you intended to stuff a manual into the prompt and ask questions about it.

And the obvious one: none of this is a wake word. Always-on listening is a fourth component again, and the usual answer is a small dedicated detector rather than running Whisper continuously, which would burn the CPU budget you need for everything else.

Who should build this

Anyone who wants an assistant that works with the network unplugged. This is the strongest reason, and it is not a performance argument. Every word stays on the device, which for a kitchen or a children's room is the whole point, and the quality is now good enough that the trade is no longer painful.

People shipping a persona rather than an agent. Bolt's RoleBench and IHEval numbers are well ahead of its size class, and a device character that stays in voice and obeys its system prompt is exactly what those measure. If you want something that reliably calls tools, look hard at the BFCL gap first.

Not people on 8 GB who want everything. The accurate ears, the drafter and a deep conversation do not all fit at once. Pick two, or move up to a machine with 16 GB and stop doing arithmetic. An alternative worth knowing about is a single model that does the whole chain itself: we covered one in the PersonaPlex-7B spotlight, and collapsing ASR, LLM and TTS into one set of weights is a genuinely different answer to this problem.

Sources and further reading

The ten-minute version: pull base.en, Piper and the model, run llama-server with -c 8192, and add up the resident memory on your own box before you decide whether the drafter or the better ears go in the remaining space. That one sum tells you more about whether this will work on your hardware than any benchmark table, including the ones above.

Not independently tested. Every number here is from NVIDIA's model card, the whisper.cpp README, the Piper voice repository or the llama.cpp source, and no part of this stack was run on our hardware.
Date checked: 2026-11-19

Prev Article
GLM-5.3-Flash on Two DGX Sparks: The TensorFold Recipe
Next Article
How to create Logos with Midjourney

Related to this topic: