Join 10k+ people to get notified about new posts, news and tips.
Do not worry we don't spam!
By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use
Search
GDPR Compliance
We use cookies to ensure you get the best experience on our website. By continuing to use our site, you accept our use of cookies, Privacy Policy, and Terms of Service.
SingularityByte.com values the privacy of our users. Therefore, this privacy policy explains in detail how we use and protect the information we collect when you visit our website.. Read this privacy policy completely. Please refrain from visiting the site if the terms outlined below are not satisfactory to you. We reserve the right to change this policy at any time and will list these changes in the updates section of the policy. By reading this notice and visiting the site, you agree that you understand that customers will not be personally notified when this policy changes. Therefore, we advise our customers to frequently review our privacy policy so that they remain aware of its updates. By using the site, you accept that the posted policy and all its changes apply to your interaction with SingularityByte.com.
Information Collected by SingularityByte.com
Personal information may be collected by this site in many ways. This information includes:
Personal identifying information like your name, address, email, phone number, age, gender, and other personal data
Server data related to the IP address you used to visit our website, which includes your address, browser, OS, access time, and site activity.
Financial information related to your orders including your payment method and identifying payment information. We rarely store financial information collected on our site for transaction purposes. That information gets sent directly to our payment processor.
Social network data including Facebook permissions and user information from other networks, provided you log onto our site using one of these media sites.
Mobile device information such as your device ID, model, and location, if you use our site by accessing trough our website.
How We Use This Information
Our website uses information collected to: • Manage your account information • Customize ads • Deliver promotions • Email your account confirmation • Manage purchases and payments • Increase site efficiency • Notify you of updates • Offer new products • Monitor and prevent theft • Request your customer feedback • Resolve account disputes • Respond to your service requests
Information Disclosure
Normally, your information stays on our site. However, below we have listed the situations that may require us to share the information we collect from you: • When required by law, such as for fraud protection • With our third-party providers for payment processing and hosting • With your consent for marketing purposes • When you post comments on the site • To our advertisers, affiliates, and partners • If this site goes bankrupt and data must be transferred
Cookies, Trackers, and Online Ads
We may use cookies, trackers, web beacons, and other technology to customize our website to improve your experience. We may customize the site using this information. These trackers do not have access to your personal information and can be removed from your browser options. In addition, third-party software provides ads for our site for marketing campaigns. These programs have access to tracking technology to optimize your ad experience. For more information about these ads, visit [link to the privacy policies of affiliate advertisers]. Website analytics such as through Google Analytics may also be used to track users and remarket our website. We do not give these vendors access to your personal information.
Other Sites
Our website may contain links to third-party websites in the form of policies, ads, and other non-affiliated links. Once you leave our site, we are no longer responsible for how your information is collected and disclosed. Please refer to the privacy policies of those third-party sites for more information.
Information Security
We take technical and administrative precautions to protect your data, but we cannot guarantee its safety against all types of fraud or misuse. If you provide personal information, we cannot verify its total security against all types of interception.
Do-Not-Track
Some browsers offer Do-Not-Track settings to prevent any information from being distributed. Since these settings have not been legally established as standard practice, we do acknowledge these settings.
Additional Options
At any time, you may opt to review or change your account settings, including contact information. If you wish to delete your account, you may do so to remove most of your information, however, some identifying information will be retained to prevent fraud. You may also opt-out of emails and other correspondences from our site at any time.
Microsoft Clarity
We partner with Microsoft Clarity and Microsoft Advertising to capture how you use and interact with our website through behavioral metrics, heatmaps, and session replay to improve and market our products/services. Website usage data is captured using first and third-party cookies and other tracking technologies to determine the popularity of products/services and online activity. Additionally, we use this information for site optimization, fraud/security purposes, and advertising. For more information about how Microsoft collects and uses your data, visit the Microsoft Privacy Statement.
Contact Us
If you have questions or concerns about this privacy policy, please feel free to contact us at: desk@SingularityByte.com
Build a Local Voice Assistant on Nemotron 3.5 Bolt E4B
NVIDIA built Nemotron 3.5 Bolt E4B for local voice assistants, but it cannot hear or speak. The 4 GB is the LLM alone: adding Whisper and Piper takes a real loop to 4.4 GB, and the DFlash drafter costs more memory than the ears do.
NVIDIA built Nemotron 3.5 Bolt E4B for local voice assistants and said so on the model card: "fast local voice/device assistants, interactive personas, and embedded/edge interfaces." It cannot hear and it cannot speak. It is a text model, and the 4 GB in its name is the budget for that text model alone. A voice assistant is three processes, not one, and the other two have to come out of the same RAM.
That is not a complaint, because NVIDIA designed it this way on purpose and the model card says so: it targets "low-memory, low-TTFT operation" so an agent can "respond quickly while preserving headroom for applications or adjacent model pipelines." The adjacent model pipeline is your speech stack. We covered the model itself, its Gemma 4 provenance and its licence in the Bolt E4B spotlight; this is the build, and the arithmetic nobody publishes with the headline.
Three processes, not one
A local voice loop is speech to text, then text to text, then text to speech. Bolt is the middle box only. The two on either side are separate models with separate weights, and on a device with a fixed memory budget they compete with the thing you actually came for.
The standard open-source answer for the outer two is whisper.cpp for the ears and Piper for the voice. Both are small, both run on CPU, and both are MIT licensed. That makes the arithmetic tractable, which is the first useful thing to do before you buy a single board computer on the strength of a headline number.
The real memory budget
Every figure below is from the publishers' own documentation: NVIDIA's model card, the whisper.cpp README's memory table, and the file sizes in the Piper voice repository. Nothing here was measured by us.
Component
Role
On disk
Resident
Nemotron 3.5 Bolt E4B, Q4_K_M
The assistant
6.18 GB
3.5 to 5.0 GB
DFlash drafter, BF16 GGUF
Optional, cuts latency
736.81 MB
about 0.74 GB
whisper.cpp base.en
Ears, budget
142 MiB
about 388 MB
whisper.cpp small.en
Ears, accurate
466 MiB
about 852 MB
Piper en_US-lessac-medium
Voice
60.3 MiB
about 60 MB
So a minimal loop is roughly 3.95 GB, using base.en and no drafter. A comfortable one with accurate transcription and speculative decoding is about 5.2 GB. Neither is 4 GB, and the gap is not rounding: it is a third process you did not budget for.
Two numbers in that table deserve a second look. The Q4_K_M file is 6.18 GB on disk but only 3.5 GB resident at a shallow cache, because a GGUF is memory mapped and the weights that matter are the ones you touch. And the resident figure climbs to 5.0 GB as the KV cache deepens, which in a voice assistant means a long conversation quietly eats the headroom your speech stack is living in.
The drafter costs more than the ears
NVIDIA ships a companion DFlash drafter for speculative decoding, and for a voice assistant it is the most interesting file in the whole release. It is five Qwen3-style decoder layers, 360,480,000 parameters, in a single BF16 GGUF of 736,812,096 bytes. It carries no token embeddings and no output head, borrowing the target model's at runtime, which is why it is as small as it is.
It proposes up to seven candidate tokens per step and the target model verifies them. For conversation that is close to the ideal case: short, predictable, high-agreement replies are exactly what a drafter guesses well. NVIDIA's own stated purpose is "lower-latency local serving."
Here is the trade nobody puts on a slide. The drafter is 737 MB, while Whisper small.en needs about 852 MB and base.en about 388 MB. On a tight 8 GB device you are choosing between hearing people accurately and answering them quickly, and the two options cost almost the same. That is the real design decision here, and it only becomes visible once you stop treating the LLM as the whole system.
Brevity is the voice benchmark nobody quotes
Benchmark tables for small models lead with tool calling and reasoning. For a voice assistant the ranking is different, because every token the model emits is a token the TTS has to render and the user has to sit through. A model that answers correctly in forty words beats one that answers correctly in two hundred, and no reasoning benchmark captures that.
These are NVIDIA's published figures at Q4_K_M, the quantization you would actually deploy.
Benchmark
Why it matters here
Bolt E4B
Gemma 4 E4B
Qwen3.5 4B
Brevity Benchmark
Shorter answers, shorter waits
82.84
78.49
75.17
MultiChallenge
Holding a multi-turn conversation
55.60
39.36
52.23
RoleBench
Staying in character as a persona
71.21
60.51
52.35
IHEval
Obeying the system prompt over the user
82.40
66.61
62.87
BFCL-v3
Tool calling, if it must act
67.31
65.03
70.56
Tau2-Bench
Agentic task completion
52.32
38.72
61.74
The pattern is clean and it decides your architecture. Bolt wins the conversational axes by wide margins and loses the agentic ones to Qwen3.5 4B. If your assistant answers questions, holds a persona and controls a few local devices, Bolt is the better pick and brevity alone justifies it. If it is really an agent that happens to have a microphone, the tool-calling gap is the number that will hurt, and our spotlight argued exactly that case.
Where the latency actually goes
Time to first token is the figure NVIDIA optimised and the one everyone quotes, but it is only the middle third of what the user experiences. The perceived delay is the sum of four things: finishing the transcription after the user stops talking, the model's time to first token, generating enough tokens to start speaking, and the TTS producing its first audio chunk.
That last point is the one people get wrong. You do not wait for the full reply before speaking. Stream the model's output into Piper sentence by sentence, start playback on the first complete sentence, and the user hears a response while the model is still writing. Done properly, the model's total generation time stops mattering and only its time to first sentence does, which is where a brevity-tuned model and a drafter both pay off twice.
For orientation on the middle third, NVIDIA reports sub-second responses on an RTX 5070 and roughly six seconds to first token on a Jetson Orin Nano 8GB. Those are vendor figures at batch size 1, not ours, and the Jetson number is the one to plan around if you are building something that lives on a shelf. Six seconds is past the point where a person assumes the device did not hear them, which is an argument for a cheap acknowledgement sound the moment transcription ends, and for watching what the newer edge inference silicon does to that figure.
Pulling the three pieces
Everything below runs on CPU if it has to, and all three projects ship prebuilt. The model is the only large download.
mkdir -p ~/voice/models && cd ~/voice/models
# 1. the assistant (6.18 GB)
curl -L -o bolt-e4b-q4km.gguf \
https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB/resolve/main/Nemotron-3.5-Bolt-E4B-Q4_K_M.gguf
# 2. the ears (466 MiB; swap small.en for base.en to save ~460 MB resident)
curl -L -o ggml-small.en.bin \
https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-small.en.bin
# 3. the voice (60 MiB, plus its json config)
curl -L -O https://huggingface.co/rhasspy/piper-voices/resolve/main/en/en_US/lessac/medium/en_US-lessac-medium.onnx
curl -L -O https://huggingface.co/rhasspy/piper-voices/resolve/main/en/en_US/lessac/medium/en_US-lessac-medium.onnx.json
du -sh ~/voice/models
Run the model as a server rather than a one-shot process, so the weights stay resident between utterances. Loading 6 GB from disk on every question is the single biggest self-inflicted latency problem in a hobby voice assistant. If your box has no CUDA, a GGUF this size is still workable on Vulkan or plain CPU, and the Colibri 2.1 write-up covers what each backend costs you.
Note the sampling values. NVIDIA recommends temperature 1.0 and top_p 0.95 for this model in both thinking modes, which is higher than the temperature most people reach for by habit. The context is set to 8192 rather than the model's 64K maximum on purpose: a deeper cache is resident memory you are taking away from Whisper.
Turning on the drafter
Speculative decoding in llama.cpp needs the draft model passed alongside the target. The flags are -md for the draft model and --draft-max for how many tokens it may propose, and seven matches what the DFlash checkpoint is built to offer.
Speculative decoding speeds up generation, not prefill, so it shortens the gap between words rather than the gap before the first one. In a streamed voice loop that still helps, because it gets you to the end of the first sentence sooner. Budget the extra 737 MB before you enable it, and drop to base.en if that is what it takes.
Choosing your ears
The Whisper size you pick is the second real decision, and it is a different trade from the drafter. Transcription errors do not degrade gracefully in a voice assistant: a misheard word does not produce a slightly worse answer, it produces a confident answer to a question nobody asked.
Model
Disk
Resident
Use it when
tiny.en
75 MiB
about 273 MB
Wake words and short fixed commands only
base.en
142 MiB
about 388 MB
Quiet room, close microphone, tight budget
small.en
466 MiB
about 852 MB
The sensible default for open speech
medium
1.5 GiB
about 2.1 GB
Only if you have the headroom to spare
The English-only variants are the right call for an English assistant: same size, better accuracy than the multilingual weights at that size. Going above small.en on an 8 GB device means giving up either the drafter or the conversation depth, and medium at 2.1 GB resident competes directly with the model itself.
Three components, three licences
The stack is permissive but not uniformly so, and the model is the restrictive part. whisper.cpp and Piper are both MIT. Bolt E4B is under OpenMDW 1.1, which is narrower than the Apache 2.0 of the Gemma 4 model it was post-trained from, and that narrowing is the thing we flagged in the spotlight. Read it before you ship a product rather than after, because the licence on the middle box governs the assistant even though the outer two are free of conditions.
What will bite you
The KV cache grows with conversation length, so an assistant that is comfortable at turn three can be swapping at turn thirty. Cap the context deliberately and summarise or drop old turns rather than letting it grow into the memory your speech stack needs.
Bolt's chat template rejects stray tool responses: a tool role message that does not immediately follow an assistant message with tool_calls is refused. NVIDIA documents an asynchronous pattern where you acknowledge the call and report the result as pending. If your assistant controls anything that takes real time, such as a kettle or a garage door, you will hit this on day one.
The 64K context is half of Gemma 4 E4B's 128K. For a voice assistant that is irrelevant, because nobody dictates 64,000 tokens, but it matters if you intended to stuff a manual into the prompt and ask questions about it.
And the obvious one: none of this is a wake word. Always-on listening is a fourth component again, and the usual answer is a small dedicated detector rather than running Whisper continuously, which would burn the CPU budget you need for everything else.
Who should build this
Anyone who wants an assistant that works with the network unplugged. This is the strongest reason, and it is not a performance argument. Every word stays on the device, which for a kitchen or a children's room is the whole point, and the quality is now good enough that the trade is no longer painful.
People shipping a persona rather than an agent. Bolt's RoleBench and IHEval numbers are well ahead of its size class, and a device character that stays in voice and obeys its system prompt is exactly what those measure. If you want something that reliably calls tools, look hard at the BFCL gap first.
Not people on 8 GB who want everything. The accurate ears, the drafter and a deep conversation do not all fit at once. Pick two, or move up to a machine with 16 GB and stop doing arithmetic. An alternative worth knowing about is a single model that does the whole chain itself: we covered one in the PersonaPlex-7B spotlight, and collapsing ASR, LLM and TTS into one set of weights is a genuinely different answer to this problem.
The ten-minute version: pull base.en, Piper and the model, run llama-server with -c 8192, and add up the resident memory on your own box before you decide whether the drafter or the better ears go in the remaining space. That one sum tells you more about whether this will work on your hardware than any benchmark table, including the ones above.
Not independently tested. Every number here is from NVIDIA's model card, the whisper.cpp README, the Piper voice repository or the llama.cpp source, and no part of this stack was run on our hardware. Date checked: 2026-11-19
Prev Article
GLM-5.3-Flash on Two DGX Sparks: The TensorFold Recipe