How a 28.9M-Parameter LLM Runs on an $8 ESP32-S3 Microcontroller

This is part two of our microcontroller AI series. In part one we argued the honest version of "AI on a microcontroller": a real sensor node runs TinyML classifiers, not language models, and the closest anyone had come to an LLM on an ESP32 was a 260K-parameter toy that "generates story fragments" and little else. Well, someone just moved that line. The developer slvDev shipped esp32-ai, a 28.9-million-parameter language model that runs on an $8 ESP32-S3 at about 9.5 tokens per second and writes coherent little stories. That is roughly 100 times bigger than the toy. Here is the trick that makes it fit, what the numbers actually say, and why it still does not overturn part one.

What changed since part one

Part one drew a hard line between two things people both call "edge." A microcontroller like the ESP32 has kilobytes to low megabytes of usable memory and runs TinyML: tiny quantized classifiers that spot a keyword, a sound, or an anomaly. A single-board computer like a Raspberry Pi has gigabytes and can run a small language model on its CPU. The gap between them is three orders of magnitude, and no amount of quantization closes it.

The esp32-ai project does not break that rule. It bends the memory question instead. The previous ESP32 language-model demo we cited held a 260K-parameter network. This one holds 28.9M parameters, coherent enough to finish a sentence and keep a short story on track, on the same class of $8 chip. The output is the same category as before, short synthetic stories, but the model behind it is two orders of magnitude larger. The interesting part is not the story. It is how 28.9M parameters fit somewhere that cannot hold them in RAM.

The trick: Per-Layer Embeddings from flash

Start with the constraint. Transformer inference wants its weights in fast memory, because it touches them on every token. The ESP32-S3 has 512KB of on-chip SRAM. A 28.9M-parameter model at 4-bit is roughly 14.5MB. It cannot live in RAM. Bolting on the 8MB of PSRAM does not fix it either, and PSRAM is slow anyway, which matters later.

The move is to notice that not every parameter is a compute parameter. Per-Layer Embeddings (PLE) is a design from Google's Gemma models. The idea: give each token, at each layer, its own learned embedding vector pulled from a big lookup table, instead of folding all that capacity into the dense weights the math runs over. The table is huge in parameter count but you only ever read the handful of rows a given token needs. It is a dictionary, not a computation.

That property is what makes it fit a microcontroller. The esp32-ai project splits the model across all three memory tiers by what each part actually needs:

The dense math the chip runs on every token is tiny, about 559K parameters, and it fits in fast SRAM. The 25M parameters of extra capacity sit in flash as a memory-mapped table. Per token the model reads about 450 bytes from that table, six rows, in roughly 0.12 milliseconds. The author measures the table at about 0.7 percent of per-token memory time. You get the model quality of 28.9M parameters while paying, on the hot path, for about half a million.

This is the same family of idea as the memory-saving quantization tricks we covered in our Google TurboQuant writeup. The theme is consistent: on constrained hardware, the wall is memory, and the wins come from not holding things you do not need in the fast tier.

Does the table actually help, or is it plumbing?

A fair objection: maybe the gain comes from the extra wiring, not from the 25M-parameter table itself. The author ran the ablation to check, measuring perplexity (lower is better, it is how well the model predicts held-out text) across four configurations.

PLE cuts perplexity from 12.58 to 11.41, about 9.3 percent, while keeping the compute core the same size. FatEmbed, a simpler way to spend the same parameter budget, does worse. The control run, ple_notable, wires up the same plumbing but without a real lookup table, and it lands worse than baseline. The author's summary is the right one: "the table does the work, not the plumbing." And the gain survives quantization. The PLE advantage measured in full precision actually holds or grows after 4-bit post-training quantization, at 124 to 126 percent retention, so the technique is not a floating-point artifact that evaporates on the shipping format.

Where the time goes: the bottleneck moved, it did not vanish

Fitting the model is one problem. Running it fast enough to be bearable is another, and this is where the honest engineering shows. At ~9.5 tok/s a token takes about 105 milliseconds. Here is where that time goes, per the author's profiling on the dual-core chip.

Read that top row twice. The output head, the layer that turns the model's internal state into a probability over all 32,768 vocabulary tokens, eats 57.6 ms, more than half the budget. It lives in PSRAM, and PSRAM reads at about 60.7 MB/s. Scanning the 2.43MB of head weights once takes roughly 40 ms just to move the bytes, before any math. Compute gets the leftover ~17 ms. The flash lookup table everyone worries about? 8.5 ms, and most of that is not the table reads themselves.

So the PLE trick did not make the model free. It moved the bottleneck. The dense core no longer bounds you; PSRAM bandwidth does. The author's own math puts the theoretical ceiling around 58 tok/s if you were perfectly bandwidth-bound, and the current runtime sits well under that because the compute stages have not all been overlapped with the reads yet. This is the useful lesson for anyone building on constrained hardware: solve the memory-capacity wall and you often just meet the memory-bandwidth wall behind it.

Running it yourself

The project ships everything: the training and quantization code in Python, the ESP32 firmware, and the exported model. You do not need a training run to try it, the repo includes a pre-exported model you can flash. The firmware builds with the Arduino ESP32 core (3.3.10) driven by arduino-cli, not ESP-IDF or PlatformIO. The one wrinkle worth knowing is that the model binary is flashed to its own partition, separate from the firmware, so firmware-only changes do not force you to rewrite the 15MB model.

# 1. Export and verify the model against a golden reference on your host
cd src && uv run python export.py && cd ..
cc -O3 -o /tmp/esp32-llm-verify firmware/host_verify/verify.c -lm
/tmp/esp32-llm-verify firmware/model/model.bin firmware/model/golden.txt

# 2. Compile and upload the firmware to an ESP32-S3
arduino-cli compile --fqbn esp32:esp32:esp32s3 firmware/esp32_llm
arduino-cli upload   --fqbn esp32:esp32:esp32s3 -p /dev/ttyUSB0 firmware/esp32_llm

# 3. Write the model binary to its own flash partition (only needed after export)
esptool.py --chip esp32s3 write_flash 0x110000 firmware/model/model.bin

# 4. Watch it generate
arduino-cli monitor -p /dev/ttyUSB0 -c baudrate=115200

What comes out is a stream of short TinyStories-style prose at about 9.5 tokens per second, rendered on a small attached display. The repo has a demo GIF if you want to see it move before you buy a board. Treat the code and weights as an experiment: this is a research demo of a storage technique, not a maintained product, so pin your toolchain versions and expect to read the source.

What it proves, and what it does not

Here is the honest verdict, and it is two-sided on purpose.

What it proves: you can hold a 28.9M-parameter model on a $8 microcontroller and run it, if you are willing to store most of it as a memory-mapped flash table and read only the rows each token needs. Per-Layer Embeddings, pulled from a research idea into a working ESP32 firmware, is a real and reusable technique. If your problem is "this model is a bit too big for the fast memory I have," moving the embedding capacity to a slower tier you read sparsely is a lever worth knowing. The ablation shows the capacity is doing real work, and it survives 4-bit quantization.

What it does not prove: that a microcontroller can run a useful language model. This model writes toy stories. It will not answer a question, follow an instruction, call a tool, or know a single fact, because TinyStories does not teach any of that, and 28.9M parameters could not hold much of it anyway. The output category is exactly what part one described, just bigger and more coherent. And the moment you chase real capability you need a real vocabulary, a real instruction-tuned model, and a KV cache for context, and every one of those pushes you straight back off the microcontroller and onto a Raspberry Pi or a phone. Part one's cheat sheet still stands: put the reflex on the sensor, keep the brain where the memory is.

Who should care, and what to watch

If you build products on constrained hardware, this is not a component you ship. It is a technique you file away. The people who should actually clone the repo are hobbyists who want to see an LLM breathe on an $8 chip, and researchers hunting for ways to fit more capacity into fixed memory budgets. The one thing worth watching: whether anyone applies the same flash-resident PLE trick to a small instruction-tuned model rather than a story generator. That is the experiment that would test whether this is a curiosity or a path. Until then, it is a very good curiosity, and a clean demonstration that on tiny hardware the binding constraint is always memory.

Sources and further reading

• esp32-ai repository (slvDev)
• RESULTS.md: methodology, ablations, and measurements
• TinyStories dataset (Hugging Face)
• Gemma 3 model card (Per-Layer Embeddings)
• ESP32-S3 datasheet (Espressif)
• Part one: edge AI on microcontrollers, what actually fits

Tested on: n/a (no hardware). All figures are author-reported from the repository's RESULTS.md, dated 2026-07-21.
Date checked: 2026-07-24