TL;DR
- A 768B model on one 128 GB APU: hot experts stay in GPU memory, the rest stream from the SSD
- 40.8 tok/s drafted on UD-IQ1_S; plain decode ties Project Maya at 19.3 tok/s
- Bit-identical output across batching, budgets, GPU vs CPU. Self-reported, 8-day-old project
System Requirements
| RAM | 128 GB unified memory, about 113 GiB of it given to the GPU |
| GPU | AMD Ryzen AI MAX+ 395 / Radeon 8060S (gfx1151) only |
| VRAM | Shared memory: about 106 GiB of experts resident, the rest streams from NVMe |
A 768-billion-parameter model on one mini PC, writing at 40 tokens per second. Not a distill, not a smaller sibling wearing its name: DeepSeek-V4.1-Flash-1017 itself, at 1 to 2 bits per expert weight, on a single 128 GB AMD Strix Halo.
That is what libosmos claims. Speed is not the most interesting part, though. Its headline guarantee is that the same input produces the same output, bit for bit, however you run it: alone or batched, at any memory budget, on the GPU or the CPU, with or without speculative decoding.
The repo is eight days old. It went up on October 29, shipped nine releases between November 4 and November 6, and has six stars at the time of writing. Here is what it does, what the numbers do and do not show, and whether your box can run it.
TL;DR: DeepSeek-V4.1-Flash on a single Strix Halo
- One model, one machine. DeepSeek-V4.1-Flash-1017 on a Ryzen AI MAX+ 395 with 128 GB, MIT licensed, written in C and HIP with no Python at runtime.
- Faster than the reference engine where it counts. Against Project Maya 1.1.55 on the same files: +9 to +28% on prompt reading, +17 to +22% on drafted decode, and a tie on plain decode.
- Deterministic. Output hashes match across batching, memory budgets, GPU vs CPU and drafted vs plain decoding, enforced by tests that compare bits, not tolerances.
The project's own numbers, measured November 5 on libosmos 0.3.2 at AMD's performance profile, 64K context, greedy decoding:
| tok/s | UD-IQ1_M libosmos | UD-IQ1_M Maya | UD-IQ1_S libosmos | UD-IQ1_S Maya |
|---|
| Prompt, 2,048 tokens | 293 | 243 | 331 | 259 |
| Prompt, 8,192 tokens | 441 | 406 | 473 | 431 |
| Decode, drafted | 33.6 | 27.5 | 40.8 | 34.9 |
| Decode, plain | 17.1 | 17.0 | 19.3 | 19.2 |
| Bit-reproducible output | yes | not measured | yes | not measured |
If you have seen "38 tok/s" circulating, that is the drafted UD-IQ1_S figure at the full 1M-token context (37.5). The 40.8 above is at 64K.
The problem: a 768B model and 128 GB of memory
DeepSeek-V4.1-Flash is roughly 768B parameters. About 557B of that sits in routed experts and about 196B in Engram tables, DeepSeek's large hashed lookup tables that the model reads a few rows from per token instead of multiplying through. Even at 1 to 2 bits, unsloth's expert shards alone are 122 GB (UD-IQ1_S) or 140 GB (UD-IQ1_M), before the table.
A Strix Halo has 128 GB of unified memory shared between CPU and GPU. So the model does not fit, and something has to live on the SSD. Until now the realistic choices were a multi-GPU server or a slow streaming engine. Our Colibri 2.1 coverage logged this same model, UD-IQ1_S, at about 5 tok/s on Vulkan. NVIDIA Flare wants four 96 GB cards for it.
libosmos takes the narrow road instead: one model, one chip, and every kernel written for that pairing.
How libosmos fits a 768B model in 128 GB
The idea is the same one behind every streaming MoE engine. A mixture-of-experts model only wakes a few experts per token, so you keep the popular ones close and fetch the rest on demand. The details are where the speed comes from.
- Experts by residency. Dense weights plus as many experts as fit (about 106 GiB) stay in GPU memory, chosen by how often a calibration text picks them. The resident set then follows your text, swapping hot experts in.
- Direct SSD reads. Missing experts are read with
O_DIRECT straight from the GGUF files into host memory the GPU can read: no page cache, no CPU copy. A priority queue from the author's libevt puts the read the forward pass is waiting on first, and guesses one layer ahead.
- Integer experts. Each routed expert's input is rounded to int8 in blocks of 32 and multiplied with exact integer dot products (
v_dot4 on the GPU, vpdpbusd on the CPU), then scaled in one fixed order. That shared contract is why the CPU and GPU agree bit for bit.
- Compact attention state. Keys stay in DeepSeek's own FP8 and FP4 formats, about 0.9 KB per token. A full 1M-token context costs about 0.85 GiB.
- DSpark drafting. DSpark is DeepSeek's own small draft model. It proposes up to five tokens, and one batched forward pass of the big model checks them. Accepted tokens are nearly free, which is where the drafted numbers come from.
The author credits Project Maya (MIT, 2.6K stars) for the int8 expert approach and the reference tools, and sends fixes back upstream. Read libosmos as a specialized, deterministic rework of Maya's ideas for one chip, not a design from scratch.
Why bit-for-bit output matters if you build agents
Most local engines are deterministic only in the loose sense. Greedy decoding should give the same answer, but batching, memory pressure and kernel scheduling change the floating-point summation order, and the output drifts. libosmos says Maya's greedy decode varied run to run when it measured Maya 1.1.46. That is libosmos's claim about a competitor, and it has not been re-measured on 1.1.55.
Drift matters in practice. If you regression-test prompts, cache responses, or debug why an agent took a wrong turn, you want a rerun to reproduce the failure, not a slightly different one. libosmos enforces its guarantee with GPU tests that fingerprint every output stream across configurations, and with kernel checks that deliberately mutate inputs and must fail. Answers hash the same across 0.1.3, 0.2.0, 0.3.0 and 0.3.2, according to the project.
Agents and energy
An agent spends most of its time reading, not writing. Maya's own agent bench (four agents at once, eight turns each, 131K context) sends about 99K new prompt tokens against about 8K answer tokens. On UD-IQ1_M:
| libosmos 0.3.2 | Maya 1.1.55 |
|---|
| Wall time, all 32 turns | 446 s | 542 s |
| First token, p50 / p95 | 6.2 / 54.7 s | 6.3 / 81.3 s |
| Turn done, p50 / p95 | 44.1 / 94.0 s | 55.3 / 121.9 s |
| Package power, mean | 127 W | 127 W |
| Energy per answer token | 7.07 J | 8.74 J |
Switch the box to AMD's stock low-power profile and drafted decode drops only to 30.7 tok/s at about 71 W, or 2.35 J per token. That is 8 to 9% slower for 39 to 46% less energy. Power here is the APU package (CPU, GPU and NPU) read from the driver, not the wall socket, so add your SSD and the rest of the system on top.
Hands-on: install libosmos (from the docs)
We have not run this. Everything below comes from the project's README. Check the requirements first, because they are strict:
- AMD Ryzen AI MAX+ 395 (gfx1151) with 128 GB. Nothing else is supported.
- Linux with ROCm 7.x (developed on HIP 7.15, kernel 7.2, gcc 14).
- GPU memory raised to about 113 GiB via a kernel module option. That is a system setting, and the README leaves it to you.
- The model on an NVMe SSD with a filesystem that accepts
O_DIRECT reads: ext4, btrfs and xfs work, tmpfs and some FUSE filesystems do not, and libosmos refuses them by name.
- For the optional CPU lane: AVX-512 VNNI and VBMI, which this chip has.
- Disk: 122 GB (UD-IQ1_S) or 140 GB (UD-IQ1_M) of expert shards, plus unsloth's 203 GB MXFP8 Engram table that both packs share.
# 1. Raise the GPU memory limit (reboot afterwards)
echo "options ttm pages_limit=29637703" | sudo tee /etc/modprobe.d/ttm.conf
# 2. Build, download the pinned files, verify SHA-256, start the API on :8091
git clone --recursive https://github.com/mrsirg97-rgb/libosmos
cd libosmos
python3 tools/install/osmos_install.py ud-iq1_s --serve # or ud-iq1_m
# Already have the shards? Point the installer at them
python3 tools/install/osmos_install.py ud-iq1_s --have ~/Models/ds1017 --serve
# 3. Talk to it like any OpenAI-compatible server
curl -s http://localhost:8091/health
curl -s http://localhost:8091/v1/models # the model id the server reports
curl -s http://localhost:8091/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"messages":[{"role":"user","content":"Write a bash one-liner that finds the 10 largest files under /var"}],"stream":false}'
The server, osmos-api, handles POST /v1/chat/completions (streamed or whole, with tools and reasoning), GET /v1/models and GET /health. That means anything that already talks to an OpenAI-style endpoint can use it, from coding agents to the servers in our Ollama vs llama.cpp vs vLLM comparison. The installer also offers --abliterated, which overlays huihui-ai's patch (abliteration edits the attention output weights to suppress refusals).
Picking a pack: UD-IQ1_S or UD-IQ1_M
| UD-IQ1_M (quality) | UD-IQ1_S (speed) |
|---|
| Perplexity, 8,191 tokens of text and code | 2.459 | 2.501 |
| unsloth six-benchmark score | 85.55 | 83.53 |
| Decode, plain / drafted | 17.1 / 33.6 tok/s | 19.3 / 40.8 tok/s |
| Expert shards on disk | 140 GB | 122 GB |
UD-IQ1_S's experts are 12.5% smaller, so more of them stay resident and fewer come off the SSD. That is the whole speed difference. ISTA-DASLab's GSQ-RCO IQ2_XXS quant, which the engine was first built on, decodes about as fast as UD-IQ1_S but scored worst of the three (perplexity 2.546), so it is supported but not packaged.
Limitations and gotchas
- One chip. No other AMD part, no NVIDIA, no Apple Silicon. If you are still choosing hardware, our Strix Halo vs DGX Spark vs M4 Max comparison covers the tradeoffs.
- Greedy only in the benchmarks. Every number above uses greedy decoding with one sequence at a time. The determinism guarantee is stated for greedy output.
- 1-bit experts cost something. The project reports perplexity and unsloth's benchmark score, but no comparison against the full-precision model. Do not assume this is the same DeepSeek-V4.1-Flash you get through the API.
- Brand new, one maintainer. Eight days, 169 commits, one anonymous author, and no third-party reproduction yet. The repo ships an
AGENTS.md operating contract, which indicates coding agents did much of the work. To its credit, DESIGN.md logs every measurement, including the ideas that did not help.
- The comparison is with one engine. Maya is the obvious reference on this chip, but there are no llama.cpp numbers on the same box.
Who should use it, and what comes next
If you own a 128 GB Strix Halo and want the strongest open model you can run locally for agent work, this is currently the fastest documented way to do it. If you need reproducible outputs for testing or evaluation, libosmos is worth a look even if Maya's plain decode is just as fast.
v0.4.0, released November 6, changes no speed or output. It puts the engine behind a single interface as groundwork for a second model family, Qwen3.8-Flash-Next, the model Strata runs on a 12 GB gaming GPU. If that lands, the "one model" limitation loosens.
Your ten-minute step: if you have the box, run cat /etc/modprobe.d/ttm.conf and df -T on your model drive. Those two checks tell you whether the installer will even start. If you already downloaded unsloth's shards for llama.cpp, --have saves you the download.
Sources
Not independently tested. All figures are the project's own, measured 2026-11-05 on libosmos 0.3.2 on one Ryzen AI MAX+ 395 with 128 GB.
