TL;DR
- Reads published GGUF files unchanged, IQ1_S through BF16, on CPU, Vulkan and CUDA
- 104.8 tok/s on Qwen3.6-35B-A3B, measured on one 24 GB card that draws 70 watts
- An agent interstitial turn drops from 86 seconds to 4.9, on every backend
System Requirements
| RAM | 16 GB runs a mid-size model from RAM; the project benchmark box has 62 GB |
| GPU | Any Vulkan GPU, or NVIDIA for CUDA; every headline figure is a 24 GB RTX PRO 4000 at 70 W |
| VRAM | None required. A 68 GB model runs on 24 GB; whatever does not fit streams from NVMe |
✓ Apple Silicon
Thirty-three days ago we told you not to run colibri expecting a local chatbot. At one to two tokens per second, we said, the interaction model is closer to submitting a job than holding a conversation. Two releases later that sentence is half wrong, and the half that broke is worth more than the half that held. colibri 2.1.0 landed today, reads your existing GGUF files exactly as published, and decodes Qwen3.6-35B-A3B at 104.8 tokens per second on a card rated at 70 watts.
What shipped, and when
colibri is a mixture-of-experts inference engine written in pure C, one file per model family, Apache 2.0, by Vincenzo Fornaro. Version 2.1.0 was published on 31 October 2026. Version 2.0.0 landed on 6 October, which means both releases arrived after our September piece.
The project reports 387 pull requests since 2.0.0, 208 of them from 35 contributors. We counted independently against the GitHub API: 1,409 commits between the two tags and 411 pull requests merged in the calendar window, which brackets the project's figure rather than contradicting it. The repository is at 45,377 stars and 5,178 forks. When we covered it in September it had 38,000.
GGUF, read exactly as published
This is the headline, and the wording matters. coli import takes a GGUF file where it lies and every backend runs it with no conversion and no second quantization, from IQ1_S through Q8_0, F16, BF16 and MXFP4. The same blocks and the same tokens on the CPU, on Vulkan and on CUDA.
The phrase "no second quantization" is doing real work. The usual path for a new engine is to convert a GGUF into its own container, which often means decoding someone's carefully chosen quantization and re-encoding it. Whatever the quant author decided gets partly undone. colibri 2.1 reads the published blocks directly, which is why the project can say unsloth's Dynamic UD quants run as they are. The person who tuned which tensors keep more bits still gets their result.
Four model families arrived with it: Mistral-Large-4 (1T total, 52B active, with text, images, reasoning and tools), Muse Spark Open at 816B, Flint at 40B with 2.5B active, and LFM3-VL-2B with its vision tower, plus the published DeepSeek V4.1 Flash and Kimi K3 GGUFs.
On Mistral-Large-4, one note for anyone who went looking last week and came back empty. The weights repository was created on 30 September, which puts it behind newer Mistral uploads in a listing sorted by date, so it is easy to miss. The community GGUF conversions appeared on 28 and 29 October. colibri shipped support within days of those files existing.
The numbers, and the 70 watt card nobody mentioned
Every headline figure in this release comes from one machine: an i5-13500 with 62 GB of RAM and an RTX PRO 4000 Blackwell SFF. That card has 24 GB and a 70 watt board power. Each model is served by coli serve, measured as a fresh 450-token prompt then a 128-token streamed answer with the request's HTTP included, the better of two runs.
| GGUF | CUDA prompt | CUDA decode | Vulkan decode | CPU decode |
|---|
| Qwen3.6-35B-A3B UD-Q4_K_M | 3,491 | 104.8 | 92.9 | 13.9 |
| LFM3-VL-2B Q4_K_M | 12,317 | 235.1 | 214.1 | 34.5 |
| OLMoE-1B-7B-0125-Instruct Q4_K_M | 10,418 | 354.6 | 299.4 | 47.4 |
| LFM3-VL-2B F16 | 11,524 | 103.4 | 99.3 | 6.4 |
| Qwen3.8-Flash-Next IQ2_XS (68 GB file) | 217.1 | 51.4 | 50.6 | 9.3 |
All figures are tokens per second, published by the project. Hold the 70 watts in your head while you read them, because it is the part the coverage will drop. A 68 GB model decoding at 51.4 tok/s on a 24 GB card is the result people will quote. That it happened on a card you could run off a modest power supply is the result that should change what you buy.
It also makes a neat pair with our piece on unlocking full-rate tensor cores last week. That story was about spending watts to buy matrix throughput, with its headline gains measured at 450 W and the project warning you about power draw. This one buys throughput by deciding where weights live, on a card that sips. Same series, opposite directions, and both are real.
The streaming tier is still slow, and that is still the point
The original colibri trick has not gone anywhere. What neither the card nor the RAM holds comes off two NVMe drives, and the price is what it always was.
| GGUF | File size | Backend | Decode tok/s |
|---|
| Inkling-Small UD-IQ2_XXS | 77 GB | Vulkan | 10.92 to 11.53 |
| MiMo-V2.6-Flash Q2_K_S | 106 GB | Vulkan | 7.8 |
| DeepSeek V4 Flash 0731 UD-IQ2_XXS | 85 GB | Vulkan | 6.18 to 6.25 |
| DeepSeek V4.1 Flash UD-IQ1_S | 233 GB | Vulkan | 5.01 to 5.29 |
| GLM-5.3-Flash UD-IQ2_XXS | 95 GB | Vulkan | 2.92 to 3.40 |
| Muse Spark Open UD-IQ1_S | 176 GB | Vulkan | 1.55 to 1.58 |
| Mistral-Large-4 UD-IQ1_S | 233 GB | CUDA | 1.34 to 1.36 |
| GLM-5.2 UD-IQ1_S | 202 GB | Vulkan | 0.633 to 0.658 |
| Kimi K3 UD-IQ1_S | 594 GB | Vulkan | 0.349 |
Kimi K3 is 594 GB of file producing roughly one token every three seconds. That is not a chat. It is a 2.8 trillion parameter model answering on a desktop, which is a different and still remarkable thing, and the project has never pretended otherwise. Read the two tables together and the real shape of colibri 2.1 appears: one binary containing a fast local engine and a slow impossible one, and which you get depends entirely on whether your model fits.
The dividing line is roughly five tokens per second. Above it you can work interactively. Below it you are queuing jobs. DeepSeek V4.1 Flash at 233 GB sits right on that line.
The number that will actually change your day
Buried under the throughput tables is the biggest practical win in the release, and it is not a throughput number at all.
Agent harnesses constantly fire short requests between real turns: name this conversation, summarise what just happened, pick a title. Until now each of those made the engine take the conversation's whole state, so a trivial request paid the cost of the entire context. colibri 2.1 parks the conversation instead, on every backend.
The measured result, on a Hermes Agent second turn at 14,810 tokens on Qwen3.8-Flash-Next: 86 seconds down to 4.9 seconds. On the CPU with Qwen3.6-35B it goes from 51.6 seconds to 3.3 seconds. That is a 17x and a 15x on wall-clock latency for a pattern every agent harness uses, including Hermes Agent itself, which is what they benchmarked. It is on by default; COLI_PARK=0 turns it off.
In the same area, Qwen3.8-Flash-Next now runs a 262,144-token context at roughly the speed of 64k: prompt reading goes from 15.6 to 149 tok/s and decode from 9.6 to 45.7. For comparison the project reports 188 and 48.4 at 65,536 tokens, so the claim is that the 256k penalty has largely gone, not that long context became free.
Same model, two projects, two numbers
Qwen3.8-Flash-Next appears in both this release and our write-up of Strata, which reported 94 tok/s for the same model on a 12 GB RTX 5070. colibri reports 51.4. Those do not compare, and it is worth being exact about why.
| Claim | Project | What it actually is |
|---|
| 94 tok/s on 12 GB | Strata | Qwen3.8-Flash-Next in the GSQ-RCO quantization, RTX 5070 |
| 51.4 tok/s on 24 GB | colibri 2.1 | The IQ2_XS GGUF, a 68 GB file, RTX PRO 4000 at 70 W, HTTP included |
Different quantizations of different sizes, different cards, different harnesses, and only one of them counts the HTTP round trip. Anyone putting these side by side in a table is comparing two measurements that share nothing but a model name. We have done this check before and the answer is nearly always the same: the headline number is real and the comparison is not.
Two defaults changed under you
colibri's README states a design policy we think is worth holding it to: the default policy never silently changes model precision or router semantics. Version 2.1 changes two defaults, and the distinction matters.
KV caches are now float16 rather than float32, which halves their memory. GLM-5.3 at 131,072 tokens now holds 11.9 GB instead of 23.9. This is cache precision, not weight precision or router behaviour, so it does not breach the stated policy. It does mean a 2.1 run and a 2.0 run are not the same measurement, and if you are tracking quality across versions you should know which you have. The escape hatches are Q36_KV=f32 for Qwen3.6 and the KV16 and Q38_KV switches elsewhere; the float16 MLA latent cache does not apply on CUDA or Metal.
The second is conversation parking, described above, now on by default for any engine serving one conversation. Both changes are documented in the release notes under a heading that says exactly what they are, which is more than most projects manage. Read that section before you compare anything to a previous run.
Where colibri is weakest, by its own numbers
The most useful thing in the repository is a benchmark that makes the project look bad, and it publishes it anyway. GLM-5.2 decode across hardware classes:
| Hardware | GLM-5.2 decode tok/s |
|---|
| Six RTX 5090, every expert resident | 5.8 to 6.8 (9.0 to 9.2 with selective NUMA interleave) |
| 128 GB CPU only, Ryzen AI Max+ 395 | 1.83 warm |
| Single RTX 5070 Ti laptop-class box | 1.07 |
| The 25 GB laptop where the project started | 0.05 to 0.1 cold |
Six RTX 5090s is roughly 192 GB of VRAM holding every expert, and it returns 6 to 9 tokens per second. That is the honest boundary of this design. colibri is built for scarcity, and its advantage is placement under pressure. Remove the pressure and the placement machinery has nothing left to win, at which point a conventional server engine is the better tool. If you already own that much hardware, this is not the engine for you, and the project's own ladder is what tells you so.
What it borrows, and from whom
colibri is not a llama.cpp fork. It is C written from scratch with no BLAS and no Python at runtime, Python being confined to setup, the launcher, converters and the API gateway. But it is unusually specific about what it took from where, and that credit trail is the fastest way to understand what the project actually is.
It reimplements tiktoken's byte_pair_encode exactly, merging the adjacent pair whose concatenation has the lowest vocabulary id, so a tiktoken-derived vocabulary needs no merges list. The GBNF grammar subset follows llama.cpp's syntax and its set-of-stacks approach, and the Metal path borrows its buffer residency trick. vLLM is the reference for output semantics, matched position by position. transformers is the CI oracle: the build reproduces a random-init model token for token against it. Nine research projects on expert importance, re-routing, merging and scheduling are credited by name.
That last item is the one to weigh. An engine that checks itself token for token against transformers on every build has a defensible answer to the question every alternative runtime eventually faces, which is whether it is still running your model or something that resembles it.
The bus factor, in numbers
Thirty-five contributors is the headline, and it is true. The distribution behind it is extremely concentrated, which matters if you are putting this under something that has to keep running.
Of the 208 contributor pull requests in this release, 92 came from one person. We checked that independently and counted 93 merged in the window, so the project's figure is accurate. The maintainer has 3,085 commits in the repository; the next highest contributor has 218. This is a one-author project with one very active collaborator and a long tail, moving at about 56 commits a day for the last 25 days.
None of that is a criticism. It is the shape of most young infrastructure, and the project is four months old with 155 open issues, which is the normal cost of growing this fast. It is simply the thing to know before you depend on it.
Running it
The release ships prebuilt archives for Linux x86-64, Windows with and without CUDA, and Apple Silicon with Metal, each checked by CI before publishing. The one-step setup picks the right one or builds from source when a compiler is present.
coli import model.gguf
coli chat --model model.colibri
# a published split GGUF, read where it lies
hf download unsloth/DeepSeek-V4.1-Flash-1017-GGUF --include "UD-IQ1_S/*" --local-dir v41
cd v41/UD-IQ1_S
coli import DeepSeek-V4.1-Flash-1017-UD-IQ1_S-00001-of-00005.gguf
COLI_VULKAN=1 coli chat --model DeepSeek-V4.1-Flash-1017-UD-IQ1_S.colibri
# the new chat UI
coli serve
coli ui
# put the old defaults back, if you are comparing against a 2.0 run
COLI_PARK=0 coli serve
Q36_KV=f32 coli serve
Point coli import at the first part of a split file with the others beside it. The new coli ui gives you reasoning in its own block, images, web search, a live map of the experts as they fire, a monitor, and coding agents already pointed at the local server, in seven languages. If you have fast storage, two NVMe drives on independent controllers measured 37.5% faster decode than one.
Grading our own September call
We said two things in September. One held and one did not.
What held: the giant models still cost what we said they cost. We wrote that you are buying access rather than throughput, and that for batch or overnight work a slow honest answer beats no answer. Kimi K3 at 0.349 tok/s and GLM-5.2 at 0.633 say that is still exactly right. We also said your disk matters more than your GPU, which remains true for everything in the streaming tier.
What broke: "do not run it expecting a local chatbot." That was a fair read of the engine in September, when the thing only ran models far too large for your machine. It is no longer a fair read of the binary you download today, because the models most people actually run now decode between 92 and 354 tok/s on one mid-range card. The sentence was not wrong about the engine we tested. It aged badly because the project shipped a second product inside the same binary, and we did not see that coming.
The lesson we are keeping: when a project's identity is a constraint, "it streams experts from disk because they do not fit," check whether the constraint still binds before repeating the conclusion that came with it.
Who should run this now
Run it if you have a mid-range card and a pile of GGUF files. This is the new audience and the reason 2.1 matters. Your existing downloads work unchanged across CPU, Vulkan and CUDA, which is a genuinely useful property if you move between machines or run AMD and NVIDIA in the same house.
Run it if you drive a local agent harness. The 17x on interstitial turns is the single largest practical improvement here, and nothing else in the local inference world is specifically fixing that pattern right now. Pair it with speculative decoding if your model has a draft head. Qwen3.6's now runs on every backend and is worth 15 to 16 tok/s on CUDA, taking decode from 99.7 to 104.5 up to 115.8 to 120.1; the README calls it on by default and the release notes show a Q36_MTP=1 switch, so check which you have before you measure.
Run it if you want to evaluate a frontier model you cannot otherwise touch. Mistral-Large-4 at 1T parameters on a 24 GB desktop, at 1.34 tok/s, is a real option for an evaluation run even though it is unusable for conversation.
Do not run it if you own serious hardware. The six-5090 number is the project telling you this itself. And do not run it expecting the storage commitment to be small: 594 GB for Kimi K3 and 233 GB for Mistral-Large-4 are real purchases before you generate a token.
Sources and further reading
Your next ten minutes
If you already have colibri installed, read the "What changes by default" section of the release notes before you benchmark anything, because two defaults moved and one of them halves your KV cache. If you do not have it, the fastest useful test is not a frontier model at all: take a GGUF you already downloaded, run coli import on it, and see whether it runs on the backend you could not previously use it on. That is a five-minute answer to the only question that matters for most readers, which is whether this engine does anything for the files already on your disk.
Not independently tested. Every performance figure above is the project's own published measurement, taken from the v2.1.0 release notes and the repository README, and the single-machine figures all come from one i5-13500 with 62 GB of RAM and an RTX PRO 4000 Blackwell SFF at 24 GB and 70 W. Numbers from different tables were produced by different runs on different files and are not comparable with each other. Repository statistics, commit and pull-request counts, contributor distribution and the Mistral-Large-4 upload dates were checked against the GitHub and Hugging Face APIs on the date below. The comparison with Strata uses that project's own self-reported figure as published in our earlier coverage.
Date checked: 2026-10-31
