# SingularityByte (full text) > Full-text companion to /llms.txt. The most recent SingularityByte articles per section, with each body inlined as plain text so a model can read the actual content without fetching the HTML pages. For the link map only (titles, URLs, descriptions), see [/llms.txt](https://singularitybyte.com/llms.txt). Coverage is open-weights-first for developers, indie hackers, AI engineers, researchers, and technical founders who run local models. This file is generated by the CMS from live content. ## Tutorials # How a 28.9M-Parameter LLM Runs on an $8 ESP32-S3 Microcontroller Source: https://singularitybyte.com/tutorials/run-28m-llm-on-esp32-s3-microcontroller-2026.html This is part two of our microcontroller AI series. In [part one](/tutorials/edge-ai-on-microcontrollers.html) we argued the honest version of "AI on a microcontroller": a real sensor node runs TinyML classifiers, not language models, and the closest anyone had come to an LLM on an ESP32 was a 260K-parameter toy that "generates story fragments" and little else. Well, someone just moved that line. The developer slvDev shipped [esp32-ai](https://github.com/slvDev/esp32-ai), a 28.9-million-parameter language model that runs on an $8 ESP32-S3 at about 9.5 tokens per second and writes coherent little stories. That is roughly 100 times bigger than the toy. Here is the trick that makes it fit, what the numbers actually say, and why it still does not overturn part one. ## What changed since part one Part one drew a hard line between two things people both call "edge." A microcontroller like the ESP32 has kilobytes to low megabytes of usable memory and runs TinyML: tiny quantized classifiers that spot a keyword, a sound, or an anomaly. A single-board computer like a Raspberry Pi has gigabytes and can run a small language model on its CPU. The gap between them is three orders of magnitude, and no amount of quantization closes it. The esp32-ai project does not break that rule. It bends the memory question instead. The previous ESP32 language-model demo we cited held a 260K-parameter network. This one holds 28.9M parameters, coherent enough to finish a sentence and keep a short story on track, on the same class of $8 chip. The output is the same category as before, short synthetic stories, but the model behind it is two orders of magnitude larger. The interesting part is not the story. It is how 28.9M parameters fit somewhere that cannot hold them in RAM. Spec | Value Total parameters | 28.9M (25M live in flash) Dense compute core | ~559K params (273KB at 4-bit, fits SRAM) Config | d_model 96, 6 layers, ple_dim 128, vocab 32,768 Model file | 14.9MB (group-128 ragged-int4) Board | ESP32-S3 N16R8 (16MB flash / 8MB PSRAM), ~$8 Speed | ~9.5 tok/s end to end (author-reported) Training data | TinyStories ## The trick: Per-Layer Embeddings from flash Start with the constraint. Transformer inference wants its weights in fast memory, because it touches them on every token. The ESP32-S3 has 512KB of on-chip SRAM. A 28.9M-parameter model at 4-bit is roughly 14.5MB. It cannot live in RAM. Bolting on the 8MB of PSRAM does not fix it either, and PSRAM is slow anyway, which matters later. The move is to notice that not every parameter is a compute parameter. Per-Layer Embeddings (PLE) is a design from Google's Gemma models. The idea: give each token, at each layer, its own learned embedding vector pulled from a big lookup table, instead of folding all that capacity into the dense weights the math runs over. The table is huge in parameter count but you only ever read the handful of rows a given token needs. It is a dictionary, not a computation. That property is what makes it fit a microcontroller. The esp32-ai project splits the model across all three memory tiers by what each part actually needs: Memory tier | Size | What lives there | Why SRAM | 512KB | Dense compute core (~559K params, 273KB at 4-bit) | Touched every token, must be fast PSRAM | 8MB | Output head and working buffers | Big, scanned once per token Flash | 16MB | 25M-param PLE lookup table (12MB), memory-mapped | Only a few rows read per token The dense math the chip runs on every token is tiny, about 559K parameters, and it fits in fast SRAM. The 25M parameters of extra capacity sit in flash as a memory-mapped table. Per token the model reads about 450 bytes from that table, six rows, in roughly 0.12 milliseconds. The author measures the table at about 0.7 percent of per-token memory time. You get the model quality of 28.9M parameters while paying, on the hot path, for about half a million. This is the same family of idea as the memory-saving quantization tricks we covered in our [Google TurboQuant writeup](/tools/google-turboquant.html). The theme is consistent: on constrained hardware, the wall is memory, and the wins come from not holding things you do not need in the fast tier. ## Does the table actually help, or is it plumbing? A fair objection: maybe the gain comes from the extra wiring, not from the 25M-parameter table itself. The author ran the ablation to check, measuring perplexity (lower is better, it is how well the model predicts held-out text) across four configurations. Configuration | Compute core | Total params | Perplexity Baseline | 559K | 3.7M | 12.58 PLE | 558K | 28.9M | 11.41 FatEmbed | 559K | 28.9M | 11.94 ple_notable (control) | 558K | 3.7M | worse than baseline PLE cuts perplexity from 12.58 to 11.41, about 9.3 percent, while keeping the compute core the same size. FatEmbed, a simpler way to spend the same parameter budget, does worse. The control run, ple_notable, wires up the same plumbing but without a real lookup table, and it lands worse than baseline. The author's summary is the right one: "the table does the work, not the plumbing." And the gain survives quantization. The PLE advantage measured in full precision actually holds or grows after 4-bit post-training quantization, at 124 to 126 percent retention, so the technique is not a floating-point artifact that evaporates on the shipping format. ## Where the time goes: the bottleneck moved, it did not vanish Fitting the model is one problem. Running it fast enough to be bearable is another, and this is where the honest engineering shows. At ~9.5 tok/s a token takes about 105 milliseconds. Here is where that time goes, per the author's profiling on the dual-core chip. Stage | Time per token | Bound by Output head | 57.6 ms | PSRAM bandwidth Attention | 25.6 ms | Compute PLE lookup | 8.5 ms | Flash reads Feed-forward | 6.9 ms | Compute Input processing | 4.4 ms | Compute Read that top row twice. The output head, the layer that turns the model's internal state into a probability over all 32,768 vocabulary tokens, eats 57.6 ms, more than half the budget. It lives in PSRAM, and PSRAM reads at about 60.7 MB/s. Scanning the 2.43MB of head weights once takes roughly 40 ms just to move the bytes, before any math. Compute gets the leftover ~17 ms. The flash lookup table everyone worries about? 8.5 ms, and most of that is not the table reads themselves. So the PLE trick did not make the model free. It moved the bottleneck. The dense core no longer bounds you; PSRAM bandwidth does. The author's own math puts the theoretical ceiling around 58 tok/s if you were perfectly bandwidth-bound, and the current runtime sits well under that because the compute stages have not all been overlapped with the reads yet. This is the useful lesson for anyone building on constrained hardware: solve the memory-capacity wall and you often just meet the memory-bandwidth wall behind it. ## Running it yourself The project ships everything: the training and quantization code in Python, the ESP32 firmware, and the exported model. You do not need a training run to try it, the repo includes a pre-exported model you can flash. The firmware builds with the Arduino ESP32 core (3.3.10) driven by arduino-cli, not ESP-IDF or PlatformIO. The one wrinkle worth knowing is that the model binary is flashed to its own partition, separate from the firmware, so firmware-only changes do not force you to rewrite the 15MB model. ``` # 1. Export and verify the model against a golden reference on your host cd src && uv run python export.py && cd .. cc -O3 -o /tmp/esp32-llm-verify firmware/host_verify/verify.c -lm /tmp/esp32-llm-verify firmware/model/model.bin firmware/model/golden.txt # 2. Compile and upload the firmware to an ESP32-S3 arduino-cli compile --fqbn esp32:esp32:esp32s3 firmware/esp32_llm arduino-cli upload --fqbn esp32:esp32:esp32s3 -p /dev/ttyUSB0 firmware/esp32_llm # 3. Write the model binary to its own flash partition (only needed after export) esptool.py --chip esp32s3 write_flash 0x110000 firmware/model/model.bin # 4. Watch it generate arduino-cli monitor -p /dev/ttyUSB0 -c baudrate=115200 ``` What comes out is a stream of short TinyStories-style prose at about 9.5 tokens per second, rendered on a small attached display. The repo has a demo GIF if you want to see it move before you buy a board. Treat the code and weights as an experiment: this is a research demo of a storage technique, not a maintained product, so pin your toolchain versions and expect to read the source. ## What it proves, and what it does not Here is the honest verdict, and it is two-sided on purpose. What it proves: you can hold a 28.9M-parameter model on a $8 microcontroller and run it, if you are willing to store most of it as a memory-mapped flash table and read only the rows each token needs. Per-Layer Embeddings, pulled from a research idea into a working ESP32 firmware, is a real and reusable technique. If your problem is "this model is a bit too big for the fast memory I have," moving the embedding capacity to a slower tier you read sparsely is a lever worth knowing. The ablation shows the capacity is doing real work, and it survives 4-bit quantization. What it does not prove: that a microcontroller can run a useful language model. This model writes toy stories. It will not answer a question, follow an instruction, call a tool, or know a single fact, because TinyStories does not teach any of that, and 28.9M parameters could not hold much of it anyway. The output category is exactly what part one described, just bigger and more coherent. And the moment you chase real capability you need a real vocabulary, a real instruction-tuned model, and a KV cache for context, and every one of those pushes you straight back off the microcontroller and onto a Raspberry Pi or a phone. Part one's cheat sheet still stands: put the reflex on the sensor, keep the brain where the memory is. ## Who should care, and what to watch If you build products on constrained hardware, this is not a component you ship. It is a technique you file away. The people who should actually clone the repo are hobbyists who want to see an LLM breathe on an $8 chip, and researchers hunting for ways to fit more capacity into fixed memory budgets. The one thing worth watching: whether anyone applies the same flash-resident PLE trick to a small instruction-tuned model rather than a story generator. That is the experiment that would test whether this is a curiosity or a path. Until then, it is a very good curiosity, and a clean demonstration that on tiny hardware the binding constraint is always memory. ## A community fork put it on an 8MB board Since this series is about what actually runs, here is a data point from the wild. A developer forked the project to get it onto a Seeed XIAO ESP32-S3, a board with only 8MB of flash instead of the 16MB the stock 14.9MB model needs. The full model does not fit, so the fork retrains a smaller variant with an 8,192-token vocabulary instead of 32,768. That drops the model to about 8 million parameters and a 4.12MB file, which fits an 8MB board with room to spare. Same Per-Layer Embeddings architecture, same TinyStories training, just a smaller tokenizer. It works, and here is the twist: on the XIAO ESP32-S3 the fork reports about 11.97 tokens per second (82.9 ms per token), faster than the original's 9.5 tok/s. The reason is exactly the bottleneck described above. The output head is PSRAM-bandwidth-bound, and a smaller vocabulary means a smaller head: 0.82MB staged versus 2.43MB in the original. Shrink the thing that dominates the per-token memory scan and the whole model speeds up. It is this article's thesis confirmed on real silicon: on this hardware the binding constraint is memory bandwidth, so the size of the vocabulary head moves the number more than anything else. A sample of what the 8MB build wrote, unedited: Once upon a time, there was a little girl named Lily. She loved to play with her toys and run around in the park. One day, she found a big, red ball in the park. She wanted to play with it, but her mom said no. The fork, with an 8MB partition table, a PlatformIO build, and the smaller re-export, is on GitHub: [karamble/esp32-ai](https://github.com/karamble/esp32-ai/tree/xiao-esp32s3-8mb-variant). ## Sources and further reading - [esp32-ai repository (slvDev)](https://github.com/slvDev/esp32-ai) - [RESULTS.md: methodology, ablations, and measurements](https://github.com/slvDev/esp32-ai/blob/main/RESULTS.md) - [TinyStories dataset (Hugging Face)](https://huggingface.co/datasets/roneneldan/TinyStories) - [Gemma 3 model card (Per-Layer Embeddings)](https://ai.google.dev/gemma/docs/core/model_card_3) - [ESP32-S3 datasheet (Espressif)](https://documentation.espressif.com/esp32-s3_datasheet_en.html) - [Part one: edge AI on microcontrollers, what actually fits](/tutorials/edge-ai-on-microcontrollers.html) The original 28.9M model's figures are author-reported from the repository's RESULTS.md (2026-07-21). The 8MB-board figures (Seeed XIAO ESP32-S3, vocab-8192 re-export, about 11.97 tok/s) are reported by the community fork linked above. Date checked: 2026-07-24 --- # Scale Your n8n Chatbot With a Supabase Vector Database (RAG) Source: https://singularitybyte.com/tutorials/scale-n8n-chatbot-supabase-vector-database-rag-2026.html A folder of markdown files makes a fine chatbot right up until it does not. Add enough documents and the agent can no longer keep every file name in its head, and it starts answering from the wrong one. The fix is not a bigger prompt. It is retrieval by meaning: embed your documents into vectors, store them in a database, and let the agent search for the most relevant chunks instead of guessing a file name. This is retrieval-augmented generation, and Supabase gives you the vector database for it without leaving the stack you already run. This is part two of a two part thread. Part one built [a website chatbot backed by markdown files](tutorials/build-website-knowledge-chatbot-n8n-markdown-2026.html). Everything there still stands: the same AI Agent, the same Chat Trigger, the same website widget. We change exactly one thing, the tool the agent uses to find information, and in return the chatbot scales from a dozen documents to thousands. ## What Actually Changes Part Markdown chatbot Vector RAG chatbot Knowledge store Files on disk Supabase table of embedded chunks Ingestion None, just edit files A workflow that embeds and upserts Retrieval Read a whole named file Semantic search for top matching chunks Agent, memory, Chat Trigger, website embed Identical. Only the retrieval tool is swapped. Because only the tool changes, you can run both side by side and migrate when you are ready. The website embed from part one does not change at all. ## Prerequisites You need n8n, Ollama, and Supabase. The [local-ai-packaged stack](tutorials/run-full-local-ai-stack-with-local-ai-packaged.html) ships all three together, which is why we recommend it for this series. Pull an embedding model alongside your chat model: ``` ollama pull nomic-embed-text ollama pull qwen2.5:7b-instruct-q4_K_M ``` nomic-embed-text turns text into 768 dimensional vectors and runs comfortably on a CPU. It is the embedding model the reference workflows use, and the vector width matters in the next step. ## Step 1: Turn Supabase Into a Vector Store Supabase is Postgres, and Postgres becomes a vector database with the pgvector extension. In the Supabase SQL editor, enable the extension, create a table for embedded chunks, and add the search function n8n calls. Run this once: ``` create extension if not exists vector; create table documents ( id bigserial primary key, content text, metadata jsonb, embedding vector(768) ); create function match_documents ( query_embedding vector(768), match_count int default null, filter jsonb default '{}' ) returns table ( id bigint, content text, metadata jsonb, similarity float ) language plpgsql as $$ begin return query select id, content, metadata, 1 - (documents.embedding <=> query_embedding) as similarity from documents where metadata @> filter order by documents.embedding <=> query_embedding limit match_count; end; $$; ``` Two details decide whether this works. The vector width, 768, must match your embedding model exactly, or inserts fail. And match_documents is the function n8n's Supabase Vector Store node calls by name, so do not rename it without updating the node. ## Step 2: The Ingestion Loop Now build a second workflow whose only job is to get documents into that table. This is the part markdown lookup did not have. A production version, like the one behind the Decred knowledge assistant, walks a folder, skips anything it has already embedded, and re-embeds only what changed. The node chain looks like this: ``` [ "Manual or Schedule Trigger", "Read Files from Disk (glob /data/shared/**/*.md)", "Loop Over Items (Split in Batches)", "Hash File (crypto, SHA-256)", "Check hash against document_metadata (Supabase)", "Switch: new file / unchanged / changed", "Extract Document Text", "Supabase Vector Store (insert)" ] ``` The Supabase Vector Store node in insert mode does the heavy lifting. Attach three sub-nodes to it: - Embeddings Ollama pointed at nomic-embed-text. This turns each chunk into the 768 dimensional vector the table expects. - Default Data Loader to carry document metadata (file name, source URL, hash) alongside the text. - Recursive Character Text Splitter with a chunk size around 400 and a small overlap. Chunking is what makes semantic search precise: the agent retrieves a paragraph, not a whole file. The dedup step is what keeps a growing knowledge base cheap. Hash each file, store the hash in a document_metadata table, and on the next run compare. Unchanged files are skipped, changed files get their old vectors deleted and re-inserted. Without this, every run re-embeds everything. ## Step 3: Swap the Retrieval Tool Back in the chatbot workflow from part one, delete the lookup_document tool and add a Supabase Vector Store node in retrieve-as-tool mode. Give it the same documents table, the match_documents query name, and its own Embeddings Ollama sub-node, again nomic-embed-text, so the query is embedded the same way the documents were. Name the tool something the agent will understand, like search_knowledge, and set a topK of 4 or 5 to start. That is the number of chunks the search returns per question. The system prompt gets simpler than the markdown version, because the agent no longer needs a list of file names: ``` You answer questions about MySite from a knowledge base. Use search_knowledge to find relevant information before answering. Answer only from what the search returns. If it returns nothing useful, say you do not have that information. Never invent product details. ``` The chat model, the memory with Session ID set to From input, and the Chat Trigger are untouched. Reactivate the workflow. The website widget from part one keeps talking to the same chat URL, and now every answer is backed by semantic search. ## Why the Numbers Matter Three settings shape answer quality, and all three are worth a moment. Setting Start with What it trades Chunk size 400 characters Smaller is more precise but loses context; larger keeps context but blurs the match Chunk overlap 40 to 80 characters Overlap stops ideas from being cut in half at a boundary topK 4 to 5 More chunks give the model more to work with, but add noise and tokens If answers feel thin, raise topK before you touch chunk size. If answers wander, lower it. Retrieval tuning is empirical, so change one number at a time and re-test the same handful of questions. ## Local or Cloud Everything above runs locally on Ollama, which keeps your documents and your queries on your own hardware and costs nothing per call. If you would rather trade privacy for a bit more accuracy, swap the two Ollama sub-nodes for OpenAI equivalents: an OpenAI Embeddings node (remember to change the table to vector(1536) to match) and an OpenAI Chat Model. The rest of the workflow, and the entire website embed, stay exactly the same. That is the point of building it this way: the pieces are independent, so you can upgrade one without rebuilding the others. ## Where You Land You now have a chatbot on your website that searches a real vector database by meaning, dedupes its own ingestion, and runs on hardware you control. It scales to thousands of documents, answers from specific paragraphs rather than whole files, and shares its plumbing with anything else in your n8n instance that reads the same documents table. The natural next step is to let other systems reach this knowledge base, not just the website widget. That is exactly what [turning the workflow into an MCP tool](tutorials/n8n-mcp-server-expose-workflow-as-ai-tool-2026.html) does, so an AI agent elsewhere can query your knowledge base directly. ## Sources and Further Reading - [Supabase vector columns and pgvector](https://supabase.com/docs/guides/ai/vector-columns) - [n8n Supabase Vector Store node](https://docs.n8n.io/integrations/builtin/cluster-nodes/root-nodes/n8n-nodes-langchain.vectorstoresupabase/) - [nomic-embed-text on Ollama](https://ollama.com/library/nomic-embed-text) - [local-ai-packaged on GitHub](https://github.com/coleam00/local-ai-packaged) Back to the start of the thread: [the simple markdown chatbot](tutorials/build-website-knowledge-chatbot-n8n-markdown-2026.html), if you skipped straight here. Build that first, then come back and swap in the vector tool. It is a fifteen minute change once the ingestion loop is running. --- # Build a Website Knowledge Chatbot With n8n (No Vector Database Needed) Source: https://singularitybyte.com/tutorials/build-website-knowledge-chatbot-n8n-markdown-2026.html You do not need a vector database to put a useful AI chatbot on your website. If your knowledge fits in a folder of markdown files, an n8n agent can read the right file on demand and answer from it, and a small JavaScript snippet drops the chat window onto any page. This is the fastest path from "I have some docs" to "there is a chatbot on my site that knows them," and it runs entirely on your own machine. This is the first of a two part thread in the n8n Automation Stack series. Here we build the simple version: markdown files as the knowledge base, a chat widget on your site. The [follow up swaps the markdown lookup for a Supabase vector database](tutorials/scale-n8n-chatbot-supabase-vector-database-rag-2026.html) when your knowledge outgrows a handful of files. If you are new to n8n, start with [what n8n is](tutorials/what-is-n8n-developer-introduction-2026.html) and [how to install it](tutorials/how-to-install-n8n-docker-npm-bundled-stacks-2026.html). ## What You Are Building Three pieces, wired together in one n8n workflow: Piece n8n node Job Knowledge base Files on disk One markdown file per topic in a shared folder Lookup tool Sub-workflow + Tool node Reads the file the agent asks for and returns its text Brain AI Agent + Chat model Picks a document, reads it, answers the question Front door Chat Trigger A public webhook the website chat widget talks to No embeddings, no database migrations. The agent does keyword-level reasoning over whole documents, which is plenty when you have a dozen or two well-named files. ## Prerequisites You need a running n8n with a local model runtime. The easiest way to get both, plus everything the follow up article needs, is the bundled stack we covered in [our local-ai-packaged guide](tutorials/run-full-local-ai-stack-with-local-ai-packaged.html). It brings up n8n, Ollama, and Postgres together in one command. Pull a small instruct model once it is running: ``` ollama pull qwen2.5:7b-instruct-q4_K_M ``` That is the model doing the talking in this build. Anything in the 7B to 14B instruct range works; smaller is faster, larger follows instructions more reliably. ## Step 1: The Knowledge Base Is Just Files Create a folder that n8n can read and drop one markdown file per topic into it. n8n containers usually mount a shared directory at /data/shared, so a clean layout looks like this: ``` /data/shared/mysite/docs/ getting-started.md pricing.md returns-policy.md api-authentication.md ``` The file name is the lookup key, so name files after the questions people ask. There is no ingestion step and nothing to rebuild. Edit a file, save it, and the next answer uses the new text. That immediacy is the whole appeal of this approach. ## Step 2: A Sub-Workflow That Reads One Document The agent should not touch the filesystem directly. Instead, build a small second workflow that takes a document name and returns its text, then hand that workflow to the agent as a tool. Keeping it separate means the agent can only read files through this one narrow door, and you can test the door on its own. The sub-workflow is three nodes behind an Execute Workflow Trigger whose input is a single field, document: ``` { "nodes": [ { "name": "When Called", "type": "n8n-nodes-base.executeWorkflowTrigger" }, { "name": "Build File Path", "type": "n8n-nodes-base.set" }, { "name": "Read Document", "type": "n8n-nodes-base.readWriteFile" }, { "name": "Extract Text", "type": "n8n-nodes-base.extractFromFile" } ] } ``` The Set node turns the document name into a safe path. Sanitising the name matters: it stops the agent from walking out of your docs folder with a crafted value. Use an expression on a single string field named filePath: ``` =/data/shared/mysite/docs/{{ $json.document.toLowerCase().replace(/[^a-z0-9_-]/g, '-') }}.md ``` Read Document points at {{ $json.filePath }}, and Extract Text runs in text mode so the output is plain markdown the model can read. That is the entire tool. ## Step 3: The Agent Now the main workflow. Drop in an AI Agent node and give it three attachments: a chat model, a memory, and the tool you just built. - Chat model: an Ollama Chat Model node pointed at qwen2.5:7b-instruct-q4_K_M. - Memory: a Simple Memory (or Postgres Chat Memory) node with Session ID set to From input, so each website visitor keeps their own conversation. - Tool: a Call n8n Workflow Tool node pointing at the sub-workflow from step 2. Name it lookup_document and describe it clearly. The description is what the agent reads to decide when to call the tool, so it earns its keep. A system prompt on the agent ties it together: ``` You answer questions about MySite using a document library. Available documents: getting-started, pricing, returns-policy, api-authentication. To answer, call lookup_document with the single most relevant document name, then answer only from what it returns. If no document fits, say so plainly. Never invent product details. ``` Listing the available document names in the prompt is the trick that makes file-based lookup work. The model maps the question to a file, calls the tool, and answers from the returned text. ## Step 4: The Front Door Replace any manual trigger with a Chat Trigger node (When chat message received). This node is what the website widget talks to. Two settings matter: Setting Value Why Make Chat Publicly Available On The widget calls it from a browser with no n8n login Allowed Origins (CORS) https://yoursite.com Only your site may open the chat; blocks everyone else Leave the response mode at the default so the Chat Trigger streams the agent's answer straight back to the widget. Turn off Append n8n Attribution in the node options for a clean embed. Activate the workflow, then open the node and copy its chat URL. It has this shape: ``` https://your-n8n-host/webhook//chat ``` Test it from n8n's built-in chat panel before touching your website. Ask something that should hit a document and confirm the agent calls lookup_document and answers from the file. ## Step 5: Put the Chat on Your Website n8n publishes an official chat widget, @n8n/chat, under an MIT license. You load it from a CDN and initialise it with one function call. Add this to any page: ``` ``` That is the exact shape a real deployment uses. A few options are worth understanding: Option Effect webhookUrl The Chat Trigger URL from step 4. This is the only required option. mode window gives a floating bubble in the corner; fullscreen mounts the chat into a target element. chatSessionKey Must match the Chat Trigger. The widget generates a session id so returning visitors keep context, which is why the agent memory used From input. initialMessages / i18n The greeting bubble and the widget labels. Set them to your brand voice. If the widget loads but every message fails, the cause is almost always CORS. The origin you are embedding on must be listed in the Chat Trigger's Allowed Origins. That single field is the most common reason a working chat looks broken on the live site. ## Where This Runs Out of Road File-based lookup is honest about what it is. The agent matches the question to a file name and reads the whole file, so it works best when documents are short, topics are cleanly separated, and there are not too many of them. Past roughly a few dozen files, listing every name in the system prompt stops being practical and the model starts guessing the wrong file. It also cannot answer across documents or find a paragraph buried in a long one. There is no semantic search, only whole-file retrieval. When you hit that wall, you do not throw this away: you keep the same agent, the same Chat Trigger, and the same website embed, and you swap the file-reading tool for a vector search tool. That is exactly what [part two](tutorials/scale-n8n-chatbot-supabase-vector-database-rag-2026.html) does with Supabase and embeddings. ## Sources and Further Reading - [@n8n/chat widget on npm](https://www.npmjs.com/package/@n8n/chat) - [n8n Chat Trigger node documentation](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-langchain.chattrigger/) - [n8n AI Agent and tools tutorial](https://docs.n8n.io/advanced-ai/intro-tutorial/) - [local-ai-packaged on GitHub](https://github.com/coleam00/local-ai-packaged) Next in the series: [scaling this chatbot with a Supabase vector database](tutorials/scale-n8n-chatbot-supabase-vector-database-rag-2026.html), so it can search thousands of documents by meaning instead of matching file names. The website widget does not change. Only the tool behind it does. --- # How to Install n8n: Docker, npm, and Two Bundled AI Stacks Source: https://singularitybyte.com/tutorials/how-to-install-n8n-docker-npm-bundled-stacks-2026.html There are five realistic ways to get n8n running, and the one you pick determines how much pain you feel later. A throwaway Docker container takes 30 seconds and loses everything on restart. A Compose file with Postgres takes ten minutes and survives reboots, upgrades, and backups. Two bundled stacks install n8n alongside a local model runtime in one command, at the cost of pulling in a lot of software you may not want. This is part two of the n8n Automation Stack series. Part one covered [what n8n is and the four concepts behind it](/tutorials/what-is-n8n-developer-introduction-2026.html). Here we get it installed properly, including the environment variables that decide whether your webhooks work behind a reverse proxy. Part three depends on getting that last part right. ## The Five Paths Compared Path | Setup time | Database | Best for Docker, throwaway | 30 seconds | SQLite, discarded | Clicking around for the first time Docker with a volume | 2 minutes | SQLite, persisted | Personal use, homelab, single user Compose with Postgres | 10 minutes | PostgreSQL | Anything you would be annoyed to lose npm | 5 minutes | SQLite | Node developers, custom node work Bundled AI stack | One command | PostgreSQL | You want a local model runtime too ## Path 1: Docker With a Volume This is the right default for a single user. The only change from the throwaway command in part one is a named volume and dropping --rm: ``` docker volume create n8n_data docker run -d --name n8n \ -p 5678:5678 \ -v n8n_data:/home/node/.n8n \ docker.n8n.io/n8nio/n8n ``` Everything lives in that volume: the SQLite database, your credentials, and the encryption key. Back up the volume and you have backed up the instance. ## Path 2: Docker Compose With Postgres n8n defaults to SQLite. It supports PostgreSQL as the alternative, and MySQL and MariaDB were deprecated back in v1.0, so Postgres is the only real choice once you outgrow a single file. Move to Postgres when you want concurrent writes, a backup story that is not a file copy, or queue mode later. ``` services: postgres: image: postgres:16 restart: unless-stopped environment: POSTGRES_DB: n8n POSTGRES_USER: n8n POSTGRES_PASSWORD: change_me volumes: - pg_data:/var/lib/postgresql/data n8n: image: docker.n8n.io/n8nio/n8n restart: unless-stopped ports: - "5678:5678" environment: DB_TYPE: postgresdb DB_POSTGRESDB_DATABASE: n8n DB_POSTGRESDB_HOST: postgres DB_POSTGRESDB_USER: n8n DB_POSTGRESDB_PASSWORD: change_me N8N_ENCRYPTION_KEY: put_a_long_random_string_here volumes: - n8n_data:/home/node/.n8n depends_on: - postgres volumes: pg_data: n8n_data: ``` Bring it up with docker compose up -d. One refinement worth knowing: n8n supports a _FILE suffix on individual variables, so DB_POSTGRESDB_PASSWORD_FILE can point at a file instead of putting the password in the Compose file. ## Path 3: npm Useful if you are writing custom nodes and want the code on your host rather than inside a container: ``` npm install n8n -g n8n start ``` Data goes to ~/.n8n by default. You can move it with N8N_USER_FOLDER. You are responsible for the service manager, the Node version, and the upgrade path, which is why most people end up on Docker anyway. ## Back Up the Encryption Key Before You Do Anything Else This is the single highest-consequence detail on this page. n8n encrypts stored credentials with a key. If you do not set N8N_ENCRYPTION_KEY, n8n generates a random one on first launch and writes it into the user folder. Restore your database onto a fresh instance without that key and every stored credential is unreadable. The workflows come back, the connections do not. Set it explicitly from the start and store it wherever you keep secrets. Note also that key rotation exists but is gated behind N8N_ENV_FEAT_ENCRYPTION_KEY_ROTATION, and the docs describe it as a one-way change that wants a full database backup first. Getting it right on day one is much cheaper. ## The Variables That Decide Whether Webhooks Work Run n8n on localhost and everything works. Put it behind nginx, Caddy, or Apache with TLS, and webhooks start handing out URLs nobody can reach. The cause is always the same: n8n does not know its own public address. Variable | Default | What it does N8N_HOST | localhost | Host name n8n believes it is served from N8N_PORT | 5678 | Port it binds N8N_PROTOCOL | http | Protocol it advertises WEBHOOK_URL | none | Public webhook base URL when behind a proxy N8N_EDITOR_BASE_URL | none | Public URL of the editor front end For an instance served at https://automation.example.com, the working set looks like this: ``` N8N_HOST=automation.example.com N8N_PROTOCOL=https N8N_PORT=5678 WEBHOOK_URL=https://automation.example.com/ N8N_EDITOR_BASE_URL=https://automation.example.com/ ``` TLS terminates at the proxy. n8n keeps speaking plain HTTP on 5678 behind it, and N8N_PROTOCOL=https only tells n8n what to advertise in the URLs it generates. One more proxy setting matters, and it is the one that breaks part three of this series: response buffering must be off. n8n's MCP server transport uses Server-Sent Events, and a buffering proxy will hold the stream until it times out. In nginx that is proxy_buffering off; on the relevant location block. ## Path 4: local-ai-packaged If you want n8n and a local model runtime together, local-ai-packaged bundles n8n, Ollama, Postgres, Qdrant, and a few other services behind a single command. You get a working local RAG setup without wiring five containers yourself. The trade-off is that you inherit the whole stack, including parts you may not use, and its upgrade cadence rather than n8n's. We walked through the full install, the hardware acceleration profiles, and a first local RAG agent in [our local-ai-packaged guide](/tutorials/run-full-local-ai-stack-with-local-ai-packaged.html). ## Path 5: ODS ODS takes the same idea further and installs a private AI server in one command, with hardware auto-detection choosing a model tier for you. n8n is one of the services it brings up, alongside llama-server, Open WebUI, Whisper, Qdrant, and SearXNG. Pick this when the goal is a private AI server that happens to include automation, rather than an automation server that happens to include AI. Details, including what the installer actually does, are in [our ODS write-up](/tutorials/ods-private-ai-server-one-command.html). ## Lock It Down Before You Expose It An n8n instance holds credentials for every service it touches. Treat it as one of the more sensitive boxes you run. Do not publish port 5678 to the internet directly. Bind it to localhost or a private network and let the reverse proxy be the only public listener. In Compose that means 127.0.0.1:5678:5678 instead of 5678:5678. Three settings are worth knowing about: Variable | Default | Why you care N8N_SECURE_COOKIE | true | Cookies only over HTTPS. The usual reason people turn this off is testing over plain HTTP. Turn it back on. N8N_BLOCK_ENV_ACCESS_IN_NODE | false | Off by default, meaning expressions can read the host environment. Set it to true if anyone else edits workflows. N8N_RESTRICT_FILE_ACCESS_TO | none | Confines filesystem access to directories you name. That second one deserves emphasis. With the default setting, anyone who can edit a workflow can read your environment variables, which is where your database password lives. ## Upgrading and Rolling Back On Docker, an upgrade is a pull and a recreate: ``` docker compose pull docker compose up -d ``` Two habits make this safe. Pin a specific image tag rather than tracking latest, so an upgrade is a deliberate act you can reverse by editing one line. And snapshot the database before a major version jump, because n8n runs schema migrations on start and those do not roll back cleanly. ## Which Path Should You Pick Running it for yourself on a homelab box: Docker with a volume. Anything other people depend on: Compose with Postgres, an explicit encryption key, and pinned tags. Writing custom nodes: npm. Already planning to run local models: one of the bundled stacks, which saves you a day of container wiring. Whichever you choose, set N8N_ENCRYPTION_KEY now and put it somewhere you will still be able to find it in a year. ## Sources and Further Reading - [n8n self-hosting documentation](https://docs.n8n.io/hosting/) - [n8n environment variable reference](https://docs.n8n.io/deploy/host-n8n/configure-n8n/basic-configuration/use-environment-variables/) - [n8n database configuration](https://docs.n8n.io/deploy/host-n8n/configure-n8n/basic-configuration/use-environment-variables/database.md) - [local-ai-packaged on GitHub](https://github.com/coleam00/local-ai-packaged) Next in the series: [turning one of these workflows into an MCP tool](/tutorials/n8n-mcp-server-expose-workflow-as-ai-tool-2026.html) an AI agent can call directly. If your instance sits behind a proxy, check proxy_buffering before you start. --- # What Is n8n? A Developer Introduction to Workflow Automation Source: https://singularitybyte.com/tutorials/what-is-n8n-developer-introduction-2026.html n8n is a workflow automation tool you run on your own hardware. You wire together triggers, API calls, and code steps on a canvas, and it executes them on a schedule, on a webhook, or on demand. If you have ever glued three services together with a cron job and a 200-line Python script that nobody else can maintain, n8n is aimed squarely at you. This is the first article in our n8n Automation Stack series. It covers what n8n is, the four concepts you need before anything else makes sense, what changed in 2026, and when reaching for n8n is the wrong call. No workflow building yet. That starts in the next two parts. ## What n8n Actually Is, and What It Is Not n8n is a node-based automation engine with several hundred built-in integrations. You self-host it, your credentials stay in your database, and every workflow is JSON you can export and commit to git. That last property is the one that matters most for developers, and it is the main thing hosted automation tools do not give you. Here is the part most introductions skip: n8n is not open source. It is source-available under the Sustainable Use License v1.0. You can read the code, modify it, and self-host it, but the licence sets explicit boundaries: You may use or modify the software only for your own internal business purposes or for non-commercial or personal use. You may distribute the software or provide it to others only if you do so free of charge for non-commercial purposes. In practice that means running n8n for your own company or your own projects is fine. Wrapping it up and reselling it as a hosted service is not. Separately, any source file with .ee. in its filename or .ee in its directory name sits outside that licence entirely and needs a commercial Enterprise licence. We cover open-weights models and genuinely open tooling here, so it is worth being precise rather than calling everything self-hostable "open source". n8n is fair-code. For most readers that distinction changes nothing. If you are building a product on top of it, read the licence before you write the business plan. ## The Four Concepts That Make n8n Click Almost every point of confusion for new users comes from missing one of these four ideas. ### Nodes A node is one step. It might call an HTTP endpoint, run a snippet of JavaScript or Python, branch on a condition, or talk to Postgres. Data flows between nodes as an array of JSON items, and this is the single most important thing to internalise: a node does not receive one object, it receives a list. If your workflow mysteriously runs five times, it is because the node upstream handed it five items. ### Triggers A trigger is how a workflow starts. The common ones are schedule (cron), webhook (an inbound HTTP request), and manual execution from the editor. A workflow has exactly one active trigger path. Getting the trigger right is most of the design work. ### Credentials Credentials are stored separately from workflows and referenced by name. This is why you can export a workflow to JSON and share it without leaking your API keys. It also means the encryption key that protects those credentials is the single most important thing to back up. Lose it and every stored credential becomes unreadable, even with an intact database. We come back to this in part two. ### Executions Every run is recorded with its input and output at each node. When something breaks at 3am, the execution log tells you which node failed and exactly what data it received. This is the feature that makes n8n worth using over a shell script, and it is the one people discover last. ## Test Runs and Production Runs Are Not the Same Thing n8n distinguishes between executing a workflow from the editor and running a published one. They use different URLs, and this trips up almost everyone the first time they build a webhook. A test execution shows live data in the editor, which is what you want while building. A production execution does not display in the editor at all. You inspect it afterwards through the Executions tab. If you have ever pointed an external service at an n8n webhook and watched nothing happen, you were almost certainly still on the test URL, which only listens for a single request. Remember this one. It becomes load-bearing in part three, when we expose a workflow as a tool over the network. ## What Changed in n8n During 2026 If your mental model of n8n is from 2024, several things are different. - The AI Agent node was rebuilt with tool calling across Claude, GPT-4o, Gemini, Mistral, Groq, and any OpenAI-compatible endpoint. That last one is what lets you point it at a local model. - Four memory node types shipped: in-memory, Redis, Postgres, and Motorhead. Conversation state no longer has to be something you hand-roll. - The Canvas UI arrived in version 1.30 and replaced the flat left-to-right builder. Nodes can be grouped into labelled clusters and collapsed, which matters once a workflow passes about 20 nodes. - Per-node retry with exponential backoff is now built in, instead of being something you construct out of error branches. - Execution replay landed in June 2026 and lets you trace variables line by line through JavaScript and Python nodes on a failed run. - MCP support means n8n can act as a Model Context Protocol server or client. This is the subject of part three. ## When n8n Is the Right Tool It is not always. Being honest about this saves you a rewrite later. Approach Best when Falls down when Cron plus a script One trigger, one action, no credentials to manage You need retries, logging, or a second maintainer n8n Several services, scheduled or event-driven, you want run history and self-hosting Sub-second latency, or logic better expressed as plain code Zapier or Make You want zero infrastructure and the volume is low Per-task pricing at scale, or data that cannot leave your network Application code The logic is the product You are rebuilding retries, scheduling, and an audit log by hand The honest failure mode of n8n is complex branching business logic. Once a workflow has 40 nodes and six conditional paths, it is harder to read than the equivalent 100 lines of code, and much harder to diff in review. Use it as glue between systems, not as a programming language. ## What Self-Hosting Actually Costs n8n itself is free to self-host under the terms above. The community edition has no execution limits, so the cost is your hardware and your time. A small instance handling a few thousand executions a month runs comfortably on 1 vCPU and 1 GB of RAM. Add Postgres instead of the default SQLite once you care about concurrent writes or want backups that are not a file copy. If you plan to run local models alongside it, the memory budget is dominated by the model, not by n8n. Our [full local AI stack guide](tutorials/run-full-local-ai-stack-with-local-ai-packaged.html) covers those numbers in detail. ## Your First Ten Minutes The fastest way to get a feel for the editor is a throwaway container. This is not how you should run it permanently, and part two explains why, but it is enough to click around: ``` docker run -it --rm --name n8n -p 5678:5678 docker.n8n.io/n8nio/n8n ``` Open http://localhost:5678, create the owner account, and build the smallest possible workflow: a Schedule Trigger into a Code node that returns { hello: "world" }. Execute it, then open the Executions tab and look at the recorded input and output. That loop of build, run, inspect is the whole tool in miniature. Note the --rm flag. Everything you build in that container disappears when you stop it, credentials included. That is deliberate for a first look and unacceptable for anything else. ## What Comes Next in This Series [Part two](tutorials/how-to-install-n8n-docker-npm-bundled-stacks-2026.html) covers setting n8n up properly: the barebones Docker and npm paths, the environment variables that decide whether your webhooks work behind a reverse proxy, and the two bundled harnesses that install n8n alongside a local model stack. [Part three](tutorials/n8n-mcp-server-expose-workflow-as-ai-tool-2026.html) turns a workflow into an MCP tool that an AI agent can call directly. If you would rather jump straight to building something, our [AI generator tutorial](tutorials/build-your-own-ai-generator-n8n-ollama-claude-api-2026.html) wires n8n to Ollama and the Claude API end to end. ## Sources and Further Reading - [n8n documentation](https://docs.n8n.io/) - [n8n Sustainable Use License](https://github.com/n8n-io/n8n/blob/master/LICENSE.md) - [n8n self-hosting documentation](https://docs.n8n.io/hosting/) - [n8n release notes](https://docs.n8n.io/release-notes/) --- # Turn an n8n Workflow Into an MCP Tool Your AI Agent Can Call Source: https://singularitybyte.com/tutorials/n8n-mcp-server-expose-workflow-as-ai-tool-2026.html An AI agent that has to click through your admin panel is slow, brittle, and breaks the moment you move a button. An agent that can call get_order_status(order_id) and get structured JSON back does not. The Model Context Protocol, or MCP, is the open standard that describes those callable tools, and n8n can now act as an MCP server: any workflow you build becomes a tool an agent can invoke over the network. This is part three of the n8n Automation Stack series. Part one covered [the core concepts](/tutorials/what-is-n8n-developer-introduction-2026.html), part two covered [installing n8n properly](/tutorials/how-to-install-n8n-docker-npm-bundled-stacks-2026.html). That second one matters here more than usual, because the MCP transport is the one thing a default reverse proxy configuration will silently break. ## What You Get - One n8n workflow exposed as an MCP tool with a typed input schema. - Bearer-authenticated access from Claude Desktop, Cursor, or any MCP client. - The four self-hosting failures that stop it working, and the fix for each. - A tested look at the second, instance-level MCP server: what it exposes, what it costs to run, and the security boundary you are trusting. ## Two Directions, Do Not Mix Them Up n8n sits on both sides of MCP, and the node names are easy to confuse. Node | Direction | Use it when MCP Server Trigger | n8n exposes tools | An external agent should call your workflow MCP Client Tool | n8n consumes tools | Your n8n AI Agent needs a third-party MCP server This article builds the first one. The second gets a paragraph at the end. ## Prerequisites A self-hosted n8n instance you can reach over HTTPS, an MCP client, and a workflow worth exposing. Pick something with a clear input and a clear output. A lookup against a database, a status check against an internal API, or a search over your own content all work well. Anything that takes ten parameters and returns prose does not. ## Step 1: Add the MCP Server Trigger Create a new workflow and add the MCP Server Trigger node. It replaces the usual trigger, so this workflow will not have a Schedule or Webhook node. The node gives you a path, which becomes part of the URL agents connect to, and an authentication choice. Two options are available, both configured through HTTP request credentials: - Bearer auth: a token in the Authorization header. Use this unless you have a reason not to. - Header auth: a custom header name and value, for fitting an existing scheme. Set a path you will recognise later. The node then shows you two URLs, and the difference between them is the same test versus production split from part one. ## Step 2: Attach a Tool The trigger on its own exposes nothing. Tools attach to it as sub-nodes, and two kinds can connect: - Custom n8n Workflow Tool, which exposes another of your workflows as a callable tool. - MCP Client Tool, which re-exposes tools from another MCP server. Use the first. Attach a Custom n8n Workflow Tool, point it at the workflow doing the actual work, and then spend your time on the two fields that decide whether an agent uses your tool correctly: the name and the description. The description is not documentation. It is the prompt the model reads when deciding whether to call your tool. Write it for the model: ``` { "name": "get_order_status", "description": "Look up the current status of a customer order by its ID. Returns status, carrier and tracking number. Use when the user asks where an order is.", "parameters": { "order_id": "The order ID, format ORD-12345" } } ``` Vague descriptions produce agents that either never call the tool or call it constantly with nonsense arguments. Say what it returns and when to use it. ## Step 3: Test Before You Publish Click Listen for Test Event and the node hands you a test URL. Point your MCP client at it and call the tool once. The test URL displays live data in the editor, which is the whole reason to use it. You can see the arguments the model sent and the data each node returned. Note that it listens for a single event, so if a second call appears to hang, that is why. ## Step 4: Publish and Connect for Real Publishing the workflow activates the production URL. Production executions do not display in the editor at all. To inspect them you open the Executions tab, which is where all your debugging happens from this point on. Point your MCP client at the production URL with the same credentials. In Claude Desktop that is an entry in the MCP servers section of the config; in Cursor and other clients it is equivalent. The tool should appear in the client's tool list with the name and description you set in step 2. ## Step 5: The Self-Hosting Reality Check This is the part that is not in most walkthroughs, and it is where a self-hosted instance behind a proxy stops working. ### There is no stdio transport Many MCP servers run as a local subprocess and speak over standard input and output. The n8n MCP Server Trigger does not support stdio. It is SSE and HTTP only, meaning your n8n instance has to be reachable over the network from wherever the client runs. If your client only supports stdio, you need a bridge process, not this node. ### Proxy buffering breaks it Server-Sent Events depend on the server pushing bytes as they happen. A reverse proxy that buffers responses will hold those bytes until the buffer fills or the connection times out, and the symptom is a connection that appears to open and then never delivers anything. The n8n documentation is explicit that a reverse proxy in front of the MCP endpoint must have buffering disabled and compression settings adjusted. In nginx that is proxy_buffering off; on the location handling the MCP path. In Apache, SetEnv proxy-sendchunked 1 and no output filters on that path. This is the same setting part two flagged, and it is the first thing to check when a connection opens but no tools appear. ### Multiple replicas break it differently If you run n8n with more than one webhook replica behind a load balancer, SSE connections fail when a follow-up request lands on a different replica than the one holding the stream. The fix is routing: send every request matching /mcp* to one dedicated replica. Single-container deployments are unaffected, which is most self-hosted setups. ### Fire tool calls one at a time We connected a real MCP client to a self-hosted instance and the first thing that broke was concurrency. Sending several tool calls in one parallel batch returned transport errors, some coming back as a bare "not connected" even though the server was up. Running the same calls one after another worked every time. This is the SSE transport doing exactly what the buffering and replica warnings above predict: a single streamed connection does not want several requests racing through it at once. If your client or your agent framework likes to fan out tool calls in parallel, throttle it to one in flight. A call that fails this way is worth a single sequential retry before you assume anything is actually wrong with the server. ## The Other Direction: Consuming MCP Servers The MCP Client Tool node is the mirror image. Attach it to an n8n AI Agent and the agent gains the tools of an external MCP server. It connects over an SSE endpoint, and it supports bearer, single or multiple custom headers, OAuth2, or no authentication at all. The useful detail is tool filtering. You can expose All tools from the server, a Selected subset, or All Except a blocklist. Handing a model 40 tools when it needs 3 measurably degrades tool choice, so select deliberately. ## A Third Thing, Which Is Not This n8n also has an instance-level MCP server, a different feature with a confusingly similar name. It lets an MCP client connect to n8n itself at /mcp-server/http and build, search, test, and publish workflows by prompting. Authentication is OAuth2 or a personal access token. We wired one into a client and used it, so the rest of this section is from that rather than from the docs. This is a development tool, not a runtime one. It is how you build workflows by talking to a model, not how an agent calls your workflow in production. ### What it actually exposes Connecting turns n8n into a toolbox with four kinds of tool, and the difference between them is entirely about blast radius: Family | Does | Cost of a mistake Read | List and inspect workflows, executions, credentials, tags | None Build | SDK reference, node search, config validation | None, it only reads and checks Write | Create, update, publish, archive, restore versions | Mutates your instance Execute | Run and test workflows | Runs real workflows with real credentials That last row is the one to respect. An execute call is not a dry run. If the workflow it triggers calls a paid model API, hits a database, or posts to a webhook, it does all of that for real. Point an agent at this server and "test my workflow" can mean spending money and sending outbound requests. ### Two things it does right Listing credentials returns names and types only, never secret values. You can let a model see that an OpenAI credential and a Postgres credential exist, and reference them by ID when building, without ever exposing the keys. That is the correct design and it held up in practice. It also refuses to let you guess. The server makes you pull the SDK reference and the best-practices guidance for a technique before it will accept workflow code, so a model cannot hallucinate node parameters straight into your instance. Building is slower as a result, and correct more often. ### The opt-in is real, and stricter than you expect Workflows are exposed individually. On the instance we connected to, a small minority of the total were flagged available over MCP; the rest were invisible to the connected client even though the same token could list them through the management tools. The exposure is also not client-scoped: every client using that token sees the same enabled set. Treat "enabled for MCP" as a deliberate per-workflow decision, not a default. ### The token is the whole security boundary A personal access token here is a long-lived bearer credential. The one we generated was a JWT with no expiry claim, so it does not time out on its own, and it grants read, build, write, and execute across everything the issuing user can reach. It sits in your client's config file in plain text. Which means: store it like a password, do not commit the config that holds it, and rotate it in n8n's settings if it ever leaks. Anyone who reads that file can drive your instance. One more caveat. Several third-party guides cite an enablement environment variable and a minimum version that do not match the official documentation, which describes the module as enabled by default and disabled with N8N_DISABLED_MODULES=mcp. Check your own instance before copying either claim. ## What This Is Actually Good For Two directions, two audiences. The Server Trigger from steps one through five is for exposing what you already built. If you run n8n and have workflows an agent should be able to trigger, an internal lookup, a status check, a search over your own content, wrapping them as tools is a thin layer over work that already exists. The instance-level server is the inversion: it is for building and auditing n8n from a chat window. Asking a model which workflows are active, which have not run since last quarter, or which are exposed over MCP is a fast way to take stock of an instance that has grown past what you remember. Building a new workflow through it means the model discovers nodes and validates them against your real credentials before anything is saved. It is the most useful when your instance is large enough that clicking through the editor has become the slow path. ## Limitations and Gotchas - No stdio transport. The instance must be network-reachable from the client. - SSE needs an unbuffered path end to end, including any CDN in front. - Concurrent tool calls over the SSE transport fail. Send one at a time. - Production executions are invisible in the editor. Use the Executions tab. - The tool description is prompt text and drives model behaviour. Treat it as code. - An execute call on the instance-level server runs the real workflow with real credentials. It is not a dry run. - The bearer token is the whole security boundary, is long-lived, and sits in a config file in plain text. Anyone who reads it can drive your instance. ## When Not To Bother Skip the Server Trigger if you only need an agent to call one HTTP endpoint. An MCP server wrapping a single API call adds a hop and a service to maintain. Point the agent at the API. Skip the instance-level server on a small instance you already know by heart. Its payoff scales with how many workflows you have forgotten about, and on a handful of workflows the editor is faster than a chat window. ## Sources and Further Reading - [n8n MCP Server Trigger node docs](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-langchain.mcptrigger/) - [n8n MCP Client Tool node docs](https://docs.n8n.io/integrations/builtin/cluster-nodes/sub-nodes/n8n-nodes-langchain.toolmcp/) - [Connect to the n8n MCP server](https://docs.n8n.io/connect/connect-to-n8n-mcp-server/) - [Model Context Protocol specification](https://modelcontextprotocol.io/) Ten minute version: add an MCP Server Trigger to any existing workflow, set bearer auth, attach a Custom n8n Workflow Tool, and call the test URL from your client. If nothing arrives, check proxy_buffering first. --- # Turn Your PC Into a Private AI Server in One Command with ODS Source: https://singularitybyte.com/tutorials/ods-private-ai-server-one-command.html You have assembled this stack by hand before. Ollama or llama.cpp for inference, Open WebUI on top, Whisper for voice, a vector database for RAG, n8n to wire it together, ComfyUI because why not. It takes a weekend, and it breaks the next time you touch a driver. ODS does the assembly for you: one command, hardware auto-detection, and a chat window in about two minutes. It is Apache 2.0, 3,446 stars, and it was still getting commits the day we wrote this. Here is what it actually installs, and what that install script actually does to your machine. ## First, the name confusion If you went looking for this project after seeing it in a July roundup, you were probably sent to a repo called DreamServer under an org called Light-Heart-Labs. That link still works, but it is a redirect. The project renamed itself: github.com/Light-Heart-Labs/DreamServer now returns an HTTP 301 to github.com/Osmantic/ODS, which stands for Osmantic Deployment System. Several aggregators have not caught up. If a tutorial tells you to clone DreamServer, it is stale but harmless. ## What you get ODS is a Docker Compose stack with a CLI wrapped around it. The pitch is turning a PC, Mac or Linux box into a private AI server, and the component list is genuinely broad: Layer | What ships Inference and chat | llama-server, LiteLLM gateway, Open WebUI, TEI embeddings Voice | Whisper (speech to text), Kokoro (text to speech) Agents and automation | Hermes Agent, n8n, OpenCode, Agent Policy Engine Knowledge and search | Qdrant, SearXNG, Perplexica Creative | ComfyUI Operations | Dashboard with GPU metrics, Privacy Shield, Token Spy, Langfuse Most of these we have covered individually. [OpenCode](/tools/opencode.html) is in there as the coding assistant, and the inference layer is the same llama.cpp lineage that sits under [Ollama](/news/ollama-v0-32-interactive-agent-2026.html). The value here is not any single component, it is that they come pre-wired with a gateway in front and one config file behind. ## The part that matters: what the installer does The documented install is a pipe from the internet into your shell, which should always make you pause: ``` curl -fsSL https://install.osmantic.com/ods.sh | bash ``` So we downloaded it and read it instead of running it. It is 507 lines of Bash. Here is the honest audit. Network calls. The only external hosts in the entire script are github.com (to clone the repo), install.osmantic.com (itself), and git-scm.com in a help message. There is no telemetry. We grepped for the usual suspects, analytics endpoints, PostHog, Segment, and any stray curl to a reporting URL. Nothing. Privilege. It runs as your user. sudo appears in exactly two situations: installing git if you do not have it, and force-removing root-owned container data during cleanup. It does not ask for root to run the stack. Where it lands. The install directory and repo URL are both environment variables, so you can redirect it before it touches anything: ``` # Read it first, then run it deliberately curl -fsSL https://install.osmantic.com/ods.sh -o ods.sh less ods.sh # Point it wherever you want ODS_INSTALL_DIR="$HOME/ai/ods" bash ods.sh ``` That is a clean bill of health for a curl-pipe-bash installer, which is not something we get to write often. It still deserves the same treatment as any other: download, read, then run. ## Hardware tiers ODS detects your accelerator and picks a default model rather than making you guess a quantization. The mapping, from the project's own table: Hardware | Memory | Default model | Context NVIDIA | 8 GB | Qwen3.5 2B (Q4_K_M) | 8K NVIDIA | 12 GB | Phi-4 14B (Q4_K_M) | 16K NVIDIA | 24 GB | Qwen3.5 27B (Q4_K_M) | 32K NVIDIA | 48 GB | DeepSeek R1 Distill Llama 70B | 32K Apple Silicon | 8 GB | Phi-4 Mini (Q4_K_M) | 128K Apple Silicon | 16 GB | Qwen3.5 9B (Q4_K_M) | 32K Apple Silicon | 64 GB+ | Qwen3.6 35B-A3B (UD-Q4_K_M) | 128K AMD Strix Halo | 64 GB unified | Qwen3.6 35B-A3B (UD-Q4_K_M) | 128K Intel Arc | 6 to 8 GB | Phi-4 Mini / Qwen3.5 9B | 128K / 32K Tiers as documented by the project. We did not verify model selection on each backend. Bootstrap mode is the nice touch: it starts you on a small model so the chat UI answers within a couple of minutes, then pulls the full tier model in the background. ## Driving it The ods CLI is the reason this beats a hand-rolled compose file. Services are toggles, not YAML edits: ``` ods status # health checks and GPU state ods list # every service and whether it is up ods enable n8n # turn a service on ods disable whisper # turn one off ods logs llm # tail a service ods model swap T3 # jump to another hardware tier ods mode local # local, cloud or hybrid inference ods preset save # snapshot the whole config ``` Open WebUI lands on localhost:3000. The llama-server API is on 11434 under Docker on Linux, or 8080 when it runs natively on macOS and Windows, and every port is overridable through the environment. Local is the default mode. Cloud providers are opt-in, either with --cloud at install time or ods mode later, and the project is explicit that your prompts stay on your machine unless you choose otherwise. Given how the local-first label has been stretched lately, including by Ollama shipping a cloud-hosted default agent in v0.32, an installer that ships local-only and makes you ask for the cloud is worth noting. ## Requirements and gotchas - Docker is mandatory everywhere. On Windows that means Docker Desktop with the WSL2 backend; on macOS, Docker Desktop plus Apple Silicon, since Intel Macs are not supported. - On macOS, llama-server runs natively for Metal while everything else runs in containers, so it is a hybrid deployment rather than a pure container stack. - The tier selector deliberately avoids Qwen3 Coder Next on unified-memory hosts, citing correctness issues on those backends. - 187 open issues against 3,446 stars. Normal for a young project moving fast, but this is not a set-and-forget appliance yet. - The last tagged release is v2.5.3 from May 26, while commits landed as recently as the day we published. Running from main gets you fixes the tag does not have, with the usual tradeoff. - Disk usage is not documented, and it will be substantial: a dozen container images plus a quantized model in the tens of gigabytes. - No GPU means CPU fallback, which works but is slow enough that the project recommends cloud mode instead. ## Who should run it Run ODS if you want the whole private-AI stack rather than one piece of it, and you would rather spend the evening using it than wiring it. It suits a homelab box, a spare workstation with a decent GPU, or an Apple Silicon machine with memory to spare. The service toggles make it a reasonable way to try Perplexica or Qdrant without committing to another compose file you will maintain forever. Skip it if you only need inference, where plain llama.cpp or Ollama is less machinery for the same result, or if you are on hardware without an accelerator, where the whole point evaporates. And if you are on a shared or production server, read those 507 lines yourself before running them. We found nothing alarming, but "some blog checked it" is not an audit. ## Sources and further reading - [GitHub: Osmantic/ODS (formerly DreamServer)](https://github.com/Osmantic/ODS) - [The install script, in full](https://install.osmantic.com/ods.sh) - [ODS releases (latest tag v2.5.3)](https://github.com/Osmantic/ODS/releases) - [Open WebUI documentation](https://docs.openwebui.com) - [llama.cpp, the inference engine underneath](https://github.com/ggml-org/llama.cpp) Tested on: not deployed. We statically audited the 507-line installer (network calls, privilege escalation, telemetry, install paths) on Linux 6.1, and verified the repository metadata, redirect and release history against the GitHub API on 2026-07-22. We did not run the full stack: this bench has 8 GB of RAM and no GPU, which is below the lowest documented tier, so any performance numbers from it would have been misleading. Component lists, hardware tiers and port defaults are as documented by the project. Date checked: 2026-07-22 --- # Quantization Formats Explained: NVFP4, MXFP4, and FP8 Source: https://singularitybyte.com/tutorials/quantization-formats-explained.html OpenAI shipped gpt-oss in MXFP4. DeepSeek trained V3 in FP8. NVIDIA's Blackwell cards added native NVFP4, and vLLM, SGLang, and llama.cpp have spent early 2026 racing to support all of it. Underneath the acronym soup is one practical question: the quantization format you pick decides whether a model fits on your GPU at all, and how fast it runs once it does. This is the evergreen map of that landscape, with the bit layouts, the memory math, and an honest "what runs fast on what hardware" verdict. Numbers here are vendor/spec- or community-reported and labeled inline. ## The one formula that matters Quantization stores each weight in fewer bits. Everything else follows from that. To a first approximation: ``` VRAM (GB) = params_in_billions * bits_per_weight / 8 * 1.2 # +20% overhead # plus the KV cache, which grows with context length and batch size ``` So a format is not an abstract quality knob. It is the difference between a model loading and an out-of-memory error. Here is what the common formats do to three model sizes (weights only, before the KV cache). Model | FP16 (16-bit) | FP8 (8-bit) | 4-bit (INT4 / FP4) | Smallest GPU at 4-bit 8B (Llama 3.1) | ~17-19 GB | ~9-10 GB | ~5 GB | 8 GB consumer card 70B (Llama 3.3) | ~168 GB | ~84 GB | ~46 GB | 1x 48 GB (L40S) or 2x RTX 4090 120B (gpt-oss MoE) | ~288 GB | ~140 GB | ~81 GB | 1x H100 80 GB Figures are community-reported, rounded, and include roughly 20 percent overhead. The headline is stark: a 120B model that needs four H100s in FP16 squeezes onto a single 80 GB card at 4-bit. That is the entire reason this topic exists. ## The precision ladder Modern formats step down a ladder: FP16/BF16 (the baseline), FP8 (half the bytes), then the 4-bit floats MXFP4 and NVFP4 (a quarter). A floating-point number splits its bits into a sign, an exponent (dynamic range), and a mantissa (precision). Fewer bits means less of both, so the trick is choosing where to spend them and how to rescale blocks of values to claw accuracy back. Format | Sign/Exp/Mantissa | Block size | Scale format | Effective bits/weight FP8 E4M3 | 1 / 4 / 3 | per-tensor or per-channel | FP32/FP16 | 8 FP8 E5M2 | 1 / 5 / 2 | per-tensor or per-channel | FP32/FP16 | 8 MXFP4 | 1 / 2 / 1 (E2M1) | 32 | E8M0 (power-of-2) | ~4.25 NVFP4 | 1 / 2 / 1 (E2M1) | 16 | E4M3 + FP32 tensor scale | ~4.5 All four are spec-defined. Note that MXFP4 and NVFP4 use the identical 4-bit element (E2M1); the entire difference between them is how they scale blocks, which we get to below. ## FP8: the safe 2x FP8 is the least scary cut, and the one most production stacks reach for first. It comes in two flavors: E4M3 (4 exponent, 3 mantissa bits, max value 448) is used for weights and activations during inference; E5M2 (5 exponent, 2 mantissa, much wider range) is used for gradients during training. More mantissa for forward-pass precision, more exponent for gradient range. At 8B parameters and up, FP8 typically costs under 1 percent on MMLU versus BF16 (vendor-reported: Llama 3.1 8B drops 68.8 to 68.3, Llama 3.3 70B holds at 82.0). It halves memory and, on the right hardware, runs at full Tensor Core speed. This is not just an inference trick: [DeepSeek V3](/models/deepseek-v3-0324.html) was trained in FP8 at 671B scale with under 0.25 percent relative error versus BF16. The catch is hardware, which we will hit shortly: FP8 needs an NVIDIA Hopper card or newer to actually go fast. ## The FP4 twins: MXFP4 vs NVFP4 Plain 4-bit floats are nearly useless on their own. The E2M1 element can only represent values like 0, 0.5, 1, 1.5, 2, 3, 4, 6. One outlier in a weight matrix forces a coarse scale and flushes small values to zero. The fix is microscaling: split the tensor into small blocks and give each block its own shared scale factor, so each block adapts to its own local range. Both modern FP4 formats do this; they differ in how. Property | MXFP4 (OCP standard) | NVFP4 (NVIDIA) Element | E2M1 (4-bit) | E2M1 (4-bit), identical Block size | 32 elements | 16 elements Block scale | E8M0 (powers of 2 only) | FP8 E4M3 (fractional allowed) Second-level scale | None | FP32 per tensor Standard | Open (Microsoft, AMD, Arm, Intel, Meta, NVIDIA, Qualcomm) | Proprietary The crux is the scale format. MXFP4's E8M0 scale can only be a power of two, so adjacent block scales are 2x apart; if a block's true maximum sits between two powers of two, you either overflow or waste precision. NVFP4's smaller 16-element blocks and fractional E4M3 scales (plus a tensor-wide FP32 scale) track the real distribution far more tightly. It shows in the numbers: on Llama 3.1 8B, MXFP4 drops MMLU-Pro from 44.2 to 32.5, while NVFP4 only falls to 38.8 (community-reported). On large models the gap shrinks; NVFP4 on DeepSeek-R1 lands within about 1 percent of FP8 (vendor-reported). MXFP4's win is openness and adoption: [OpenAI's gpt-oss](/models/openai-gpt-oss-models.html) ships natively in MXFP4 (MoE weights only, attention and routing kept higher), which is how the 120B variant fits on a single 80 GB GPU. NVFP4's win is accuracy per bit, at the cost of being NVIDIA-only. ## The formats you are probably already using Most people running models locally have never touched FP4. They use GGUF K-quants through llama.cpp and Ollama, or AWQ/GPTQ through vLLM. These are INT-based and predate the new float formats, and they are still excellent. GGUF format | Bits/weight | Size (8B model) | Quality Q4_K_M | 4.89 | 4.58 GB | The popular sweet spot Q5_K_M | 5.70 | 5.33 GB | Near-lossless on most models Q6_K | 6.56 | 6.14 GB | Almost lossless Q8_0 | 8.50 | 7.95 GB | Effectively lossless Bits-per-weight figures are from the official llama.cpp quantize README. AWQ and GPTQ produce INT4 (W4A16) Hugging Face checkpoints for vLLM; GGUF is the one that also runs on CPU and, via MLX or Metal, on Apple Silicon. INT4 (AWQ) is competitive with FP4 on accuracy at 4-bit; the difference is that FP4 can also quantize activations and, on Blackwell, run them through dedicated hardware. For now, if you are on a Mac or an older GPU, GGUF Q4_K_M remains the default that just works. ## "Runs" versus "runs fast": the hardware catch This is the part the format names hide. A quantized model can load (memory savings) on almost anything, but it only runs fast where the GPU has dedicated hardware for that format. Mismatch the two and you get the file-size benefit with no speedup, because the runtime quietly upcasts to a precision the silicon understands. Format | Blackwell (B200, RTX 50) | Hopper (H100/H200) | Ada (RTX 4090) | Ampere (A100, RTX 30) FP16 / INT8 / INT4 | Fast | Fast | Fast | Fast FP8 | Fast | Fast | Loads, falls back to BF16 | No (INT8 max) MXFP4 | Fast | Emulated | Emulated | No NVFP4 | Fast | Emulated | Emulated | No The practical rules: FP8 needs Hopper or newer (the RTX 4090's Ada chip has the data type but no scaling hardware, so it silently runs BF16). The FP4 formats only accelerate on Blackwell; on a Hopper H100 an NVFP4 model fits in less memory but does not run faster. Ampere and older are stuck at INT8/INT4. Apple Silicon has none of these float formats in hardware and uses MLX or GGUF 4-bit over unified memory instead. Runtime examples once you have matched format to card: ``` # vLLM, FP8 on a Hopper or Blackwell card vllm serve meta-llama/Llama-3.1-70B-Instruct --quantization fp8 # llama.cpp, a GGUF Q4_K_M model on anything (CPU, Apple Silicon, any GPU) llama-cli -m Llama-3.1-8B-Q4_K_M.gguf -c 8192 -p "Explain quantization." ``` ## Accuracy versus compression The rough ladder, for large models: 8-bit is essentially free (under 1 percent loss), 4-bit costs a small but real amount (roughly 1 to 3 percent), and sub-4-bit starts to hurt. Two patterns matter. First, bigger models tolerate aggressive quantization better; the same MXFP4 that mangles an 8B model barely dents a 70B-plus one (DeepSeek-R1 in MXFP4 holds above 99.5 percent on several benchmarks, vendor-reported). Second, quantizing activations as well as weights (W4A4) is far harsher than weights-only, which is why NVFP4's accuracy-preserving scaling matters most there. If you are running a small model, stay at Q5 or higher; if you are running a 70B, 4-bit is genuinely fine. ## So which should you pick? Your hardware | Best format | Why Blackwell (RTX 50, B200) | NVFP4 | Only arch with fast FP4; best accuracy per bit Hopper (H100/H200) | FP8 | Native, near-lossless, 2x memory Ada / Ampere GPU | INT4 (AWQ/GPTQ) or GGUF | No FP8/FP4 hardware; INT is the fast path Apple Silicon | MLX 4-bit or GGUF Q4_K_M | Unified memory, no FP4/FP8 hardware CPU / old GPU | GGUF Q4_K_M | Runs everywhere, sane quality Weight quantization is only half the memory story. On long contexts the KV cache dominates, and that has its own quantization track; we covered one approach in our [Google TurboQuant writeup](/tools/google-turboquant.html). And if your target is a tiny device rather than a datacenter card, the calculus changes again, which we walked through in [edge AI on a sensor node](/tutorials/edge-ai-on-microcontrollers.html). Match the format to the silicon, do the memory math first, and you will know whether a model fits before you ever hit download. ## Sources and further reading - [NVIDIA: Introducing NVFP4](https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/) - [OCP Microscaling Formats (MX) spec](https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf) - [Hugging Face: MXFP4 quantization](https://huggingface.co/docs/transformers/en/quantization/mxfp4) - [DeepSeek-V3 technical report (FP8 training)](https://arxiv.org/abs/2412.19437) - [llama.cpp quantize README (GGUF bpw)](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md) - [vLLM quantization docs](https://docs.vllm.ai/en/latest/features/quantization/) Date checked: 2026-06-09. Format specs are from vendor and OCP documentation; accuracy and memory figures are vendor/spec- or community-reported and labeled inline. --- ## Models # Inkling Source: https://singularitybyte.com/models/thinking-machines-inkling.html On July 15, 2026, Thinking Machines Lab shipped Inkling: 975 billion parameters, Apache 2.0, weights on Hugging Face on day one. That makes it the largest open-weights model ever released by a US lab, and the first foundation model from the company Mira Murati founded after leaving OpenAI. It reasons natively over text, images, and audio, carries a 1-million-token context window, and exposes a thinking effort dial you can turn from 0.2 to 0.99. And in a refreshing break from launch-day tradition, the lab's own announcement admits it is not the strongest model available. Here is what Inkling actually is, what it scores, and how to run or fine-tune it today. ## TL;DR - What it is: a 975B-parameter Mixture-of-Experts model with 41B active per token, multimodal input (text, image, audio), 1M-token context, trained on 45T tokens, released under a clean Apache 2.0 license with weights on Hugging Face at launch. - Why it matters: it is the US answer to the Chinese open-weight wave, pitched honestly as the best open base for customization rather than a leaderboard king, with a token-saving effort dial and fine-tuning via the Tinker API. - The catch: you need roughly 600 GB of VRAM even at 4-bit, llama.cpp support is still sitting in an unmerged PR, and on raw benchmarks it trails the closed frontier by a wide margin. ## What Thinking Machines actually shipped Thinking Machines Lab launched in early 2025 with a large chunk of OpenAI's former leadership and a product, Tinker, that sells managed fine-tuning rather than chat subscriptions. Inkling is the model that strategy was waiting for: a big, permissively licensed base that customers can specialize on Tinker and then run anywhere. Two models were announced. Inkling itself, 975B total and 41B active, is downloadable now as a BF16 checkpoint plus an NVFP4 build for Blackwell systems. Inkling-Small, a 276B model with 12B active, was previewed for a later release and will be the one most of us can realistically host. The positioning is the unusual part. The announcement states plainly that "Inkling is not the strongest overall model available today, open or closed" and sells it instead as a broad, honest base for customization. After a year of launch posts claiming frontier supremacy from every direction, a vendor leading with its own limitations is worth noticing. ## A DeepSeek-style recipe with real twists A Mixture-of-Experts (MoE) model splits its feed-forward layers into many expert subnetworks and routes each token through only a few, so you hold 975B parameters on disk but pay inference for 41B. Inkling follows the recipe popularized by [DeepSeek-V4](models/deepseek-v4.html) and [GLM-5.2](models/glm-5-2.html): 66 layers, 256 routed experts plus 2 shared, 6 routed experts active per token, sigmoid routing with auxiliary-loss-free load balancing. Then it deviates. There is no RoPE: Inkling uses a learned relative-position bias, which the lab claims extrapolates better to long sequences. Attention interleaves 512-token sliding-window layers with global layers at a 5:1 ratio. Short convolutions sit after the key and value projections, an inductive-bias trick you rarely see at this scale. And the multimodal path is encoder-free: images enter as raw 40x40 pixel patches through a four-layer MLP and audio as spectrogram tokens, no bolted-on vision tower. Sebastian Raschka's [architecture notes](https://sebastianraschka.com/blog/2026/inkling-architecture-benchmark-notes.html) are the best independent walkthrough of these choices, including a caution: with conventional grouped-query attention instead of DeepSeek-style MLA, raw decoding speed is probably not Inkling's advantage. ## Benchmarks, with the usual asterisk All numbers below are first-party, reported at effort 0.99, and not reproduced on our bench. The comparison columns come from Thinking Machines' own model card. What stands out is not any single score but the shape: solid everywhere, dominant nowhere, and honest about the gap to the closed frontier. Benchmark (reported) Inkling Claude Fable 5 GPT-5.6 Sol SWE-Bench Verified 77.6% 95.0% 82.2% Terminal Bench 2.1 63.8% not listed not listed AIME 2026 97.1% 99.9% 99.9% GPQA Diamond 87.2% not listed not listed HLE (text only) 29.7% 53.3% 47.2% SimpleQA Verified 43.9% 68.3% 71.6% MMMU Pro (vision) 73.5% not listed not listed VoiceBench (audio) 91.4% not listed not listed All figures are self-reported by Thinking Machines at launch. Treat them as vendor claims until independently reproduced. Within the open-weight field the picture is friendlier. 77.6% on SWE-Bench Verified clears Nvidia's [Nemotron 3 Ultra](models/nvidia-nemotron-3-ultra.html) at 71.9%, and the 91.4% VoiceBench score is unusual for a model whose audio path has no dedicated encoder. The clear weak spot is factual recall: 43.9% on SimpleQA means you want retrieval in front of it for anything knowledge-heavy. Against [Kimi K3](models/kimi-k3.html), announced the very next day at 2.8T parameters, Inkling loses the size contest but wins on availability: its weights are downloadable today, while Moonshot's are promised for July 27. ## The effort dial is the actual feature Most reasoning models give you a binary: thinking on or off. Inkling exposes thinking effort as a continuous parameter from 0.2 to 0.99, and the lab's own charts show it matching Nemotron 3 Ultra on Terminal Bench 2.1 while spending roughly a third of the tokens at lower effort settings. For agent pipelines, that is a budget knob you control per request instead of a model-picker decision. Turn it down for routing and classification calls, up for the hard synthesis step, all against one deployed checkpoint. If the numbers hold up in independent testing, this is the feature other open labs copy next. ## Run it in about 10 minutes Be honest with yourself about hardware first. The BF16 checkpoint wants about 2 TB of aggregated VRAM (think 16x H200), and even the NVFP4 build needs 600 GB or more across at least four Blackwell-class GPUs. This is not a laptop model. It is barely a single-server model. If you have the metal, vLLM works out of the box: ``` # Download the weights (BF16 ~2TB; NVFP4 for Blackwell systems) huggingface-cli download thinkingmachines/Inkling # Serve with vLLM on a multi-GPU node vllm serve thinkingmachines/Inkling \ --tensor-parallel-size 8 \ --max-model-len 262144 # Community GGUF quants exist (unsloth/inkling-GGUF), but llama.cpp # support is still in PR #25731. Until it merges, build from that PR # and always pass --jinja so the chat template is applied. ``` Everyone else calls it hosted. Together, Fireworks, Modal, Databricks, and Baseten all serve Inkling, and any OpenAI-compatible client works: ``` # Inkling via Together's OpenAI-compatible endpoint. from openai import OpenAI client = OpenAI( api_key="...", # your Together key base_url="https://api.together.xyz/v1", ) resp = client.chat.completions.create( model="thinkingmachines/Inkling", messages=[{"role": "user", "content": "Summarize this repo's auth flow and list its weak points."}], ) print(resp.choices[0].message.content) ``` Fine-tuning is the intended path, though: Inkling is live on Thinking Machines' Tinker platform with 64K and 256K context options at a 50% launch discount, plus a free playground for a limited time. Simon Willison's [first look](https://simonwillison.net/2026/Jul/16/inkling/) runs the multimodal loop through the Tinker API: he had Inkling draw an SVG pelican on a bicycle, then describe its own rendering. It analyzed the image coherently, then called its pelican a stork. Frontier is a spectrum. ## Limitations and gotchas - Hardware floor: 600 GB VRAM minimum at 4-bit, about 2 TB at BF16. Self-hosting means a GPU server, not a workstation. Wait for Inkling-Small if you want local. - llama.cpp support is not mainline yet. The GGUF quants from Unsloth and others require building llama.cpp from PR #25731 and passing --jinja for the chat template. - Benchmarks are launch-day and first-party. The honest framing earns goodwill, but the numbers still deserve independent reproduction. - SimpleQA at 43.9% is weak factual recall for a model this size. Pair it with retrieval for knowledge work. - Output is text only. It reads images and audio but will not generate them. - Training data disclosure is thin. The card acknowledges "content that may be subject to intellectual property protection" without detail, a point Willison flags too. ## Who should use it Use Inkling if you fine-tune. That is what it is for: a permissive Apache 2.0 base with multimodal input, a 1M context, and a managed tuning platform attached, from a US lab if your compliance people care about that. It also makes sense as a hosted workhorse for agent pipelines where the effort dial can cut your reasoning-token bill against a single deployment. Hold off if you want the strongest general model (the lab itself points you elsewhere), if you need something you can run on a workstation today (watch for Inkling-Small, or grab [GLM-5.2](models/glm-5-2.html)), or if your workload is pure coding, where [Kimi K2.7 Code](models/kimi-k2-7-code.html) gives you more per active parameter. For everyone else, the real story is strategic: the US open-weights scene finally has a frontier-scale entrant, after a year in which [Qwen](models/alibaba-qwen-3-5.html), DeepSeek, and Moonshot made the open frontier look like a one-country race. ## Sources and further reading - [Thinking Machines: Introducing Inkling (official announcement)](https://thinkingmachines.ai/news/introducing-inkling/) - [Inkling model card (hardware requirements, full benchmark table)](https://thinkingmachines.ai/model-card/inkling/) - [Hugging Face: thinkingmachines/Inkling (BF16 + NVFP4 weights)](https://huggingface.co/thinkingmachines/Inkling) - [Hugging Face blog: Welcome Inkling](https://huggingface.co/blog/thinkingmachines-inkling) - [Simon Willison: first look at Inkling](https://simonwillison.net/2026/Jul/16/inkling/) - [Sebastian Raschka: Inkling architecture and benchmark notes](https://sebastianraschka.com/blog/2026/inkling-architecture-benchmark-notes.html) - [Unsloth: Inkling GGUF quantizations](https://huggingface.co/unsloth/inkling-GGUF) Tested on: not independently benchmarked. Inkling needs 600+ GB of VRAM even quantized, which is beyond our local bench. All benchmark figures are Thinking Machines' launch-day numbers; hardware and access details are drawn from the model card and the sources above. Date checked: 2026-07-22 --- # Kimi K3 Source: https://singularitybyte.com/models/kimi-k3.html On July 16, 2026, Moonshot AI shipped Kimi K3, and the ceiling for open-weight models moved again. It is a 2.8-trillion-parameter Mixture-of-Experts model with a 1-million-token context window, and Moonshot says the full weights land on Hugging Face by July 27. In blind developer testing it took first place on the Frontend Code Arena leaderboard, ahead of Claude Fable 5. It costs $3 in and $15 out per million tokens. In the same week, Anthropic is moving Fable 5 to $10 in and $50 out and pulling it out of subscription plans. If you build with models, that gap is the story. Here is what K3 is, what it actually scores, and how to point your existing agent at it today. ## TL;DR - What it is: a 2.8T-parameter Mixture-of-Experts model (activates 16 of 896 experts per token), 1M-token context, native vision, always-on reasoning. Weights promised under an open license by July 27, 2026. - Why it matters: it beats or trails the top closed US models by a hair on reported benchmarks, at roughly a third of Fable 5's price, with the weights promised for anyone to run. - The catch: the headline scores are first-party, the weights were not public at launch, and self-hosting 2.8T parameters is a data-center job. ## What Moonshot actually shipped One definition first, because it drives the cost math. A Mixture-of-Experts (MoE) model splits its feed-forward layers into many small "expert" subnetworks and routes each token to only a few of them. K3 holds 2.8 trillion parameters on disk but fires only 16 of its 896 experts per token, so you get the capacity of a very large model at the inference cost of a much smaller one. That is the same bargain behind other big open MoEs like [DeepSeek-V4](models/deepseek-v4.html) and [Qwen3.5](models/alibaba-qwen-3-5.html). It takes text and images, carries a 1-million-token context window aimed at long-horizon coding and agent runs, and ships with an always-on reasoning mode Moonshot calls thinking mode. Pricing through the Kimi API is $3.00 per million input tokens and $15.00 per million output tokens, dropping to $0.30 per million on cached input. This is the same team behind [Kimi K2.7 Code](models/kimi-k2-7-code.html) and [Kimi K2.5](models/moonshot-kimi-k2-5.html), now aiming a general flagship at the very top of the table. ## Benchmarks, with the usual asterisk Every number below is reported by Moonshot or by third-party arenas at launch, not reproduced on our bench. Read them as a starting point, not a measurement. The pattern is consistent though: K3 sits in the same band as the best closed US models, and leads several of them on agentic coding. Benchmark (reported) Kimi K3 Claude Fable 5 Max GPT-5.6 Sol Max Claude Opus 4.8 GDPval-AA v2 (real-world work, 44 occupations) 1,687 1,815 1,747.8 1,600 AA-Briefcase (long-horizon agentic) 1,527 1,587 1,495 not reported On GDPval-AA v2, a test of real tasks across 44 occupations, K3 placed third overall, behind Fable 5 Max and GPT-5.6 Sol Max but ahead of Claude Opus 4.8. On AA-Briefcase, a private long-horizon agentic benchmark from Artificial Analysis, it climbed to second, past GPT-5.6 Sol Max and behind only Fable 5 Max. The one that got the most attention: Arena ranked K3 first on Frontend Code at 1,679 points in blind developer voting, ahead of Fable 5. Its overall Coding Index came in at 76.24. Independent testers report K3 leading on Terminal-Bench 2.1 and long-horizon SWE tasks, while Fable 5 keeps the edge on several other coding suites. Call it a tie at the frontier, which is the point: an open-weight model is now trading blows with the most expensive closed ones. ## The part the leaderboards skip: freedom and price Here is why builders are paying attention beyond the score. K3 has no classifier sitting between your call and the model, and no quiet routing to a weaker fallback. Developers comparing the two report that the model you call is the model you get. That is not marketing spin from Moonshot, it is a design difference you can feel on long agent runs. Contrast that with how a refusal works on the closed side. Anthropic's own API documentation describes a stop_reason of refusal, and states plainly that on Claude Fable 5, safety classifiers return this stop reason as a normal HTTP 200 response, not an error. In practice that means a request can look like it succeeded while the work simply did not happen, and your error monitoring will not flag it unless you check the stop reason. For agent pipelines that run unattended, a silent refusal is worse than a loud one. Then there is the bill. In the same window K3 launched, Anthropic is pulling included Fable 5 access for Pro, Max, and Team subscribers on July 19, and switching it to metered usage credits on July 20 at $10 per million input tokens and $50 per million output tokens. That is double the rate of Claude Opus 4.8 and the most expensive pricing Anthropic has ever listed for a generally available model. Max subscribers are not exempt. So the same week one lab put its frontier model behind a higher paywall, another put a comparable one in the open at a third of the price. Model Input / Mtok Output / Mtok Weights Kimi K3 $3.00 ($0.30 cached) $15.00 Open, promised July 27 Claude Fable 5 $10.00 $50.00 Closed The trend under all of this is not subtle. The frontier you can actually own, run, fine-tune, and audit is increasingly being set in the open, and right now a lot of it is being set by Chinese labs: Moonshot, DeepSeek, Qwen, Z.ai. You do not have to cheer for any flag to notice that open weights change your options. When the model is yours, nobody reprices it out from under you or reroutes it mid-run. ## Run it in about 10 minutes You do not have to wait for the weights to try K3. Point any OpenAI-compatible client at the Kimi API and you are calling it in one request. ``` # Kimi K3 via the Moonshot API (OpenAI-compatible endpoint). # Get a key at platform.moonshot.ai, then: export MOONSHOT_API_KEY="sk-..." curl https://api.moonshot.ai/v1/chat/completions \ -H "Authorization: Bearer $MOONSHOT_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "kimi-k3", "messages": [ {"role": "user", "content": "Build a single-file browser desktop: draggable windows, a taskbar, and a working clock. Vanilla HTML, CSS, and JS only."} ] }' ``` Because the endpoint is OpenAI-compatible, you can also drop it straight into your existing coding agent. Set the base URL and model, keep everything else. ``` # Reuse any OpenAI-style SDK; just swap the base_url and model. from openai import OpenAI client = OpenAI( api_key="sk-...", # your Moonshot key base_url="https://api.moonshot.ai/v1", ) resp = client.chat.completions.create( model="kimi-k3", messages=[{"role": "user", "content": "Refactor this repo's auth module and add tests."}], ) print(resp.choices[0].message.content) ``` A browser desktop clone, a full HTML mockup of an old operating system UI, a from-scratch web app: these long, single-shot build tasks are exactly where a 1M-token context and cheap output tokens pay off. Give K3 the whole spec in one prompt and let it run. Once the weights land on July 27, the same jobs move to your own hardware if you have the memory for it. ## Limitations and gotchas - Weights were not public at launch. Moonshot promised them "by July 27" under an open license, following its Modified MIT pattern on earlier Kimi releases, but at launch there was no confirmed checkpoint on Hugging Face. Verify the real repo and LICENSE when it appears. - Benchmarks are first-party or arena-reported, not independently reproduced. Treat the table above as a vendor and community claim. - It is enormous. At 2.8T parameters, self-hosting is a multi-node, data-center job even after quantization. This is not a laptop model, and it will not be for a long time. - Until the weights drop, you are calling a China-hosted API. For sensitive code or data, read the Kimi terms and route accordingly. - No safety classifier cuts both ways. Great for uninterrupted agent runs, but you own more of the guardrail decisions yourself. ## Who should use it Use K3 if you want frontier-class coding and agent performance without frontier pricing, you value calling a model that does not silently refuse or downgrade, and you want the option to run the weights yourself when they land. It is a strong fit for long agent jobs, large-repo work, and anyone whose bill just doubled when Fable 5 left their plan. Hold off if you need a checkpoint you can self-host today (wait for July 27 and confirm it), if you require independently reproduced benchmarks before you trust a number, or if your data cannot leave your jurisdiction and the hosted API is your only option for now. For a smaller open coding model you can actually run this week, [Kimi K2.7 Code](models/kimi-k2-7-code.html) or [GLM-5.2](models/glm-5-2.html) are the saner starting points. ## Sources and further reading - [Moonshot: official Kimi K3 announcement](https://www.kimi.com/blog/kimi-k3) - [VentureBeat: Moonshot releases Kimi K3](https://venturebeat.com/technology/chinas-moonshot-ai-releases-kimi-k3-the-largest-open-source-model-ever-rivaling-top-u-s-systems) - [Tom's Hardware: 2.8T Kimi K3 tops Frontend Code Arena](https://www.tomshardware.com/tech-industry/artificial-intelligence/moonshot-releases-2-8-trillion-parameter-kimi-k3) - [OpenRouter: Kimi K3 API and benchmarks](https://openrouter.ai/moonshotai/kimi-k3) - [Anthropic docs: the refusal stop reason (HTTP 200 on Fable 5)](https://platform.claude.com/docs/en/build-with-claude/stop-reasons-and-fallback) - [BleepingComputer: Fable 5 subscription access and the July 19 cutoff](https://www.bleepingcomputer.com/news/artificial-intelligence/claude-fable-5-stays-free-for-paid-users-until-july-19-as-anthropic-buys-more-time/) - [Android Authority: Fable 5 metered credits at $10 / $50](https://www.androidauthority.com/anthropic-claude-fable-5-credits-usage-july-3684840/) Tested on: not independently benchmarked. Kimi K3 is a 2.8T-parameter MoE and was hosted-API only at the time of writing (weights promised for 2026-07-27), which is beyond our local bench. Every benchmark here is Moonshot-reported or arena-reported; pricing and access details are drawn from the sources linked above. Date checked: 2026-07-18 --- # LingBot-Map Source: https://singularitybyte.com/models/lingbot-map.html Point a camera at the world, and LingBot-Map builds the 3D scene while you are still filming. Released in late April 2026 by Robbyant, the robotics arm of Ant Group, this Apache 2.0 model turns an ordinary RGB video stream into camera poses and a dense point cloud in real time: about 20 frames per second, sequences beyond 10,000 frames, and roughly 13.3 GB of VRAM. No LiDAR, no depth sensor, no hours of COLMAP crunching. If you have ever wanted photogrammetry that keeps up with a moving robot, a drone, or just your phone on a walk, this is the most usable open release yet. ## Streaming 3D reconstruction from video, explained Classic 3D reconstruction is an offline job. You record everything first, then a pipeline like COLMAP or a batch model like VGGT chews on the full set of frames at once. That works for scanning a statue, but it is useless for a robot that needs to know where the wall is right now. Streaming reconstruction flips the constraint: the model sees frames one at a time, in order, and must output geometry and camera pose immediately without ever revisiting old frames. The classic answer is SLAM (simultaneous localization and mapping), which tracks handcrafted feature points and runs constant optimization loops. LingBot-Map replaces that whole machinery with a single feed-forward transformer: frames go in, poses and points come out, no iterative optimization anywhere. The catch that killed earlier attempts is drift. Streaming models like Spann3R or CUT3R accumulate small pose errors that compound over minutes of video, until the reconstruction folds in on itself. LingBot-Map's paper is essentially one long answer to the drift problem. ## How the Geometric Context Transformer works The architecture is called a Geometric Context Transformer (GCT), and it is a pure autoregressive model, the same generation-one-step-at-a-time regime your LLM uses, applied to geometry. Each incoming frame attends to three kinds of context instead of one big history buffer. First, an anchor context pins the global coordinate frame, so the scene does not slowly rotate out from under itself. Second, a pose-reference window holds the most recent frames for dense geometric cues, the local detail. Third, a trajectory memory keeps a compact record of where the camera has been, which is what corrects long-range drift when you loop back to a place you have already seen. Because the state stays compact and attention runs over a paged KV cache (the same key-value caching trick that makes LLM serving fast), per-frame cost stays flat instead of growing with video length. The paper reports the bounded context cuts per-frame compute by roughly 80x versus keeping full history, which is how a 25,000-frame indoor walkthrough demo stays stable. Training covered 29 datasets across synthetic and real scenes. ## Benchmarks: paper-reported, but the gap is not subtle All numbers below come from the [LingBot-Map paper](https://arxiv.org/abs/2604.14141), so treat them as vendor-reported until the community reproduces them. The team did release the full evaluation pipeline for KITTI, Oxford Spires, VBR, TUM-D, 7-Scenes, ETH3D, Tanks and Temples, and NRGBD, which makes checking their homework unusually easy. On Oxford Spires with 320-frame sequences, LingBot-Map reports an absolute trajectory error (ATE, how far the estimated camera path deviates from ground truth) of 6.42 m at 20.29 FPS. The streaming competition is not close: CUT3R lands at 18.16 m and Wint3R at 21.10 m. It also beats the offline batch models VGGT (24.78 m) and DA3 (12.87 m), which get to see every frame at once. Method Type Oxford Spires ATE (m) ETH3D F1 FPS LingBot-Map Streaming 6.42 98.98 20.3 Wint3R Streaming 21.10 77.28 3.9 CUT3R Streaming 18.16 n/p 29.2 DA3 Offline 12.87 n/p n/a VGGT Offline 24.78 n/p n/a All figures paper-reported (arXiv:2604.14141), not independently reproduced. "n/p" means not published in the paper's headline tables. Offline methods process the whole sequence at once, so FPS is not comparable. The long-sequence test is the one that matters for real use. Stretched to 3,840 frames on Oxford Spires, LingBot-Map's error grows by just 0.69 m to 7.11 m, while CUT3R and Wint3R blow out past 32 m. Reconstruction quality (F1, the balance of accuracy and completeness of the recovered surface) tells the same story: 98.98 on ETH3D versus 77.28 for the runner-up, plus wins on 7-Scenes (80.39) and NRGBD (64.26). ## Run it in about 10 minutes You need an NVIDIA GPU with around 14 GB of free VRAM and CUDA 12.8 drivers; a used RTX 3090 or any 4090-class card clears the bar. There is no Ollama route here, this is a Python research repo, but an unusually tidy one. If you are still deciding what box to build, our [local AI hardware guide](news/local-ai-computing-exploring-nvidia-dgx-spark-apple-m4-max-mac-studio-amd-ryzen-ai-max-395.html) covers the trade-offs. ``` # 1. Environment (Python 3.10 + PyTorch 2.8 with CUDA 12.8) conda create -n lingbot-map python=3.10 -y conda activate lingbot-map pip install torch==2.8.0 torchvision==0.23.0 \ --index-url https://download.pytorch.org/whl/cu128 # 2. Clone and install, with the browser viewer extras git clone https://github.com/Robbyant/lingbot-map.git cd lingbot-map pip install -e ".[vis]" pip install flashinfer-python # recommended, big speedup # 3. Grab a checkpoint (HF: robbyant/lingbot-map) # lingbot-map-long is the one you want for real videos. # 4. Reconstruct the bundled example scene python demo.py --model_path /path/to/lingbot-map-long.pt \ --image_folder example/courthouse --mask_sky ``` The demo spins up a viser viewer at http://localhost:8080, and you can watch the point cloud grow frame by frame in your browser. Four example scenes ship in the repo (courthouse, university, a loop-closure trajectory, and an outdoor Oxford sequence). For your own footage, pass --video_path video.mp4 instead of an image folder, and add --mask_sky outdoors so the sky does not smear into the geometry (it pulls a small ONNX segmentation model on first run, so pip install onnxruntime first). Two flags do most of the tuning work. --keyframe_interval 2 halves KV-cache memory on long videos, and --mode windowed --window_size 128 is mandatory past roughly 3,000 frames, resetting state in overlapping chunks so drift never accumulates. Tight on VRAM? --offload_to_cpu trades speed for headroom. ## Limitations and gotchas First, the hard requirement: CUDA 12.8 on NVIDIA silicon. There is no Apple Silicon or ROCm path today, though a community [Mac desktop viewer](https://github.com/donalleniii/lingbot-desktop-mac) lets a Mac display a reconstruction streamed from a Linux GPU box. Second, depth range is bounded by the training data distribution. Feed it aerial footage from 200 m up and the geometry gets vague, because the model never saw scenes at that scale. Third, quality degrades past about 320 frames of raw KV cache, so the keyframe and windowed modes are not optional extras for long captures, they are the operating manual. And the output is a point cloud with camera poses, not a textured mesh: for meshes you still hand the result to downstream tooling, or reach for a generative model like [Hunyuan3D-2](models/hunyuan3d-2.html) when the goal is an asset rather than a map. ## Who should use it Robotics and embodied AI builders are the obvious audience, since real-time ego-motion plus geometry from one RGB camera is the input layer for navigation, and it slots naturally under world models like [NVIDIA Cosmos 3](models/nvidia-cosmos-3.html). Drone and scanning hobbyists get COLMAP-grade capture that finishes when the flight does. And 3D content pipelines can use the poses and points as fast initialization for NeRF or Gaussian splatting runs, the same way [LHM-1B](models/lhm-1b.html) shortcuts human reconstruction. The 10-minute move: clone the repo, download lingbot-map-long, and run the courthouse example. Watching a building assemble itself in your browser at 20 FPS explains this release better than any benchmark table. ## Sources and further reading - [LingBot-Map on GitHub](https://github.com/Robbyant/lingbot-map) - [Model weights on Hugging Face](https://huggingface.co/robbyant/lingbot-map) - [Model weights on ModelScope](https://modelscope.cn/models/Robbyant/lingbot-map) - [Paper: Geometric Context Transformer for Streaming 3D Reconstruction (arXiv)](https://arxiv.org/abs/2604.14141) - [Community Mac desktop viewer](https://github.com/donalleniii/lingbot-desktop-mac) Tested on: not independently tested. LingBot-Map requires CUDA 12.8 on NVIDIA hardware, which we could not run for this article; all benchmark figures are paper-reported by the Robbyant team, and install steps are taken verbatim from the official repository. Date checked: 2026-07-16 --- # Mistral Medium 3.5 Source: https://singularitybyte.com/models/mistral-medium-3-5.html Mistral's middle child finally comes with weights. Mistral Medium 3.5, released April 29, 2026, is a 128-billion-parameter dense model you can download from Hugging Face today: no gate, no access form, no research-only clause. It posts 77.6% on SWE-bench Verified, takes text and images, carries a 256K context window, and ships under a modified MIT license that is free for everyone below a $20 million monthly revenue bar. If you skipped every "Mistral Medium" headline because the last one was API-only, this is the release worth reading about. ## What changed: Mistral's mid model is now open weights The original Mistral Medium 3 (May 2025) was a closed, API-only model. No parameter count, no weights, and it is now deprecated, with retirement scheduled for August 31, 2026. Medium 3.5 replaces it and flips the distribution model: the full weights sit in the official [mistralai Hugging Face repo](https://huggingface.co/mistralai/Mistral-Medium-3.5-128B), in both FP8 (about 134 GB across three safetensors shards) and BF16 (about 250 GB). We opened the file tree and checked: tokenizer, config, LICENSE, all there, nothing gated. That matters because Medium 3.5 is not a consolation-prize model. It is positioned for agentic and coding work, with a configurable per-request reasoning effort, and its official numbers land within a few points of the best open coding models on the planet. ## The Mistral naming decoder Mistral's lineup names are a trap, and we fell into it ourselves while researching this piece. Here is the map so you don't have to reconstruct it: Model Size License Weights [Mistral Small 3.1](models/mistral-small-3-1.html) 24B dense Apache 2.0 Open Mistral Medium 3 / 3.1 undisclosed Proprietary API only, deprecated Mistral Medium 3.5 128B dense Modified MIT Open [Mistral Large 3](models/mistral-large-3.html) 675B MoE (41B active) Apache 2.0 Open Yes, that means "Medium 3" and "Medium 3.5" sit on opposite sides of the open-weights line. If you see a blog calling Medium 3 an open model, close the tab. ## What it actually is Medium 3.5 (version code 26.04) is a dense transformer: all 128B parameters are active on every token, unlike the Mixture-of-Experts design in Large 3. It is multimodal, taking text and images as input, with a vision encoder Mistral trained from scratch to handle variable image sizes and aspect ratios. The context window is 256K tokens, and you can dial reasoning effort up or down per request instead of choosing between separate instruct and reasoning variants. Community reporting says Medium 3.5 merges what used to be separate specialist models (the Devstral coding line and Magistral reasoning line) into one set of weights. Mistral's own card does not state that, so treat it as plausible background, not fact. ## The license: MIT with a $20 million asterisk The LICENSE file in the repo is standard MIT plus one carve-out, quoted directly: "You are not authorized to exercise any rights under this license if the global consolidated monthly revenue of your company (or that of your employer) exceeds $20 million." Above that bar, you email Mistral for a commercial license. Practical reading: $20 million per month is roughly $240 million a year in revenue. Every indie hacker, startup, research lab, and mid-size company on earth is below it. You can use it commercially, fine-tune it, and redistribute it. It is still worth one critical note: within Mistral's own family, the smallest model (Small, Apache 2.0) and the biggest (Large 3, Apache 2.0) are more permissively licensed than this middle one. Mistral put the revenue cap exactly where the enterprise money is. ## Benchmarks: official numbers first Mistral's model card states two headline results in plain text: 77.6% on SWE-bench Verified and 91.4% on the tau3-Telecom agentic benchmark. The rest of the card's comparisons ship as chart images, so exact MMLU and GPQA digits are not published as numbers anywhere we could verify. How does 77.6% stack up? Here is the open-weights coding field as reported by community trackers and roundups (these cross-model numbers are community-reported, not Mistral's): Model SWE-bench Verified License Size DeepSeek V4 80.6% open large MoE Kimi K2.6 80.2% open large MoE Mistral Medium 3.5 77.6% (official) Modified MIT 128B dense Qwen3.x 27B 77.2% Apache 2.0 27B The independent Artificial Analysis Intelligence Index scores Medium 3.5 at 30, second of 62 models tracked at the time of writing. Third-party API benchmarks measured about 102 tokens per second on La Plateforme, where pricing is $1.50 per million input tokens and $7.50 per million output. The honest summary: the giant MoE models still hold the coding crown by about three points, but Medium 3.5 gets you within reach of them from a single-node, dense, self-hostable package. ## Run it locally, if your hardware can Let's be direct: 128B dense is not a laptop model. Dense means every token streams all 128B weights, so your tokens per second are roughly memory bandwidth divided by model size. That is why MoE models fly on the same box while this one crawls. Mistral says it self-hosts on as few as 4 GPUs, which in practice means 80GB-class cards for the FP8 weights. The one measured community datapoint we found: two RTX PRO 6000 Blackwell cards (192GB total) running FP8 under vLLM deliver 26 to 35 tokens per second on prose and 37 to 43 on code. Below that tier, the community GGUF quants from [unsloth](https://huggingface.co/unsloth/Mistral-Medium-3.5-128B-GGUF) are the realistic path: Quant File size Runs on UD-IQ2_XXS 34.9 GB 48GB Macs, 2x 24GB GPUs (quality hit) Q2_K 46.6 GB 64GB unified memory Q4_K_M 74.9 GB 128GB Mac Studio; Strix Halo 128GB and 4x 24GB GPUs fit it but run slow Q5_K_M 88.3 GB 128GB unified memory Q8_0 133 GB 192GB+ unified memory, multi-GPU servers There is an official Ollama library entry, so the setup is one line if your machine qualifies: ``` # Ollama (needs ~80GB free RAM/VRAM for the default quant) ollama run mistral-medium-3.5:128b # Self-host FP8 with vLLM (official recipe; needs a vLLM NIGHTLY build as of July 2026) vllm serve mistralai/Mistral-Medium-3.5-128B --tensor-parallel-size 8 \ --tokenizer_mode mistral --config_format mistral --load_format mistral \ --enable-auto-tool-choice --tool-call-parser mistral --reasoning-parser mistral # No big iron? The hosted API. Set MISTRAL_API_KEY first. curl https://api.mistral.ai/v1/chat/completions \ -H "Authorization: Bearer $MISTRAL_API_KEY" \ -H "Content-Type: application/json" \ -d '{"model":"mistral-medium-3-5","messages":[{"role":"user","content":"Refactor this function and explain why."}]}' ``` ## Limitations and gotchas - Dense means no free lunch. A 128B MoE with 40B active would run far faster on the same hardware. Every token here touches all 128B weights. - The license is not Apache 2.0. The $20M/month carve-out is irrelevant for most of us but makes lawyers at big companies read the file twice. - Mistral's chart-image benchmarks mean key numbers (MMLU, GPQA) are unverifiable in text form. We flagged what is official versus community-reported above. - It trails DeepSeek V4 and Kimi K2.6 on SWE-bench by about three points, per community trackers. If leaderboard-max coding is the only goal, those remain ahead (and much heavier). - Early GGUFs shipped with a broken YaRN long-context config (repetition and forgetting on long chats). It is fixed upstream, so if you downloaded a quant in the first days after launch, re-download it. - Vision does not work through Ollama (the GGUF needs a separate mmproj file); use llama.cpp for image input, with a BF16 or F32 mmproj, not F16. - vLLM support is nightly-only at the time of writing, and fine-tuning a 128B dense model is out of QLoRA range for typical rigs. If you want to fine-tune a Mistral, use [Small 3.1](models/mistral-small-3-1.html). ## Who should use it Pick Medium 3.5 if you want near-frontier agentic coding from weights you control, and you have either a multi-GPU node, a 128GB unified-memory machine, or a tolerance for API pricing. It is the strongest open Mistral for coding and agent work, sitting between the local-friendly [Small 3.1](models/mistral-small-3-1.html) and the chat-focused [Large 3](models/mistral-large-3.html). If your box tops out at 24GB of VRAM, run Small 3.1 locally and call Medium 3.5 over the API when the task deserves it. ## Sources and further reading - [Mistral: Medium 3.5 announcement](https://mistral.ai/news/vibe-remote-agents-mistral-medium-3-5/) - [Official weights repo (Hugging Face)](https://huggingface.co/mistralai/Mistral-Medium-3.5-128B) - [Mistral docs: Medium 3.5 model card](https://docs.mistral.ai/models/model-cards/mistral-medium-3-5-26-04) - [unsloth GGUF quants](https://huggingface.co/unsloth/Mistral-Medium-3.5-128B-GGUF) - [Ollama library entry](https://ollama.com/library/mistral-medium-3.5:128b) - [Artificial Analysis: independent benchmarks](https://artificialanalysis.ai/models/mistral-medium-3-5) Tested on: not independently benchmarked. A 128B dense model exceeds our local bench, so all numbers above come from Mistral's model card (marked official) or community trackers (marked community-reported), with sources linked. Date checked: 2026-07-10 --- # LongCat-2.0 Source: https://singularitybyte.com/models/longcat-2-0.html Meituan is the company that delivers your dumplings in Beijing. On June 29, 2026, its LongCat team open-sourced LongCat-2.0, a 1.6-trillion-parameter agentic model under the MIT license, with a native 1M-token context window. Two more headlines ride along: the model had already spent two months anonymously topping OpenRouter's agent leaderboards as "Owl Alpha," and Meituan says the whole thing was trained and served on domestic Chinese AI chips, no NVIDIA hardware involved. Here is what shipped, the numbers with their asterisks, and what you can actually do with it today. ## What Meituan shipped LongCat-2.0 is a sparse Mixture-of-Experts (MoE) model: 1.6 trillion total parameters, with roughly 48 billion active per token. An MoE splits its feed-forward layers into many small "expert" subnetworks and routes each token through only a few of them, so you pay storage for all 1.6T but compute for about 48B, the same bargain behind [DeepSeek-V4](/models/deepseek-v4.html) and [GLM-5.2](/models/glm-5-2.html). LongCat goes one step further: zero-computation experts and a PID controller let the activation float between about 33B and 56B per token depending on how hard the token is, per the [official README](https://github.com/meituan-longcat/LongCat-2.0). The release is complete in the way we wish every open drop were: [BF16 weights](https://huggingface.co/meituan-longcat/LongCat-2.0), [FP8](https://huggingface.co/meituan-longcat/LongCat-2.0-FP8) and [INT8](https://huggingface.co/meituan-longcat/LongCat-2.0-INT8) checkpoints, SGLang deployment guides for both GPUs and NPUs, OpenAI- and Anthropic-compatible API endpoints, a thinking-mode toggle, and MIT for all of it. Pretraining ran on more than 35 trillion tokens, including hundreds of billions of tokens of 1M-context data, and Meituan claims the run finished with no rollbacks or loss spikes. Maximum output is 131,072 tokens. This is not Meituan's first model. LongCat-Flash, a 560B MoE from 2025 ([tech report](https://arxiv.org/pdf/2509.01322)), introduced the zero-computation-expert trick, and the Flash-Thinking, Flash-Omni, and Flash-Lite variants followed. LongCat-2.0 is the trillion-scale successor, and it arrives in the same [June wave](/news/open-weight-wave-june-2026.html) that gave us GLM-5.2 and [Kimi K2.7 Code](/models/kimi-k2-7-code.html). ## Two months undercover as Owl Alpha The best part of the launch story is that the market reviewed this model before anyone knew what it was. LongCat-2.0 ran anonymously on OpenRouter as "Owl Alpha" for about two months. During that stealth run it ranked first by monthly call volume on the Hermes Agent workspace, second on Claude Code, and third on OpenClaw, according to [Decrypt's report](https://decrypt.co/372579/longcat-2-0-meituan-ai-stealth-model-openrouter). Community trackers put its throughput above ten trillion tokens a month before the reveal, a figure Meituan itself has not confirmed, so treat it as community-reported. Why that matters: developers routed real agent workloads to an unbranded model because it performed, not because of a launch blog. That is about as clean a blind test as this industry gets. ## The benchmarks, and the asterisk Meituan's official table compares LongCat-2.0 only against closed frontier models. Values marked with an asterisk are cited from the competitor's own reports; the LongCat numbers are Meituan's and have not been independently reproduced. Notably, there is no official comparison against GLM-5.2 or DeepSeek-V4, so resist the urge to line those up from different tables. Benchmark | LongCat-2.0 | GPT-5.5 | Gemini 3.1 Pro | Claude Opus 4.8 SWE-bench Pro | 59.5 | 58.6* | 54.2* | 69.2* Terminal-Bench 2.1 | 70.8 | 73.8* | 70.7* | 78.9* SWE-bench Multilingual | 77.3 | n/p | 76.9* | 84.8* BrowseComp | 79.9 | 84.4* | 85.9* | 84.3* GPQA-diamond | 88.9 | 93.6* | 94.3* | 92.4 All LongCat figures are Meituan-reported. Asterisked competitor scores are cited from those vendors' own publications; "n/p" means no public score. Source: the [official benchmark table](https://github.com/meituan-longcat/LongCat-2.0). The honest read: near-frontier, not frontier-beating. On SWE-bench Pro, the agentic coding benchmark people actually watch, 59.5 edges GPT-5.5 (58.6) and clears Gemini 3.1 Pro (54.2), but the newer Claude Opus 4.7 (64.3) and 4.8 (69.2) stay ahead. For an MIT-licensed model you can download, modify, and redistribute, beating any current closed frontier model on any agentic benchmark is the story. Expecting it to beat all of them is not the right bar. ## The Chinese-chips claim Meituan calls LongCat-2.0 the industry's first trillion-parameter model to complete full-process training and inference on domestic "AI ASIC superpods," with reporting citing a cluster of about 50,000 accelerator cards and zero NVIDIA silicon ([SiliconANGLE](https://siliconangle.com/2026/06/30/chinas-meituan-open-sources-massive-longcat-2-0-ai-model-saying-trained-domestic-chips/), [SCMP](https://www.scmp.com/tech/tech-trends/article/3358854/china-debuts-biggest-ai-model-trained-local-chips-meituan-releases-longcat-20)). The part nobody can verify: which chips. Meituan never names the vendor. The community's guess is Huawei's Ascend line, inferred from interconnect and memory clues in the documentation, but that is speculation, not confirmation. What is checkable is the deployment code: the repo ships an NPU serving path alongside the GPU one, which at minimum proves non-NVIDIA inference is a first-class citizen, not a press-release afterthought. If the training claim holds up, the export-control debate we covered in [the open-weights export fight](/news/anthropic-export-ban-case-for-open-weight-ai-2026.html) just got a large new data point. ## Architecture for the curious Two pieces are genuinely new. First, LongCat Sparse Attention (LSA), Meituan's answer to the sparse-attention design DeepSeek introduced: an indexer picks which past tokens each query actually attends to, and LSA layers three refinements on top (streaming-aware indexing, one index shared across every two layers, and a coarse-to-fine hierarchical pass). The payoff is long-context cost that scales closer to linear, which is what makes a native 1M window usable rather than theoretical. Second, the on-disk weights are bigger than the headline: alongside the 1.6T MoE parameters sits a 135B N-gram Embedding block, inherited from LongCat-Flash-Lite, which is why the checkpoint folder describes itself as roughly 1.8T parameters. A 3-step multi-token prediction module handles speculative decoding, and post-training distills three separate teacher groups (agent, reasoning, interaction) into one student, a recipe Meituan calls MOPD. ## Limitations and gotchas - You cannot run this at home. Even the FP8 checkpoint is a multi-node deployment; community estimates point at a 16x H20-class server as the floor, and that figure is itself unverified. This is a rent-a-cluster or use-the-API model. - No Ollama build and no community GGUF exist yet, so there is no one-line local install to paste here. - Benchmarks are self-reported, and the OpenRouter throughput legend is community-reported. The blind Owl Alpha traffic is the strongest independent signal we have. - The chip vendor is undisclosed. "Trained on domestic silicon" is a Meituan claim with no third-party audit. - API pricing (around $0.75 per million input tokens and $2.95 per million output at standard rates, with a launch promo roughly 60 percent below that) comes from aggregator reporting, so confirm current rates on the platform before you build a budget around them. ## Who should use it Teams running agent harnesses (Claude Code-style CLIs, browser agents, long-horizon coding loops) who want an MIT-licensed brain they can eventually self-host, and researchers who want to study trillion-scale sparse attention with full weight access. If you need a model for one GPU, this is not it; look at [Qwen 3.6 27B](/models/qwen-3-6.html) instead. If you need the absolute best agentic scores regardless of license, Claude Opus 4.8 still holds that crown. ## Try it in about 10 minutes The fastest path is the free chat at [longcat.ai](https://longcat.ai/). For API access, the platform exposes both OpenAI- and Anthropic-compatible endpoints, so existing harnesses connect with a base-URL swap; the [README](https://github.com/meituan-longcat/LongCat-2.0) documents the exact setup for Claude Code and friends. If you have serious hardware, the weights are one command away: ``` # Pull the FP8 checkpoint (bring a multi-node GPU server and patience) hf download meituan-longcat/LongCat-2.0-FP8 # Deployment recipes for SGLang (GPU and NPU) live in the repo git clone https://github.com/meituan-longcat/LongCat-2.0 ``` Then point whatever agent harness you already run at the endpoint and give it a long multi-step task. A model tuned for agents deserves a real agent workload, not a haiku request. ## Sources and further reading - [LongCat-2.0 model card (Hugging Face)](https://huggingface.co/meituan-longcat/LongCat-2.0) - [Official GitHub repo, benchmark table and deployment guides](https://github.com/meituan-longcat/LongCat-2.0) - [VentureBeat coverage](https://venturebeat.com/technology/meituan-open-sources-longcat-2-0-the-1-6t-near-frontier-agentic-coding-model-thats-been-leading-openrouter-trained-entirely-on-chinese-chips) - [Decrypt: the Owl Alpha stealth run](https://decrypt.co/372579/longcat-2-0-meituan-ai-stealth-model-openrouter) - [SCMP: China's biggest model trained on local chips](https://www.scmp.com/tech/tech-trends/article/3358854/china-debuts-biggest-ai-model-trained-local-chips-meituan-releases-longcat-20) - [MarkTechPost: architecture breakdown](https://www.marktechpost.com/2026/07/05/meituan-releases-longcat-2-0-a-1-6t-parameter-open-moe-model-with-native-1m-context-and-longcat-sparse-attention/) - [LongCat-Flash technical report (arXiv)](https://arxiv.org/pdf/2509.01322) Tested on: not independently tested. LongCat-2.0 is a roughly 1.8T-parameter checkpoint that needs a multi-node GPU or NPU cluster, beyond our bench. All benchmark figures are Meituan-reported; OpenRouter traffic figures are community-reported and flagged as such. Sources linked above. Date checked: 2026-07-09 --- # Qwen 3.6 Source: https://singularitybyte.com/models/qwen-3-6.html Alibaba's Qwen 3.6 is two stories in one. The good one for open-source builders: a dense 27-billion-parameter model that beats the previous generation's 397B Mixture-of-Experts flagship on coding, and runs on a single 24GB GPU, under Apache 2.0. The other one, worth saying out loud: the actual flagship, Qwen3.6-Max, is closed and API-only, a quiet break from the fully-open [Qwen3.5](/models/alibaba-qwen-3-5.html) era. Here is what you can download, what you cannot, and how to run the part that is open. ## What is open, and what is not Alibaba's Qwen team released two genuinely open-weight Qwen 3.6 models in April 2026, both Apache 2.0 and both already past five million downloads: Qwen3.6-27B, a dense model, and Qwen3.6-35B-A3B, a Mixture-of-Experts model with about 3 billion active parameters. Both take text, image, and video, call tools, and carry a 256K-token context that stretches to roughly a million with YaRN scaling. What did not ship as weights is Qwen3.6-Max, the roughly trillion-parameter flagship, which exists only behind Alibaba's API. So "Qwen 3.6 is open" is true for the models most people will actually run, and false for the top of the lineup. Do not let a headline conflate them. ## The interesting model: a 27B that beats a 397B The dense 27B is the one to watch. Its trick is architectural: it leans on Gated DeltaNet, a linear-attention design, for three of every four layers, with standard attention in the rest. (Linear attention scales better with context length, which is how a 27B model holds a near-million-token window without melting.) It adds multi-token prediction for faster generation and keeps a thinking mode on by default. The payoff, in Qwen's own numbers, is that this 27B edges out the previous-generation Qwen3.5-397B-A17B on several coding benchmarks while fitting on a single consumer GPU. Intelligence per parameter, not raw scale. ## Benchmarks (Qwen-reported) The numbers below are from Qwen's own launch materials for the 27B. They compare mainly against its predecessors and Claude 4.5 Opus; there is no independent reproduction, and cross-vendor comparisons to Kimi or DeepSeek come only from third-party aggregators, so treat those as community-reported. Benchmark | Qwen3.6-27B | Note SWE-Bench Verified | 77.2 | beats Qwen3.5-397B (76.2) Terminal-Bench 2.0 | 59.3 | Qwen says on par with Claude 4.5 Opus LiveCodeBench v6 | 83.9 | coding GPQA Diamond | 87.8 | reasoning AIME 2026 | 94.1 | math Qwen-reported figures for the dense 27B, not independently reproduced. Comparisons beyond the Qwen3.5 family and Claude are third-party. ## Limitations and gotchas - The flagship is closed. Qwen3.6-Max is API-only; the open story stops at 27B and 35B-A3B. - Benchmarks are Qwen-reported. Strong, but unverified, and the headline comparison is against Qwen's own older model. - Vision adds friction for some runners: the vision projector file complicates plain Ollama GGUF use, so llama.cpp, vLLM, or SGLang are the safer paths. - Thinking on by default means more output tokens; turn it off when you just want a quick answer. ## Who should use it If you want a capable, openly licensed model that runs on one 24GB GPU and is genuinely good at code and tool use, the Qwen3.6-27B is one of the best picks of 2026, and a clear upgrade path from Qwen3.5 for local builders. Reach for the 35B-A3B MoE if you have a bit more memory and want faster inference per token. Skip the closed Max unless you specifically need its ceiling and are fine with an API. For most self-hosters, the open 27B is the story. ## Run it in about 10 minutes The dense 27B fits in roughly 17GB at 4-bit, so a 24GB card or a recent Mac runs it. ``` # Quantized local run via the community GGUF. A 24GB GPU or 32GB Mac is comfortable. ollama run hf.co/unsloth/Qwen3.6-27B-GGUF:Q4_K_M # Or serve the full weights with vLLM and the long context vllm serve Qwen/Qwen3.6-27B --max-model-len 262144 --reasoning-parser qwen3 ``` Give it a real repository task with tools enabled; tool calling and the long context are where the 3.6 generation pulled ahead of 3.5. If you run a coding agent, point a harness like OpenCode at the local endpoint and let it work. ## Sources and further reading - [Qwen3.6-27B model card (Hugging Face)](https://huggingface.co/Qwen/Qwen3.6-27B) - [Qwen3.6-35B-A3B model card (Hugging Face)](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) - [Qwen3.6 on GitHub (Apache 2.0)](https://github.com/QwenLM/Qwen3.6) - [MarkTechPost: the 27B benchmarks](https://www.marktechpost.com/2026/04/22/alibaba-qwen-team-releases-qwen3-6-27b-a-dense-open-weight-model-outperforming-397b-moe-on-agentic-coding-benchmarks/) Tested on: not independently tested. The Qwen3.6-27B is reported to run in about 17GB at 4-bit on a single 24GB GPU or a 32GB Mac; benchmarks are Qwen-reported and not independently reproduced, with cross-vendor comparisons drawn from third-party aggregators. The flagship Qwen3.6-Max is closed and was not evaluated. Date checked: 2026-06-26 --- # MisoTTS Source: https://singularitybyte.com/models/misotts.html Open text-to-speech has a quality gap and a license gap. Miso Labs went after both on June 3, 2026, with MisoTTS: an 8-billion-parameter voice model that clones a speaker from about ten seconds of audio, runs on a single 24GB GPU, and ships under a license you can actually build a product on. It is English-only and its quality claims are still unverified by anyone but Miso, but as an open, commercial-friendly voice model you can self-host, it earns a look. Here is the honest rundown. ## What MisoTTS is MisoTTS (the open-weights name for the model Miso also sells as "Miso One") generates speech from text, with an optional reference clip for one-shot voice cloning. Architecturally it follows the Sesame CSM lineage: a large Llama-3.2-style transformer backbone predicts audio over time, feeding a smaller decoder that fills in the finer detail, using the Mimi neural audio codec. It is 8B parameters total, handles up to 2,048 tokens of context, and watermarks its output by default. One honest limit up front: it speaks English only. ## The license is the quiet win Most open voice models arrive with a research-only or non-commercial license, which makes them useless for a product. MisoTTS ships under a modified MIT license that grants the full commercial rights you expect (use, modify, sell), with a single added condition: if your product crosses 50 million monthly users or 10 million dollars a month in revenue, you must display "Miso Labs" in the interface. For almost every builder, that is effectively unrestricted, and it is the most important thing about this release. Compare it to the gated weights of an image model like [Ideogram 4](/models/ideogram-4.html): same "open weights" label, very different freedom. ## What about quality? This is where you should keep your skepticism. Miso published exactly one number: latency, claimed at about 110 milliseconds, faster than a human's reaction time and well under the roughly 700 milliseconds it attributes to a closed competitor. There are no MOS scores (the standard human-rated naturalness metric), no word-error-rate numbers, and no independent A/B tests against [Orpheus](/models/orpheus-tts.html), CSM, or Fish Audio. The architecture is sound and the lineage is respectable, but "sounds great" is, for now, a vendor claim. Test it on your own voices before you trust it. ## Limitations and gotchas - English only at launch. No multilingual support. - Quality is unverified. The only published metric is latency; naturalness and accuracy have no independent numbers yet. - Output is watermarked by default (via SilentCipher), which matters if you plan to post-process the audio. - It models single turns, not live conversation; there is no turn-taking or full-duplex mode. - No documented Apple Silicon path; plan for an NVIDIA GPU. ## Who should use it Builders who need a self-hostable, commercially licensed English TTS with voice cloning, and who can run a 24GB GPU: think narration, dubbing prototypes, voice agents, and accessibility tools where sending audio to a paid API is a cost or privacy problem. If you need many languages, or you need proven naturalness scores today, look elsewhere for now. The pitch is freedom and control, with quality you verify yourself. ## Run it in about 10 minutes It needs roughly 24GB of VRAM in bf16 (an RTX 3090 or 4090 will do). The repo ships its own runner. ``` # Install uv, clone, and run the included demo curl -LsSf https://astral.sh/uv/install.sh | sh git clone https://github.com/MisoLabsAI/MisoTTS.git cd MisoTTS uv sync --python 3.10 && source .venv/bin/activate uv run python run_misotts.py ``` For voice cloning, pass a short reference clip alongside your text and MisoTTS matches its register and pacing: ``` # Sketch of the cloning call (see the repo README for exact arguments) from misotts import MisoTTS tts = MisoTTS.from_pretrained("MisoLabs/MisoTTS") audio = tts.generate( text="Ship it on Friday, not before the tests pass.", speaker_audio="reference_10s.wav", ) audio.save("out.wav") ``` If you would rather not set up a GPU, there is a community demo Space on Hugging Face. Throw a tricky sentence and your own voice at it, then judge the naturalness yourself, because right now nobody else has. ## Sources and further reading - [Miso Labs: MisoTTS announcement](https://www.misolabs.ai/blog/miso-tts-8b) - [MisoTTS model card (Hugging Face)](https://huggingface.co/MisoLabs/MisoTTS) - [MisoTTS on GitHub](https://github.com/MisoLabsAI/MisoTTS) - [MarkTechPost coverage](https://www.marktechpost.com/2026/06/04/miso-labs-releases-misotts-an-8b-emotive-text-to-speech-model-with-open-weights/) Tested on: not independently tested in our environment. MisoTTS is reported to run on a single 24GB GPU in bf16; the only published quality metric is a vendor latency claim (about 110 ms), with no independent MOS or word-error-rate numbers as of this writing. English-only, with watermarked output. Date checked: 2026-06-26 --- # Cosmos 3 Source: https://singularitybyte.com/models/nvidia-cosmos-3.html Most "open" AI releases this year were chatbots. NVIDIA's Cosmos 3, announced at GTC Taipei on June 1, 2026, is something different: an open world model that understands and generates not just text and video but ambient audio and robot actions, built for physical AI. The point is not to chat. It is to generate training data for robots, predict what happens next in the physical world, and output the action a robot should take. NVIDIA shipped the weights, the training data, and the recipes under a permissive license. Here is what it is, what is actually open, and what you can run. ## What a world foundation model is A language model predicts the next token. A world foundation model predicts the next state of the world: given a scene and an intent, what the camera sees next, what it sounds like, and what action moves a robot toward its goal. Cosmos 3 unifies three things earlier systems kept separate: physical reasoning (understanding a scene), world generation (producing realistic video and audio of what happens next), and action generation (the trajectory a robot should follow). That combination is why it is aimed at robotics, autonomous vehicles, and simulation rather than at your chat window. ## How it is built Cosmos 3 uses a two-tower "Mixture-of-Transformers" design. One tower is an autoregressive reasoner, a vision-language model initialized from Qwen3-VL (8B in Nano, 32B in Super). The other is a diffusion generator that produces the video, audio, and action output. They share one architecture and pass information from reasoner to generator, aligned across video, audio, and action tokens with 3D rotary position embeddings. So it is a hybrid: autoregressive where it reasons, diffusion where it generates. It was trained on roughly 20 trillion multimodal tokens, including about a billion images, 400 million videos, plus audio and human and robot action data. It comes in three sizes. Cosmos 3 Super (64B) is the frontier-accuracy model, available now. Cosmos 3 Nano (16B) targets a single workstation GPU for near-real-time use, available now. Cosmos 3 Edge (4B), the on-device low-latency variant, is announced but not yet downloadable, so treat it as a roadmap item. ## What is actually open This is genuine open-weights, and more. The shipped models (Nano and Super) are downloadable on Hugging Face under OpenMDW-1.1, a permissive Linux Foundation license that explicitly allows commercial and non-commercial use and also covers the architecture, code, datasets, benchmarks, and training recipes. NVIDIA released six synthetic datasets and the training scripts alongside the weights, plus a separate DROID policy model for robot manipulation. Like NVIDIA's [Nemotron 3 Ultra](/models/nvidia-nemotron-3-ultra.html), the story is full openness, not just a weights drop. ## Benchmarks and the honest caveats NVIDIA claims Cosmos 3 ranks first among open models on a stack of physical-AI benchmarks (Physics-IQ, PAI-Bench, RoboLab, RoboArena, and a traffic-anomaly reasoning leaderboard), and tops Artificial Analysis open leaderboards for text-to-image and image-to-video. These are NVIDIA-reported; there is no independent reproduction yet, and NVIDIA did not publish head-to-head tables against its own Cosmos 2 or against Google's Genie line, so cross-family comparisons are not available. ## Limitations and gotchas - It is NVIDIA-only and Linux-only: Ampere, Hopper, or Blackwell GPUs, CUDA 13, BF16. There is no Apple Silicon or AMD path. - Super needs a data-center GPU; Nano targets a workstation-class card (NVIDIA cites the RTX PRO 6000) for real-time robotics. This is not a laptop model. - The on-device Edge (4B) model, the one most edge builders want, is not released yet. - Benchmarks are NVIDIA-reported and physical-AI specific; if you are not doing robotics, AV, or simulation, this is probably not your model. ## Who should use it Roboticists training manipulation policies, autonomous-vehicle teams generating rare scenarios, and anyone who needs physics-accurate synthetic data instead of expensive real-world capture. The open weights plus the open datasets and recipes make it a genuine base to build on, not just an endpoint to call. If you build chat or coding products, skip it; this is a different branch of the tree. ## Run it in about 10 minutes The fastest taste is the hosted playground on build.nvidia.com or a NIM container; the local path is Hugging Face plus Diffusers. ``` # Hosted: a NIM container for the Nano reasoner (needs an NGC API key) docker run --gpus all -e NGC_API_KEY=$NGC_API_KEY \ -p 8000:8000 nvcr.io/nim/nvidia/cosmos3-reasoner:latest # Or pull the open weights from Hugging Face huggingface-cli download nvidia/Cosmos3-Nano ``` With the weights local, the omni pipeline rolls the world forward from a single image and a prompt: ``` # Generate a short world rollout from an image from diffusers import Cosmos3OmniPipeline pipe = Cosmos3OmniPipeline.from_pretrained("nvidia/Cosmos3-Nano").to("cuda") out = pipe(image="kitchen.png", prompt="the robot arm picks up the red mug") out.save("rollout.mp4") ``` If you do not have an NVIDIA workstation, the ten-minute move is the build.nvidia.com demo and the technical report; the two-tower reasoner-plus-generator design is the part worth studying even if you cannot run the weights yet. ## Sources and further reading - [NVIDIA: Cosmos 3 announcement](https://nvidianews.nvidia.com/news/nvidia-launches-cosmos-3-the-open-frontier-foundation-model-for-physical-ai) - [Cosmos3-Nano model card (Hugging Face)](https://huggingface.co/nvidia/Cosmos3-Nano) - [NVIDIA developer blog](https://developer.nvidia.com/blog/develop-physical-ai-reasoning-world-and-action-models-with-nvidia-cosmos-3/) - [NVIDIA/Cosmos on GitHub](https://github.com/NVIDIA/Cosmos) Tested on: not independently tested. Cosmos 3 needs an NVIDIA Ampere, Hopper, or Blackwell GPU on Linux (Super is data-center class), beyond our bench. Benchmarks are NVIDIA-reported with no independent reproduction; the Edge (4B) variant is announced but not yet downloadable. Date checked: 2026-06-26 --- ## Tools # PixelRAG: Visual RAG That Retrieves Screenshots Instead of Parsing Text Source: https://singularitybyte.com/tools/pixelrag-visual-rag.html Every RAG pipeline you have built starts by throwing away information. You parse a PDF to text, and the table that held the answer becomes a column of orphaned numbers. PixelRAG takes the other road: it renders documents to images and retrieves over the pixels, so tables, charts and layout survive intact. It hit 7,063 GitHub stars since late May, ships under Apache 2.0, and v0.4.0 landed on July 16. We installed it, rendered a real paper, and ran queries against its hosted 8.28-million-page Wikipedia index. Here is what holds up. ## TL;DR - What it is: a visual RAG pipeline (render, embed, index, search) that screenshots documents and retrieves image tiles using a LoRA-tuned Qwen3-VL-Embedding-2B, instead of parsing text. - Why it matters: it runs on a laptop CPU, it is Apache 2.0, and it fixes a failure mode every text pipeline has, where table structure and superscripts are silently corrupted before your model ever sees them. - The catch: retrieval returns a picture of a page region, not an answer, so you still need a vision model to read it. Top-1 precision was noisy in our tests, and the full Wikipedia index is about 217 GB. ## The problem, shown rather than asserted We rendered the original Transformer paper, "Attention Is All You Need", and pulled page 8. That page holds Table 2, a two-level header table: BLEU and Training Cost across the top, each split into EN-DE and EN-FR below. PixelRAG keeps that page as a 1700x2200 pixel image. Every rule line, every column boundary, every empty cell stays where the typesetter put it. Now here is the same page run through PyMuPDF, which is what a conventional pipeline feeds your model: ``` import pymupdf doc = pymupdf.open("attention.pdf") print(doc[7].get_text()) ``` The output flattens to this: ``` Model BLEU Training Cost (FLOPs) EN-DE EN-FR EN-DE EN-FR ByteNet [18] 23.75 Deep-Att + PosUnk [39] 39.2 1.0 · 1020 GNMT + RL [38] 24.6 39.92 2.3 · 1019 1.4 · 1020 ``` Three things just broke, and none of them raised an error. Empty cells vanished. ByteNet has an EN-DE score and no EN-FR score, so the text gives you one number with no way to know which column it came from. Column alignment died with it. Deep-Att shows "39.2" then "1.0 · 1020", and the natural reading is EN-DE BLEU and EN-DE cost. Both are wrong: 39.2 is the EN-FR score, and that cost belongs to EN-FR too. And the superscripts silently corrupted. Look at "1020". In the paper that is 10 to the power of 20. The text extraction turned an exponent into a four-digit number, so the training cost is off by eighteen orders of magnitude, in a string your model will read as fact. This is the entire argument for pixel-native retrieval, and you can reproduce it in two commands. Text parsing does not fail loudly. It fails into plausible, confidently wrong numbers. ## How it works The pipeline is five stages, each a separate CLI verb, so you can stop after any of them: render, chunk, embed, build-index, serve. Rendering uses a browser engine for URLs and HTML, and Poppler for PDFs, cutting each page into fixed-height tiles. Embedding runs Qwen/Qwen3-VL-Embedding-2B, a 2B vision-language embedding model that the project fine-tuned with LoRA on screenshot data, with the adapters published separately. Indexing writes to FAISS by default, or Qdrant if you point it at a server. Retrieval returns tile coordinates, not prose. A hit tells you the source document, which tile, and the vertical offset within the page. Handing that crop to a vision model is your job, which is why anthropic ships as a base dependency for the reader step. If you want the reader local instead, any vision model works, including the multimodal open-weights options we covered in the [Inkling](/models/thinking-machines-inkling.html) writeup. ## Hands-on: install and render One gotcha before you start. PyPI marks the package requires_python >=3.12, while the README still says 3.10 or newer. On Python 3.9 the install simply refuses, so start with a modern interpreter: ``` python3.13 -m venv venv ./venv/bin/pip install pixelrag # PDF support is NOT in the base install. # You also need poppler on the system (pdftoppm, pdfinfo). ./venv/bin/pip install 'pixelrag[pdf]' # Render any mix of URLs and PDFs to tiles ./venv/bin/pixelshot attention.pdf -o ./tiles ``` Skip that second line and rendering dies with "pdf2image is required for PDF rendering", which then tells you to install pixelrag-render[pdf]. That package name does not resolve. The one that works is pixelrag[pdf], as documented in the README. Minor, but it will cost you a few minutes. With that sorted, the 15-page paper rendered in 7 seconds on a 6-core CPU with no GPU, producing 15 tiles at 1700x2200 pixels for 6.5 MB total. That is the honest headline for local use: the render stage is cheap and needs no accelerator. Embedding is where a GPU starts to matter, and the project quotes roughly 3 minutes on Apple Silicon or 1 minute on a GPU to index a paper. ## Hands-on: querying the hosted index You do not have to index anything to try retrieval. The project hosts a prebuilt index of 8.28 million Wikipedia pages, open to unauthenticated POSTs: ``` curl -X POST https://api.pixelrag.ai/search \ -H "Content-Type: application/json" \ -d '{"queries": [{"text": "periodic table of elements"}], "n_docs": 3}' ``` Three queries, three responses, all between 2.18 and 2.36 seconds round trip from Europe. Results below, with the similarity score and the vertical offset of the winning tile: Query | Top result | Score | Tile offset periodic table of elements | Periodic_table | 0.665 | y=0 population of Germany by year table | Yablonovka,_Saratov_Oblast | 0.612 | y=1024 population of Germany by year table (2nd hit) | Demographics_of_Germany | 0.612 | y=5120 GPU architecture comparison chart | Nvidia_Drive | 0.608 | y=4096 Measured 2026-07-22 against the hosted index. Scores are cosine similarity as returned by the API. The offsets are the interesting part. For the Germany query it returned Demographics_of_Germany at y=5120, which is deep into a long page, exactly where the population tables live rather than the intro paragraph. That is the pitch working: it located a region of a page because that region looks like the answer. The same query is also the clearest miss. A Russian village, Yablonovka in Saratov Oblast, tied the correct article at 0.612 and sorted above it. Scores across all our queries clustered between 0.53 and 0.67, so there is no clean threshold separating a strong hit from noise. Retrieve several tiles and let the reader model discard the duds. Treat top-1 as a suggestion. ## Running it on your own documents ``` # Full local pipeline ./venv/bin/pip install 'pixelrag[index,serve]' ./venv/bin/pixelrag chunk --tiles-dir ./tiles ./venv/bin/pixelrag embed --shard-dir ./tiles --output-dir ./embeddings ./venv/bin/pixelrag build-index --embeddings-dir ./embeddings --output-dir ./index ./venv/bin/pixelrag serve --index-dir ./index --port 30001 curl -X POST http://localhost:30001/search \ -H "Content-Type: application/json" \ -d '{"queries": [{"text": "training cost table"}], "n_docs": 3}' ``` Or drive it from a config file, which is the cleaner route for a repeatable corpus: ``` # pixelrag.yaml source: type: local path: ./my_docs embed: model: Qwen/Qwen3-VL-Embedding-2B device: auto # cuda on Linux, mps on macOS, cpu fallback output: ./my_index ``` ## Limitations and gotchas - Retrieval gives you an image crop, not an answer. Budget for a vision model on the read step, and for the tokens that image will cost you. - Top-1 precision was unreliable in our sampling, and similarity scores cluster too tightly to threshold. Pull 3 to 5 tiles per query. - The prebuilt Wikipedia index is roughly 217 GB in FAISS format. The hosted API is the sane way to try it. - No published accuracy comparison against text-based RAG. The README argues the case by example, as we did above, but there are no benchmark tables yet. - Python 3.12 or newer despite what the README says, plus Poppler for PDFs and a Chromium for web pages. - Storage and bandwidth are real. Page images are heavy next to text chunks: our 2.2 MB PDF became 6.5 MB of tiles. - Qdrant users run their own server; only FAISS works out of the box. ## Who should use it Use PixelRAG if your corpus is documents that were designed to be looked at: financial filings, scientific papers, datasheets, slide decks, anything where a table or a chart carries the answer. The render stage costs nothing on CPU, so you can evaluate it on your own files this afternoon and see whether visual retrieval finds things your text pipeline misses. Stay with text RAG if your corpus is already clean text like chat logs, code, or Markdown docs, where rendering to pixels adds cost and removes the exact-match precision you get from keyword search. And if you need audited accuracy numbers before adopting anything, wait: this project is under two months old, and the evidence so far is demonstrative rather than measured. ## Sources and further reading - [GitHub: StarTrail-org/PixelRAG](https://github.com/StarTrail-org/PixelRAG) - [PixelRAG project site and hosted API](https://pixelrag.ai) - [PyPI: pixelrag 0.4.0](https://pypi.org/project/pixelrag/) - [Hugging Face: the screenshot embedding LoRA adapters](https://huggingface.co/Chrisyichuan/wiki-screenshot-embedding-lora) - [Attention Is All You Need (the PDF used in our test)](https://arxiv.org/abs/1706.03762) Tested on: AMD 6-core x86_64 · 8 GB RAM · no GPU · Linux 6.1 · Python 3.13 · PixelRAG 0.4.0 from PyPI · Poppler 22.12. Render timing measured on that machine; search latency measured against the hosted api.pixelrag.ai index from Europe. Embedding and indexing were not benchmarked locally, since the 2B embedding model wants a GPU we do not have on this bench. Date tested: 2026-07-22 --- # OpenCode Source: https://singularitybyte.com/tools/opencode.html If you write code with an AI agent in your terminal, you have probably already switched to OpenCode, or you are about to. The MIT-licensed coding agent crossed 179,000 GitHub stars and around 7.5 million monthly developers by late June 2026, ships a new release almost every day, and works with 75-plus model providers instead of locking you to one lab. Then on June 18, Google pulled the plug on Gemini CLI for free and consumer users, and the migration turned into a stampede. Here is what OpenCode is, why it won, and how to point it at the open models we have been covering. ## What OpenCode is OpenCode is an open-source AI coding agent that runs in your terminal, plus a beta desktop app and a VS Code extension. It is not a chat box. It reads your files, edits code, runs shell commands, and tests its own work, the full agentic loop. It is built by Anomaly, the team formerly known as SST. (One note to avoid confusion: the original Go version split off and continues as "Crush"; the TypeScript OpenCode at github.com/anomalyco/opencode is the one everyone means now, and the old sst/opencode URL just redirects there.) The defining choice is that OpenCode ships no model of its own. It is model-agnostic, talking to 75-plus providers through Models.dev: Claude, GPT, Gemini, DeepSeek, and any OpenAI-compatible endpoint, including local servers. It supports MCP (the Model Context Protocol, the now-standard way agents call external tools) and LSP (Language Server Protocol) for real code intelligence. It ships built-in agents (a read-write "build" agent, a read-only "plan" agent, a "general" subagent) and lets you define your own. ## Why it won: the Gemini CLI exodus OpenCode was already the most-starred open coding harness. Then Google handed it the market. On June 18, 2026, Google shut down Gemini CLI for free, Pro, and Ultra consumer users with no grace period, and is ending consumer Code Assist for GitHub on July 17. The replacement, Antigravity CLI, is closed-source and cut the free tier from roughly 1,000 requests a day to about 20, a 98 percent reduction. For a developer who built a workflow around a free, open terminal agent, that is a forced migration. OpenCode is the obvious landing spot, because it still talks to Gemini models through the API if you want them, alongside 74 other providers, with no vendor able to switch it off. Qwen Code, Alibaba's Apache-2.0 fork of the old Gemini CLI, picked up the overflow. The lesson writes itself: an open, model-agnostic tool cannot be discontinued out from under you. ## One honest caveat about benchmarks You will see people quote SWE-Bench or Terminal-Bench numbers "for OpenCode." Ignore them. Because OpenCode has no built-in model, it has no intrinsic score; its ceiling is whatever model you feed it. Point it at Claude Opus 4.8 or GPT-5.5 and you get top-of-leaderboard coding (Opus 4.8 lands around 74 percent on Terminal-Bench 2.1 and 88 percent on SWE-Bench Verified). Point it at a weak local model and you get weak results. The tool is the harness, not the brain. ## Install it and point it at an open model in 10 minutes Install is one line, by whatever package manager you already use. ``` # pick one npm i -g opencode-ai@latest brew install anomalyco/tap/opencode curl -fsSL https://opencode.ai/install | bash # then just run it inside a project opencode ``` The interesting part for this site is wiring it to an open model instead of a closed API. OpenCode reads an opencode.json in your project. Here it is pointed at a local server (Ollama, llama.cpp, or LM Studio) over the OpenAI-compatible API; the same pattern works for a rented GPU running GLM-5.2 or Qwen3.6: ``` { "$schema": "https://opencode.ai/config.json", "provider": { "local": { "npm": "@ai-sdk/openai-compatible", "options": { "baseURL": "http://localhost:11434/v1" }, "models": { "qwen3.6:27b": { "name": "Qwen3.6 27B (local)" } } } }, "model": "local/qwen3.6:27b" } ``` Switch models mid-session with the /models command. That is the whole pitch: one agent, every model, including the open ones you can actually own. Pair it with [GLM-5.2](/models/glm-5-2.html) for cheap agentic coding, or a local [Qwen](/models/alibaba-qwen-3-5.html) when you want it air-gapped. ## Who should use it Anyone who wants a serious terminal coding agent without lock-in: Claude Code users who want model freedom, ex-Gemini-CLI users who just got evicted, and anyone running local or open-weight models who needs a harness that speaks their endpoint. If you are deep in a single closed ecosystem and happy there, you may not need it. Everyone else: the switching cost is one install command. ## Sources and further reading - [OpenCode official site](https://opencode.ai/) - [OpenCode on GitHub (MIT)](https://github.com/anomalyco/opencode) - [OpenCode docs: providers and local models](https://opencode.ai/docs/providers/) - [9to5Google: Gemini CLI shutting down](https://9to5google.com/2026/06/17/gemini-cli-code-assist-shutting-down/) Tested on: not benchmarked. OpenCode is model-agnostic and has no intrinsic score; its quality equals the model you connect. The facts here (stars, version v1.17.11, 75-plus providers, the June 18 Gemini CLI shutdown) are verified against the GitHub repo and primary coverage. Date checked: 2026-06-26 --- # StableDAW: One-Click AI Music Studio Source: https://singularitybyte.com/tools/stabledaw-ai-music-studio-pinokio.html StableDAW just landed in the Pinokio catalog, and it is the first AI music tool that feels like an actual studio instead of a text box with a download button. It is a browser-based digital audio workstation built around Stability AI's open-weight Stable Audio 3 models, it installs in one click, and the small model runs on a plain CPU. No account, no API key, no cloud credits. You generate, edit, mix, and perform entirely on your own machine. The catch up front, so nobody wastes a download: Stable Audio 3 makes instrumental music and sound effects. It does not sing. If you need vocals, this is not your tool. If you want a local, MIT-licensed beat factory that you fully control, read on. StableDAW, branded "theDAW" in the app, was built by GANTASMO at a Music Hackspace hackathon. ## What StableDAW actually is StableDAW (the app calls itself "theDAW") was built by [GANTASMO](https://github.com/gantasmo) during a Music Hackspace music-technology hackathon at Berklee College of Music. The code is [MIT-licensed](https://github.com/gantasmo/StableDAW) on GitHub, and [cocktailpeanut](https://github.com/cocktailpeanut) wrote the Pinokio launcher that turns the whole stack into a one-click install. It is not a one-trick prompt-to-clip generator. It is a full workstation with seven workspaces, laid out as a left-to-right workflow across the top tab bar: - MAKE generates audio from text, from your own recordings, or both. Text-to-audio, audio-to-audio, inpainting, continuation, and a "Chimera" blend that fuses multiple clips. - EDIT is a multitrack timeline with clip arrangement, trimming, fades, and a snap grid. - MIX hosts 24 FFmpeg-powered effects, a mastering chain, a drum sequencer, and a piano roll, and exports to FLAC, MP3, AAC, or Opus. - DJ is a two-deck console with sync and FX, and it maps to MIDI controllers. - VJ drives a 3D audio-reactive visualizer for live shows. - TRAIN fine-tunes LoRA adapters on your own audio (this one needs an NVIDIA GPU). - LEARN draws an interactive genealogy graph that traces how every clip descended from its parents. That last one is the tell. StableDAW keeps a persistent, searchable library and remembers the full lineage of every render: which prompt made it, which clip it was inpainted from, which stems it was split into. For anyone who has lost a good take to a closed cloud tool, that local history is the selling point. ## The engine: Stable Audio 3, open weights, no vocals Under the hood is [Stable Audio 3](https://huggingface.co/stabilityai/stable-audio-3-medium), a diffusion transformer paired with Stability AI's SAME autoencoder. It outputs 44.1 kHz stereo, which is CD quality. StableDAW ships two checkpoints, and the difference decides what hardware you need. Model Size Hardware Max length Use it for Small (433M) ~3.5 GB CPU or any GPU 120 sec Default. Loops, sketches, SFX, laptops without a GPU. Medium (1.4B) ~10.4 GB NVIDIA, ~8 GB VRAM 380 sec Full-length tracks, higher musicality. The launcher pulls both from ungated community mirrors (cocktailpeanut/stable-audio-3-* on Hugging Face), so there is no license click-through and no token. The Small model downloads automatically; you grab Medium from a button when you want it. A separate Medium-RF checkpoint is the base for LoRA training in the TRAIN tab. On licensing, the app code is MIT. The model weights sit under the Stability AI Community License, which lets organizations under 1 million dollars in annual revenue use the generated audio commercially. Past that threshold you need an enterprise agreement. For an indie producer or a small studio, the practical answer is: the music you make is yours. ## Install it in one click with Pinokio If you have not used [Pinokio](https://pinokio.computer) before, it is a desktop launcher that installs and runs AI apps inside their own isolated environments, no terminal required. Install Pinokio first, then add StableDAW from its repository page. ``` # 1. Install Pinokio from pinokio.computer (Windows, Linux, or macOS) # 2. In Pinokio, open the StableDAW launcher repo: # github.com/cocktailpeanut/stabledaw.pinokio # 3. Click Install, then Start. The launcher handles the rest. ``` Behind that one click, the launcher clones the repo, builds a Python 3.10 environment with the right torch for your platform via uv sync, runs npm ci for the React frontend, and downloads the ~3.5 GB Small model. Then it boots a FastAPI backend on port 8600 and a Vite frontend on port 5173, and opens the web UI. Two things to budget for. You need roughly 10 to 15 GB of free disk for the environment plus the Small model, and another ~10.4 GB if you add Medium. And both ports (8600 and 5173) must be free, because Vite runs with strictPort and will not shop around for another one. Platform notes: Windows installs CUDA 12.8 and Flash Attention automatically, Linux x86_64 wants CUDA 12.6, and macOS runs the Small model on CPU. That includes Apple Silicon, which works fine without any NVIDIA hardware, just slower than a CUDA box. ## Hands-on: making your first track The MAKE tab is where you live at the start. Drop a description into the prompt box, set the controls on the left (model, length, sampler steps, CFG, seed, batch), and hit CREATE in the bottom right. Be specific. Genre, tempo, instrumentation, and mood all move the output. The MAKE workspace: prompt boxes up top, sampler controls on the left, the Chimera blend stage in the center, output format on the right. A prompt that works well as a starting point: ``` Lo-fi boom bap with dusty vinyl crackle, warm Rhodes chords, mellow upright bass, soft swung drums, 84 BPM ``` Generation speed depends entirely on the model and the silicon. On a recent NVIDIA card the Small model returns a two-minute clip in a few seconds. On CPU, the same clip takes longer, and the Medium model on CPU is measured in minutes, not seconds, so save it for the GPU. These are community-reported figures; we cover the disclosure at the end. Once you have a clip you like, the generative tools stack. Paint a region and inpaint just that bar. Extend a loop with continuation. Or drag several clips into Chimera and let it beat-align them into one new generation. Every result drops into the library with its full prompt and parameters, so a good seed is never lost. ## Beyond generation: edit, mix, perform This is where StableDAW earns the "DAW" in its name. The EDIT tab gives you a real multitrack timeline. You arrange clips, trim them, add fades, and commit edits, all in the browser. The EDIT tab: a standard multitrack timeline. Drop clips from the library and arrange them. The MIX tab is a surprise for a hackathon project. It carries 24 FFmpeg-powered effects grouped into a library (mastering chain, vocal processing, lo-fi and vinyl, stereo widener, reverb and delay, club EQ, compressor, loudness normalization, and more), a QUICK MASTER section with punch, air, drive, and ceiling knobs, plus a drum sequencer and a piano roll. You chain effects left to right and hit process. The MIX tab: an FFmpeg effects library, a quick-master section, and an effects chain you build left to right. Then it keeps going. The DJ tab is a two-deck console with sync, per-deck FX, and MIDI mapping. The VJ tab is a 3D audio-reactive visualizer for live performance. And the TRAIN tab lets you fine-tune LoRA adapters on a folder of your own audio, then stack them at generation time for a custom sound, provided you have the Medium-RF model and an NVIDIA GPU. ## Limitations and gotchas It is a hackathon build, and a few rough edges show: - No vocals, period. Stable Audio 3 is instrumental and SFX only. Pair it with a separate vocal tool if you need singing. - CPU is slow. The Small model is usable on CPU; Medium really is not. Treat a GPU as the requirement for full-length work. - First boot lags. The initial PyTorch import takes about a minute. That is normal, not a hang. - The Reset button is a real reset. It wipes your generated-audio library. Export anything you care about first. - The original repo is archived. Active development moved to [gantasmo/theDAW](https://github.com/gantasmo/theDAW). The Pinokio launcher tracks the working build, so the one-click path still works, but watch the new repo for updates. ## Who should use it, and what to do in the next 10 minutes StableDAW is for producers, indie hackers, and anyone who wants AI music generation they actually own: local files, open weights, full edit history, and a clear commercial path under the Community License. It trades the polished vocals of closed tools for control, transparency, and a studio that never phones home. The 10-minute version: install Pinokio, add the StableDAW launcher, click Start, and let the Small model download. While it pulls, free up ports 8600 and 5173. When the UI opens, paste the lo-fi prompt above into MAKE and hit CREATE. You will have a track playing before your coffee is cold, and it will be sitting in a library on your own disk, not someone else's server. If you like running models locally, our [tools section](tools.html) has more open-source workstations worth your disk space. ## Sources and further reading - [Pinokio announcement: StableDAW, a full AI music studio that runs on your own machine](https://beta.pinokio.co/posts/01kty98bwb20619q6274768kcm) - [StableDAW on GitHub (gantasmo/StableDAW)](https://github.com/gantasmo/StableDAW) - [Pinokio launcher repo (cocktailpeanut/stabledaw.pinokio)](https://github.com/cocktailpeanut/stabledaw.pinokio) - [Stable Audio 3 Medium model card on Hugging Face](https://huggingface.co/stabilityai/stable-audio-3-medium) Tested on: install walkthrough verified against the Pinokio launcher and the StableDAW GitHub README. Generation timings are community-reported, not benchmarked on our own hardware. Model version: Stable Audio 3 Small (433M) and Medium (1.4B) via the StableDAW launcher. Date tested: 2026-06-15 --- # Google TurboQuant Source: https://singularitybyte.com/tools/google-turboquant.html Long context is a memory problem. The KV cache, the running store of key and value vectors that attention reads back on every token, balloons with sequence length and quietly eats your VRAM. Google Research's TurboQuant attacks exactly that: it squeezes the KV cache to 2.5 to 4 bits with no calibration data and near-optimal error. The community already wired it into llama.cpp and vLLM, so you can try it this afternoon. The headline numbers, though, need an asterisk or three. ## What TurboQuant actually is First, the thing people keep getting wrong: TurboQuant quantizes the KV cache, not the model weights. It does not replace GPTQ, AWQ, or your GGUF Q4 weights. It stacks on top of them. Weight quantization shrinks the model on disk and in memory; TurboQuant shrinks the per-token cache that grows as you generate. Different problem, complementary fix. It comes from a Google Research paper, "TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate" (Zandieh, Daliri, Hadian, Mirrokni), posted to arXiv in April 2025 and accepted at ICLR 2026. Google blogged about it in March 2026. Two properties make it interesting for builders: it is online (compresses vectors as they are written, no separate pass) and it is data-oblivious (no calibration set, no tuning run). You point it at a model and go. ## How it works The trick is a random rotation. Quantization hates outliers, because a few large coordinates force a coarse scale on everything else. TurboQuant multiplies each key or value vector by a random orthogonal matrix first, which spreads outlier energy evenly across dimensions and pushes the per-coordinate distribution toward a predictable shape. After that rotation, each coordinate is close to independent, so a per-coordinate Lloyd-Max quantizer (the analytically optimal scalar quantizer for a known distribution) does the rest. Because the distribution is known ahead of time, there are no per-block scales or zero-points to store, which is where the bit savings come from. That is the MSE variant. There is a second variant tuned for attention. Attention does not care about reconstructing vectors, it cares about dot products. So TurboQuant runs the MSE quantizer at one bit lower, then applies a 1-bit Quantized Johnson-Lindenstrauss transform to the residual error. At read time, a special estimator combines both parts to recover an unbiased inner product. The paper argues the whole scheme lands within a small constant factor of the information-theoretic distortion floor. ## The numbers, and the asterisks Google's reported results are genuinely good. On Llama-3.1-8B-Instruct, long-context quality barely moves at 3.5 bits per channel. Config | LongBench avg | Needle-in-a-Haystack | Source FP16 baseline | 50.16 | 0.997 | paper TurboQuant 3.5-bit | 50.06 | 0.997 | paper TurboQuant 2.5-bit | 49.44 | ~baseline | paper Pair that with the memory math (roughly 6x smaller KV cache at 3-bit, and up to 8x faster attention at 4-bit on an H100) and it sounds like a free lunch. It is not, and the people who run inference for a living said so. The vLLM team published the first large independent study in May 2026, testing Llama-3.3-70B, Qwen3-30B-A3B, and others on H100s. Their verdict: plain FP8 KV cache is still the best default. Here is the shape of it. Method | KV capacity | Throughput vs BF16 | Accuracy cost | Source FP8 KV | 2.0x | ~100% | ~0 | vLLM (community) TurboQuant k8v4 | 2.4x | ~75 to 80% | ~0 | vLLM (community) TurboQuant 4bit-nc | up to 3.4x | ~75% | ~4 pts reasoning | vLLM (community) TurboQuant 3bit-nc | highest | ~66 to 73% | ~20 pts reasoning, ~30% long-ctx retrieval at 256K | vLLM (community) Read that bottom row twice. At aggressive 3-bit settings on a 256K context, long-context retrieval drops by about 30 percent and reasoning accuracy falls roughly 20 points. The compression is real, but so is the tax, and FP8 gets you 2x for free with no measurable loss. TurboQuant earns its keep when you are memory-bound and FP8's 2x is not enough, not when you are chasing raw throughput. ## How to run it today There is no official Google package. Every usable build is a community fork. Two are worth your time. ### llama.cpp (local, Apple Silicon or a single GPU) Community forks add tq3_0 (3-bit, about 4.9x KV compression) and tq4_0 (4-bit, about 3.8x) cache types. Build for your backend, then pass the cache type at runtime: ``` # Build (pick your backend) cmake -B build -DGGML_METAL=ON && cmake --build build -j # Apple Silicon # cmake -B build -DGGML_CUDA=ON && cmake --build build -j # NVIDIA # Run with a 4-bit TurboQuant KV cache ./build/bin/llama-cli -m model.gguf \ --cache-type-k tq4_0 --cache-type-v tq4_0 \ -c 32768 -p "Summarize this long document..." ``` ### vLLM (server, NVIDIA) The community vLLM fork exposes TurboQuant as a kv-cache dtype. Build from source, then serve: ``` vllm serve /models/your-model \ --tensor-parallel-size 4 \ --attention-backend TRITON_ATTN \ --kv-cache-dtype turboquant35 \ --enable-turboquant ``` The under-10-minute move: take a model that currently runs out of memory at your target context, switch the KV cache to tq4_0 (or turboquant35 on vLLM), and see whether the longer context now fits. If it does and your eval holds, you win memory for almost nothing. If quality dips, step up the bit-width or fall back to FP8. ## The caveats nobody puts on the slide Three things temper the hype. No official code. Google published a paper and a blog post, not a library. TechCrunch called it a lab breakthrough, and that is accurate. Anything labeled "Google's TurboQuant library" is community code, maintained by individuals, not Google. The RaBitQ dispute. The core move, random rotation before quantization, is the same idea behind RaBitQ (Gao and Long, 2024). Critics on OpenReview and elsewhere argue the paper mischaracterized RaBitQ's method and guarantees, and that the speed comparison pitted TurboQuant on an A100 GPU against RaBitQ on a single CPU core with multithreading switched off. One author acknowledged multiprocessing was disabled. Treat the head-to-head speed claims against RaBitQ as unsettled. The famous numbers are narrow. The 8x is attention-logit computation only, not end-to-end inference. The "zero accuracy loss" applies to the 3.5-bit setting on long-context benchmarks; the paper never ran MMLU, GSM8K, or HumanEval, and the vLLM study found real drops once you push to 3-bit or stress reasoning. ## Who should use it, and who should wait Try it if you run long contexts on memory-bound hardware: a consumer GPU, a Mac, an edge box that hits out-of-memory before it hits a speed wall. A 4-bit KV cache can be the difference between a 32K context fitting or not. Models like [Gemma 4](/models/google-gemma-4.html), [Qwen3.5](/models/alibaba-qwen-3-5.html), and [DeepSeek-V4](/models/deepseek-v4.html) are all popular community targets, and [MiniMax](/models/minimax-m3.html)'s long-context models showed up in the vLLM test set too. Wait, or just use FP8, if you are on datacenter GPUs optimizing for throughput, or running aggressive low-bit configs on reasoning-heavy workloads. In those cases FP8 KV cache gives you 2x with no throughput penalty and no accuracy hit, and that is hard to beat. ## Sources and further reading - [TurboQuant paper (arXiv 2504.19874)](https://arxiv.org/abs/2504.19874) - [Google Research announcement](https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/) - [vLLM: first comprehensive TurboQuant study](https://vllm.ai/blog/2026-05-11-turboquant) - [llama.cpp community discussion and benchmarks](https://github.com/ggml-org/llama.cpp/discussions/20969) - [The RaBitQ attribution dispute](https://dev.to/gaoj0017/turboquant-and-rabitq-what-the-public-story-gets-wrong-1i00) - [ICLR 2026 OpenReview thread](https://openreview.net/forum?id=tO3ASKZlok) Tested on: not independently tested. The figures here are paper- or community-reported via the sources above, with the vLLM study and the RaBitQ dispute flagged as such. Date checked: 2026-06-09 --- # Unsloth Source: https://singularitybyte.com/tools/unsloth-the-indie-fine-tuning-backbone.html If you run fine-tuning on a single GPU, you have almost certainly used [Unsloth](https://github.com/unslothai/unsloth) and you probably did not realize it is a two-person operation. Daniel and Michael Han ship the library that quietly turned LLM fine-tuning from "rent eight H100s for a week" into "borrow an RTX 4090 for an afternoon," then went home and did it again the next day. 64,000+ GitHub stars, two thousand more every month, full Apache-2.0 core. If the local-LLM scene has unpaid infrastructure on the inference side (the GGUF quant-masters), Unsloth is the equivalent on the training side. ## What Unsloth actually does Unsloth is a drop-in replacement for HuggingFace Transformers + PEFT specifically for the fine-tuning step. The headline claim from the [official benchmarks](https://unsloth.ai/docs/blog/3x-faster-training-packing) is roughly 2x faster training and 30 to 70 percent less VRAM, with the speedups jumping further on short-sequence datasets and Qwen3-class models. The savings come from hand-tuned Triton kernels, smarter memory layout for LoRA adapters, and a packing implementation that beats Flash Attention 3 on most realistic training mixes. Concrete numbers from the docs: a 7B model needs about 5 GB of VRAM for QLoRA, 19 GB for full LoRA. A 70B model fits in 41 GB of VRAM with QLoRA, which means a single A6000 can fine-tune Llama 3.3 70B. None of that is achievable with a stock HuggingFace pipeline on the same hardware. ## Install and first run ### Install ``` pip install unsloth ``` Python 3.9 or newer, PyTorch 2.0 or newer, and a CUDA 7.0 or newer GPU. That covers everything from a V100 through Blackwell. The current release also lights up on RTX 3090, 4090, and 5090, plus the standard datacenter parts. Apple Silicon is in progress; MLX training is marked "coming soon" in the [requirements page](https://unsloth.ai/docs/get-started/fine-tuning-for-beginners/unsloth-requirements). ### Fine-tune Llama 3.1 8B in a few lines ``` from unsloth import FastLanguageModel import torch model, tokenizer = FastLanguageModel.from_pretrained( model_name = "unsloth/Meta-Llama-3.1-8B", max_seq_length = 2048, dtype = None, load_in_4bit = True, ) model = FastLanguageModel.get_peft_model( model, r = 16, target_modules = ["q_proj","k_proj","v_proj","o_proj", "gate_proj","up_proj","down_proj"], lora_alpha = 16, use_gradient_checkpointing = "unsloth", ) ``` From there you point a SFTTrainer from TRL at your dataset and let it run. The library ships notebooks for every supported model on the [model catalog page](https://unsloth.ai/docs/get-started/unsloth-model-catalog), so the cold-start path is "click the notebook, swap your dataset path, run." ## Model support The catalog is wide. The major families currently maintained: Llama 4, 3.3, 3.2, 3.1; Mistral 3.2, 3.1, Ministral; Gemma 4, 3n, 2; Qwen 3.6, 3.5, 3, 2.5; Phi 4; DeepSeek-V3 and R1; GLM. Vision-language models, audio models, and Whisper-class TTS fine-tuning are also supported via the [vision and multimodal collection](https://huggingface.co/collections/unsloth/vision-multimodal-models). Output formats matter as much as input support. Unsloth ships exporters for 4-bit, 8-bit, and 16-bit safetensors, GGUF for llama.cpp and Ollama, and the vLLM format for production serving. The path from "I fine-tuned a model" to "I am serving it on my own hardware" is one CLI command, not an afternoon of conversion scripts. ## Beyond plain SFT Reward modeling and preference optimization are first-class. The [RLHF docs](https://docs.unsloth.ai/basics/reward-modelling-dpo-and-orpo) cover DPO, ORPO, KTO, and GRPO with example notebooks. Recent releases added vision fine-tuning and audio model training, and the catalog now includes speech models like Sesame and Orpheus. The pace is unusual for a project this size: notable feature drops land monthly, and minor releases land weekly. ## The two-person team Unsloth is built by [Daniel Han](https://x.com/danielhanchen) and his brother Michael, brothers from Sydney who went through Y Combinator's summer 2024 batch. Daniel previously worked on optimized t-SNE and SVD at NVIDIA, which is exactly the right background for the hand-written kernel work that drives Unsloth's speedups. The pair posts kernel-level deep dives on the [Unsloth blog](https://unsloth.ai/about) and Twitter, and most of the framework's heavyweight performance jumps have been written up there before they ship. ## Who uses Unsloth in the wild The fingerprint shows up across the open-weights ecosystem. Hugging Face used Unsloth for [SmolLM3-3B](https://huggingface.co/unsloth/SmolLM3-3B), and the documentation pages call out direct collaborations with Qwen, Mistral, NVIDIA, and Microsoft. Beyond the named users, scanning HuggingFace model cards for "fine-tuned with Unsloth" returns thousands of community results across every base family the library supports. ## Language-specific fine-tuning is now trivial One under-rated use case is language-specific instruction tuning on a small base. If you want a Polish-speaking Llama 3.2 3B that performs better on Polish dialogue than the multilingual base, Unsloth is the path of least resistance: the [Qwen3.5 documentation](https://unsloth.ai/docs/models/qwen3.5/fine-tune) covers 201 languages and Gemma 4 covers 140, so the underlying models already speak Polish (or Czech, or Slovenian, or any of the 200-language tail). Adding a few thousand Polish instruction-output pairs is a single notebook away. The catch most teams hit is dataset quality, not training infrastructure. ## Limitations and gotchas Unsloth is still a single-GPU framework at heart. Multi-GPU support exists but is less battle-tested than the single-GPU path. The "2x speedup" headline depends on baseline; against a naive HuggingFace pipeline the speedup is sometimes much larger, and against a hand-tuned Flash Attention 3 pipeline it's smaller. Numbers in [the published methodology](https://unsloth.ai/docs/blog/3x-faster-training-packing) are honest about that. And the GGUF and vLLM export paths assume llama.cpp or vLLM in the latest minor; older runtime versions occasionally choke on tokenizer additions. ## Who should care If you fine-tune on anything smaller than an 8-GPU H100 node, you should be using Unsloth. The library is free, the model coverage is comprehensive, the export path lands on every common inference runtime, and the maintainer cadence is healthy. The cost of trying it is one pip install and a sample notebook. Pick a base model, grab a thousand examples, and have your first fine-tune running in the time it takes to read this article. ## Sources and further reading - [Unsloth on GitHub](https://github.com/unslothai/unsloth) - [Unsloth model catalog](https://unsloth.ai/docs/get-started/unsloth-model-catalog) - [Hardware and software requirements](https://unsloth.ai/docs/get-started/fine-tuning-for-beginners/unsloth-requirements) - [Benchmark methodology: 3x faster training with packing](https://unsloth.ai/docs/blog/3x-faster-training-packing) - [DPO / ORPO / KTO / GRPO documentation](https://docs.unsloth.ai/basics/reward-modelling-dpo-and-orpo) - [Daniel Han on X](https://x.com/danielhanchen) Numbers community-reported from Unsloth's published methodology and HuggingFace model card metadata. Not independently re-run. Compiled 2026-05-19. --- # Heretic: One-Command Abliteration Source: https://singularitybyte.com/tools/heretic-one-command-abliteration.html Abliteration used to be a research move that took afternoon-long PyTorch sessions, hand-picked layers, and a stomach for spending weights you might never get back. Heretic compresses that into one command. pip install -U heretic-llm, then heretic Qwen/Qwen3-4B-Instruct-2507, and roughly 45 minutes later you have an uncensored variant whose damage to the base model is smaller than anything a human has hand-tuned for it. There are already more than a thousand community Heretic variants on HuggingFace, and the wait between a fresh open-weights release and its decensored twin is collapsing toward zero. ## Why abliteration suddenly matters to open-source builders For anyone running local models, safety alignment is a tax. It pays for itself when you ship a consumer chatbot, but it gets in the way of agent loops, red-team work, fiction, security research, creative writing, and dozens of legitimate uses that hit a wall the moment the model decides to apologize. Until recently, the only way around it was a curated jailbreak prompt or a full fine-tune. Heretic kills the wait. The day a new base model lands on HuggingFace, somebody runs heretic against it, pushes the result, and within hours [bartowski](https://huggingface.co/bartowski) and [mradermacher](https://huggingface.co/mradermacher) have GGUFs out for every common quantization. That is what the 1,000+ Heretic uploads on HuggingFace actually represent: an end-to-end pipeline where any aligned open model is one command and a few hours of community labor away from being usable for whatever you actually want to do with it. ## What abliteration actually is Abliteration is a surgical edit to a model's weights, not a fine-tune. It rests on a 2024 result from [Arditi et al.](https://arxiv.org/abs/2406.11717) (NeurIPS 2024), which showed that refusal in transformer language models is mediated by a single direction in the residual stream. That direction is consistent across 13 open-weights chat models, all the way up to 72B parameters. Project it out of the activations, or subtract it from the relevant weight matrices, and the model stops refusing without losing the rest of its skills. The technique reached a wider audience through [Maxime Labonne's HuggingFace blog post](https://huggingface.co/blog/mlabonne/abliteration), which turned the Arditi paper into a recipe anyone with a GPU could follow. The recipe worked, but it was finicky. You had to pick which layers to read activations from, decide how strongly to project, and eyeball whether the resulting model was still coherent. Get the layer wrong and the model talked freely but lost its math. Project too weakly and it still refused. ## How Heretic differs from the manual approach Heretic treats abliteration as an optimization problem. It uses [Optuna's Tree-structured Parzen Estimator](https://optuna.readthedocs.io/en/stable/reference/samplers/generated/optuna.samplers.TPESampler.html) to search the joint space of layer ranges, ablation weights, and direction indices, co-minimizing two objectives at every trial: the count of refusals on a harmful-prompt probe set, and the KL divergence between the patched model and the original. The model that comes out the other end is the one that refuses least while drifting least. The numbers from the repo are striking. A search that takes 30 to 90 minutes on a single RTX 3090 now outperforms expert tuning that took days. Approach (Gemma-3-12B-IT) | KL divergence vs base | Refusals / 100 | Human effort Heretic, default config | 0.16 | 3 | One CLI command Best hand-tuned abliteration | 1.04 | ~3 | Days of expert tuning Same refusal rate, roughly 6.5 times less damage to the base model's behavior, and the cost of producing the Heretic row is one terminal command on a consumer GPU. Heretic builds on extensions to the original technique, including Lai 2025's [projected abliteration](https://huggingface.co/blog/grimjim/projected-abliteration) and [norm-preserving biprojected abliteration](https://huggingface.co/blog/grimjim/norm-preserving-biprojected-abliteration). The author, [p-e-w](https://github.com/p-e-w) (Philipp Emanuel Weidmann), keeps the codebase under AGPL-3.0 and ships sensible defaults so the typical run is a single argument. ## Hands-on: from pip install to your own variant ### Install ``` pip install -U heretic-llm ``` Heretic needs Python 3.10 or newer, PyTorch 2.2+ (2.6 recommended), and a CUDA GPU. An RTX 3090 with 24 GB of VRAM is the sweet spot for 8B to 12B models. Hybrid models like Qwen3.5 work, multimodal models work, and most MoE architectures work. Pure state-space models and a handful of research architectures are still on the to-do list. ### Run ``` heretic Qwen/Qwen3-4B-Instruct-2507 ``` That is the whole interface. Heretic streams the model, benchmarks batch sizes on your GPU, runs the optimizer for a few dozen trials, prints the final KL and refusal numbers, and writes the patched weights to disk. If you want to bias the search toward retaining more or refusing less, the config.default.toml and config.noslop.toml presets in the repo are good starting points. ### Quantize and share Patched weights are large. The community convention is to upload the full-precision Heretic variant to your HuggingFace account, then wait a day or two for bartowski or mradermacher to publish imatrix GGUFs in the usual Q4_K_M, Q5_K_M, and IQ4_XS flavors. If you cannot wait, the llama.cpp conversion scripts handle the same job in one pass. ## Names to know in the scene [p-e-w](https://github.com/p-e-w) is the author of Heretic and the maintainer of the [heretic-org](https://huggingface.co/heretic-org) HuggingFace organization. The repo is the canonical reference; the org curates clean Heretic variants for popular base models. [mlabonne](https://huggingface.co/mlabonne) popularized abliteration in the first place and continues to ship merges, fine-tunes, and educational material. His blog is the recommended entry point for anyone who wants to understand what the projection step is doing. [huihui-ai](https://huggingface.co/huihui-ai) runs the most prolific uncensoring operation on HuggingFace, with 200+ variants spanning Qwen, DeepSeek, Llama, Gemma, and most major open families. If a base model dropped last week, huihui-ai almost certainly has an abliterated version of it already. [DavidAU](https://huggingface.co/DavidAU) takes things further. His "Dark Champion" line is a series of MoE merges that combine abliterated experts with creative-writing fine-tunes, and the resulting models have a cult following among fiction writers and roleplayers running local stacks. The [Llama-3.2-8X3B Dark Champion](https://huggingface.co/DavidAU/Llama-3.2-8X3B-MOE-Dark-Champion-Instruct-uncensored-abliterated-18.4B) is a representative entry. [bartowski](https://huggingface.co/bartowski) and [mradermacher](https://huggingface.co/mradermacher) are the GGUF quant-masters. Between them they keep current quantizations available for essentially every model the local-LLM crowd cares about. They are unpaid infrastructure for the entire ecosystem, and if you run Ollama, LM Studio, or llama.cpp you have almost certainly downloaded one of their files. ## Limitations, ethics, and the AGPL question Abliteration removes the refusal behavior. It does not remove the training distribution. A Heretic variant of a model that learned bad chemistry from filtered web text still does not know good chemistry. It will answer; the answer can be wrong. The same goes for medical advice, legal advice, and any other domain where the base model was already weak. Treat the output the same way you would treat any local model: as a draft from a confident-sounding intern. The legal frame matters too. Heretic itself is AGPL-3.0, which means any service that exposes the tool over a network has to make its source available. The model weights Heretic produces are governed by the upstream license of whatever base model you patched. A Heretic variant of Llama-3 still inherits the Llama license. A Heretic variant of Qwen is still Qwen-licensed. Read those before redistribution. ## Sources and further reading - [Heretic on GitHub](https://github.com/p-e-w/heretic) (canonical repo, README, configs) - [heretic-llm on PyPI](https://pypi.org/project/heretic-llm/) - [heretic-org on HuggingFace](https://huggingface.co/heretic-org) (curated Heretic variants) - [Arditi et al. 2024, "Refusal in Language Models Is Mediated by a Single Direction"](https://arxiv.org/abs/2406.11717) (NeurIPS 2024) - [mlabonne, "Uncensor any LLM with abliteration"](https://huggingface.co/blog/mlabonne/abliteration) - [Lai 2025, "Projected Abliteration"](https://huggingface.co/blog/grimjim/projected-abliteration) - [Lai 2025, "Norm-Preserving Biprojected Abliteration"](https://huggingface.co/blog/grimjim/norm-preserving-biprojected-abliteration) - [Optuna](https://optuna.org/) (the TPE optimizer backing Heretic) Benchmarks community-reported from p-e-w's repo and HuggingFace variant cards. Not independently verified. Compiled 2026-05-19. --- ## News # Ollama Is an Agent Now: v0.32 Turns the Bare Command Into Chat, Code, and Web Search Source: https://singularitybyte.com/news/ollama-v0-32-interactive-agent-2026.html Type ollama with no arguments after updating to v0.32 and you no longer get a help page: you get an interactive agent that chats, writes code, searches the web, and, in Ollama's words, delegates real work. The tool most of us installed to run models locally is now an agent harness. Three releases shipped in ten days, and one default deserves your attention before you press enter. ## What's new in v0.32 - The bare command is an agent (v0.32.0, July 11): running ollama launches "Chat, Code, & Work", a session that can chat with models, edit code, search the web, and run multi-step tasks. In v0.32.1 it picks up your current working directory, so it operates on the project you are standing in. - Skills and longer tool runs (v0.32.2, July 20): the agent gained a skills system and unlimited tool rounds for cloud models, plus a Claude Code launch integration alongside the existing ChatGPT one. On Linux, CUDA compute capability 10.0 support arrived. - Housekeeping that matters (v0.32.1, July 16): an MLX engine cache leak that grew memory across requests on Apple Silicon is fixed, and older agent models (CodeLlama, Qwen2.5-coder, Llama 3.x, Mistral, StarCoder, base DeepSeek-R1) now carry deprecation warnings. ## Why it matters for open-source builders Every terminal agent so far has asked you to install something new. Ollama just gave one to everybody who already has it installed, which is a very large number of machines. But read the model name in the banner: the default agent runs on glm-5.2:cloud, Ollama's hosted deployment of [GLM-5.2](models/glm-5-2.html), and web search requires an ollama signin. That means the out-of-the-box experience of the local-AI tool sends your prompts, and potentially your code, through Ollama's cloud. The launcher has a model picker, so you can point the agent at a local model instead; just know that the default is no longer the thing Ollama's name stands for. We flagged the same drift when [the web search API landed in v0.31](news/ollama-update-gemma-4-speedup-new-scheduler-web-search-2026.html). The pattern is now unmistakable. ## Try it in two minutes ``` # Update, then launch the agent curl -fsSL https://ollama.com/install.sh | sh # macOS/Windows: update the app ollama # Prefer a local model? Pick one in the launcher, # or keep using the classic commands, unchanged: ollama run gemma3 ``` One-click links: [release notes on GitHub](https://github.com/ollama/ollama/releases) · [download Ollama](https://ollama.com/download) · [docs](https://docs.ollama.com) Update, type ollama, and ask it to summarize the repo you are sitting in. Then open the model picker and see how far a local model gets on the same task. That comparison, on your own hardware, is the most honest benchmark you will read this week. All changes are as documented in Ollama's official release notes for v0.32.0 through v0.32.2; we have not yet load-tested the agent mode ourselves. Date checked: 2026-07-22 --- # Ollama Update: Gemma 4 Nearly 90% Faster on Apple Silicon, a New Scheduler, and a Web Search API Source: https://singularitybyte.com/news/ollama-update-gemma-4-speedup-new-scheduler-web-search-2026.html Ollama, the default way most of us run open models locally, shipped a dense two weeks: Gemma 4 now generates tokens nearly 90 percent faster on Apple Silicon, a rebuilt scheduler squeezes real speed out of GPUs you already own, and a new web search API gives local models live internet access. Two of the three make your current hardware faster for free. The third deserves one honest caveat. Here is the rundown. ## What's new, with numbers - Gemma 4 on Apple Silicon (v0.31.1, June 30): nearly 90 percent faster token generation on average, measured on the Aider polyglot coding-agent benchmark with Gemma 4 12B (nvfp4) on an M5 Max, per the [official post](https://ollama.com/blog/faster-gemma-4-mlx-mtp). Gemma 4 ships a small draft model that proposes several tokens ahead; the main model verifies them in one pass, and Ollama now auto-tunes the draft length at runtime. On by default, and it does not change model output. A new MLX matmul kernel (2x to 2.5x faster on M5 Max) helps other models too. - New model scheduler: Ollama now measures exactly how much memory a model needs instead of estimating. Official numbers: gemma3:12b at 128K context on an RTX 4090 jumps from 52.02 to 85.54 tokens per second (64 percent faster) because all 49 layers now fit on the GPU, and image-input prompt processing on a dual-4090 rig goes roughly 10x, from 127.84 to 1,380.24 tokens per second ([details and benchmarks](https://ollama.com/blog/new-model-scheduling)). Fewer out-of-memory crashes and honest memory reporting come along for the ride. - Web search API: a hosted endpoint at ollama.com with a free tier for individuals, callable from curl, Python (ollama.web_search()), or JavaScript, built for feeding live results to local models as an agent tool ([announcement](https://ollama.com/blog/web-search)). Ollama has not published exact free-tier limits, just "generous." Smaller but welcome, in [v0.31.2 (July 6)](https://github.com/ollama/ollama/releases): flash attention enabled on Pascal-era NVIDIA GPUs (compute capability 6.x), and a fix for model loading on paths with non-UTF-8 characters. ## Why it matters for local builders The scheduler is the sleeper here. A 64 percent generation speedup at long context on a 4090, from a software update, is the kind of gain we usually pay for in silicon. If you run agent loops against a local [Qwen 3.6](/models/qwen-3-6.html) or point [OpenCode](/tools/opencode.html) at an Ollama endpoint, you get that for free by upgrading. And multi-token prediction landing in the mainstream local runtime matters beyond Gemma: it is the same speculative-decoding idea the big MoE models like [GLM-5.2](/models/glm-5-2.html) ship with, now working out of the box on a Mac. ## One honest caveat Web search is not local. Your queries go to Ollama's servers, authenticated with an API key, and the feature sits next to a paid cloud tier (Free, $20 Pro, $100 Max). Parts of r/LocalLLaMA and Hacker News read this as cloud creep in a tool people chose precisely because it kept everything on-device; that is community sentiment, not an accusation, but the underlying point stands. Know which of your calls stay local and which do not, and skip the API key entirely if air-gapped is the point of your setup. ## Try it in 10 minutes ``` # Update Ollama (or grab the installer from ollama.com/download) curl -fsSL https://ollama.com/install.sh | sh # Feel the MTP speedup on an Apple Silicon Mac ollama run gemma4:12b # Optional: web search (key from ollama.com/settings/keys; this call leaves your machine) curl https://ollama.com/api/web_search \ --header "Authorization: Bearer $OLLAMA_API_KEY" \ -d '{"query": "LongCat-2.0 benchmarks"}' ``` ## Sources and further reading - [Ollama blog: faster Gemma 4 with multi-token prediction](https://ollama.com/blog/faster-gemma-4-mlx-mtp) - [Ollama blog: new model scheduling](https://ollama.com/blog/new-model-scheduling) - [Ollama blog: web search API](https://ollama.com/blog/web-search) - [Release notes on GitHub](https://github.com/ollama/ollama/releases) - [Ollama cloud pricing tiers](https://ollama.com/pricing) Tested on: not independently tested; no Apple Silicon or RTX 4090 bench available to us this week. All performance figures are Ollama's own published benchmarks (Aider polyglot on M5 Max; RTX 4090 and dual-4090 scheduler runs), linked above. Date checked: 2026-07-09 --- # The June 2026 Open-Weight Wave: 16 Models in One Week Source: https://singularitybyte.com/news/open-weight-wave-june-2026.html The first week of June 2026 was the densest stretch of open-weight releases anyone has tracked. In roughly seven days, frontier-grade open models landed across every modality: language, vision, image, audio, speech, music, video, 3D, and world models. The roundups called it "25+ models in a week." The honest count of models you can actually name and verify is closer to sixteen, but the point stands. Open weights are no longer a trickle, they are a flood, and the license terms are getting better, not worse. Here is what shipped, what to believe, and what it means if you build on open models. ## What actually happened Between roughly June 1 and June 8, a cluster of labs shipped at once. [MiniMax M3](/models/minimax-m3.html) and ByteDance's Bernini-R video model opened the week on June 1. [Gemma 4 12B Unified](/models/google-gemma-4.html) and Ideogram 4 landed June 3. NVIDIA's Nemotron 3 Ultra dropped June 4. The rest filled in around them. No single announcement defined the week; the volume did. Two caveats before the table, because the roundups documenting this week are thin SEO posts and the numbers wander. First, "25+" is marketing math: about sixteen to eighteen models are actually named and checkable. Second, nobody has published aggregate download or star totals for the week, so anyone quoting a wave-level adoption number is inventing it. The individual releases below are real and verifiable; the framing around them is hype until proven otherwise. ## The standout releases Model | Org | Modality | License | Why it matters [Nemotron 3 Ultra](/models/nvidia-nemotron-3-ultra.html) | NVIDIA (US) | LLM / agents | OpenMDW-1.1 | 550B hybrid Mamba-MoE, 1M context, weights plus data and recipes Gemma 4 12B Unified | Google (US) | Multimodal LLM | Apache 2.0 | Text, image, audio, video in one 12B model that runs in ~16GB MiniMax M3 | MiniMax (CN) | Multimodal LLM | open | 428B / 23B active, 1M context, frontier coding and computer use [Ideogram 4](/models/ideogram-4.html) | Ideogram (US) | Image | code Apache 2.0, weights non-commercial | First open weights from a closed image lab, best-in-class text rendering Step-3.7-Flash | StepFun (CN) | Vision-LLM | Apache 2.0 | 198B / 11B active sparse MoE, 256K context Mellum2-12B | JetBrains (EU) | Coding LLM | Apache 2.0 | JetBrains' first open MoE, tuned for code completion PaddleOCR-VL-1.6 | Baidu (CN) | Document VLM | Apache 2.0 | ~1B model that beats document parsers 10x its size dots.tts | RedNote (CN) | Speech / TTS | Apache 2.0 | Fully continuous autoregressive TTS, no codec tokens Magenta RealTime 2 | Google (US) | Music | Apache 2.0 / CC-BY | Real-time music generation under 200ms, DAW plugins Holo-3.1-4B | H Company (EU) | Computer-use VLM | Apache 2.0 | 4B agent for web, desktop, and mobile automation Modality and license for the larger models (Nemotron, Gemma 4, Ideogram, MiniMax) are confirmed from primary sources. The smaller drops (Step-3.7, Mellum2, dots.tts, Holo-3.1) are reported by the week's roundups and should be treated as community-reported until you check the model card. Also shipped that week: Cosmos3-Super (NVIDIA world model), TripoSplat (VAST, image-to-3D, MIT), Higgs Audio v3 (Boson AI TTS). ## The license picture is the good news Count the licenses in that table and the trend is obvious: permissive wins. Apache 2.0 dominates the LLM, audio, document, and agent releases. NVIDIA went further than most, shipping Nemotron 3 Ultra's weights, training data, and recipes under the permissive OpenMDW-1.1. A year ago the open frontier was full of "open but non-commercial" and "research-only" asterisks. This week, most of the asterisks were gone. The holdout is image generation. Ideogram 4 made its code Apache 2.0 but kept its weights non-commercial, with a $300-a-month tier to actually ship product. That is the friction point to watch: text and agent models are racing toward genuine open-source, while image and video weights still come with a business-model string attached. If you are picking a model to build on, read the weights license, not the code license, and not the headline. ## China leads intelligence and volume, the US counters on efficiency Look at who shipped. Chinese labs (MiniMax, StepFun, Baidu, RedNote, ByteDance) accounted for the bulk of the volume, and they lead the open intelligence rankings too. The mid-June coda made it stark: Z.ai's [GLM-5.2](/models/glm-5-2.html), released June 17 under MIT, took the top of Artificial Analysis's open-weights Intelligence Index, above every US open model. NVIDIA's Nemotron 3 Ultra is the strongest US open model, and it sits a notch below the Chinese frontier. The US labs are competing on a different axis: efficiency and infrastructure. Google's Gemma 4 12B runs four modalities in 16GB of memory. NVIDIA's Mamba-MoE hybrid claims 5 to 6 times the throughput of its Chinese rivals on long agentic runs. The structural backdrop, from Hugging Face's Spring 2026 report, is that Chinese models already make up about 41% of all Hugging Face downloads, the largest single share. The open-weight center of gravity has moved, and this week did not reverse it. ## What this means if you build on open models The practical takeaway is that "wait for the open version" is now a viable strategy in almost every modality. Need a coding agent, a document parser, a TTS engine, a music generator, or a computer-use agent? There is a fresh, permissively licensed open option as of this month. The release cadence has compressed from a steady drip to a weekly flood, which means the cost of betting on a closed API just went up, because the open alternative is rarely more than a few weeks behind now. ## Try the most accessible one in 10 minutes Most of this week's headliners need a data-center GPU. The exception worth your ten minutes is Gemma 4 12B Unified, which runs on a single 16GB card or a recent Mac. ``` # The one wave model you can actually run on a laptop-class GPU ollama run gemma-4:12b # Then hand it text, an image, and a short audio clip in one prompt # and see four modalities answered by a model small enough to self-host. ``` That is the real story of the week in a single command: a model that would have been a frontier lab's crown jewel two years ago, now an "ollama run" away, under a license that lets you ship it. Watch the next 90 days for the same thing to happen to video and image weights, the last holdouts still gated behind a paywall. ## Sources and further reading - [Hugging Face: State of Open Source, Spring 2026](https://huggingface.co/blog/huggingface/state-of-os-hf-spring-2026) - [Google: Gemma 4 announcement](https://blog.google/innovation-and-ai/technology/developers-tools/gemma-4/) - [MiniMax M3 announcement](https://www.minimax.io/blog/minimax-m3) - [Artificial Analysis: GLM-5.2 leads the open-weights index](https://artificialanalysis.ai/articles/glm-5-2-is-the-new-leading-open-weights-model-on-the-artificial-analysis-intelligence-index) - [Release-week roundup (Mervin Praison)](https://mer.vin/2026/06/open-weight-ai-release-week-25-models-across-llms-image-audio-video-and-3d-june-2026/) Tested on: not independently benchmarked. This is an ecosystem roundup; model counts and the "25+" figure are roundup-reported and flagged as approximate, individual release facts (params, licenses, dates) are drawn from primary model cards and announcements where available and labeled community-reported otherwise. No aggregate download or star totals are cited because none were published. Date checked: 2026-06-26 --- # When Washington Pulled the Plug: The Anthropic Export Ban and the Case for Open-Weight AI Source: https://singularitybyte.com/news/anthropic-export-ban-case-for-open-weight-ai-2026.html On June 12, 2026, at 5:21pm Eastern, the US government sent Anthropic a letter. Ninety minutes of scrambling later, two of the most capable AI models on the planet went dark for everyone. Not throttled. Not geo-fenced. Off. If you had shipped a product on Claude Fable 5 or the model underneath it, Claude Mythos 5, your provider had just been ordered to pull the plug, and it did. This is the part of the closed-model bargain nobody puts in the architecture diagram. When you build on an API, the off switch lives in someone else's building, and now we know a government can reach in and flip it with no notice and no appeal. The market understood the lesson within hours: capital and traffic rotated toward models you can download and run yourself. Here is the bold version of where this goes. By the end of 2026, "what is our open-weight fallback" stops being a hobbyist question and becomes a standard line in every serious AI architecture review. ## Ninety minutes: what actually happened The directive did not, on paper, ban the models outright. It ordered Anthropic to suspend access to Fable 5 and Mythos 5 "by any foreign national, whether inside or outside the United States," including Anthropic's own non-citizen employees. The catch is operational. A frontier lab cannot reliably sort hundreds of millions of users by nationality in real time, so the only compliant move was the blunt one. In its own [statement](https://www.anthropic.com/news/fable-mythos-access), Anthropic said it had to "abruptly disable Fable 5 and Mythos 5 for all our customers to ensure compliance," and it asked AWS to revoke Bedrock access too. Every other Claude model stayed online. Fable 5 was three days old. It [launched on June 9](https://techcrunch.com/2026/06/09/anthropics-claude-fable-5-is-a-version-of-mythos-the-public-can-access-today/) as the public-facing version of Mythos 5, the more powerful base model whose [cybersecurity abilities](/news/ai-cybersecurity-mythos-glasswing-gpt-cyber-agentic-2026.html) Anthropic had deliberately fenced off. The government's cited concern, relayed to Anthropic, was a method of "jailbreaking" Fable 5 that could reach past those safeguards. The directive came from Commerce Secretary Howard Lutnick, addressed to CEO Dario Amodei. Anthropic pushed back hard, and the pushback is worth quoting because it doubles as the open-source argument. The company said it "disagrees that the finding of a narrow potential jailbreak should be cause for recalling a commercial model deployed to hundreds of millions of people," called the whole thing a "misunderstanding," and noted that comparable capability already ships in other public models. The rest of the industry took the broader point: if a single reported jailbreak can recall a deployed model overnight, no closed frontier model is safe from the same fate. As of late June, the models are still off. Anthropic's Managing Director for International, speaking in Seoul on June 18, said the company was "very confident that in the coming days, the models will become available again." That was several days ago. Fable 5 and Mythos 5 remain dark, with no announced return date. "Coming days" is doing a lot of work. ## How we got here: two years of open weights closing the gap None of this would matter if open models were toys. They are not anymore. The turn started in January 2025, when DeepSeek R1 showed a permissively licensed model trading blows with the closed frontier at a fraction of the cost. Since then the Chinese labs in particular have shipped real weights under real open licenses, on a release cadence the closed shops cannot match. The numbers moved with the models. Through 2025, Chinese-origin models passed US-origin models in share of Hugging Face downloads, and through 2026 the open-versus-closed capability gap on coding and agentic benchmarks narrowed from tens of points to single digits. We covered the shape of that shift in our [State of Open-Source AI](/news/hf-state-of-open-source-spring-2026.html) report, and the regulatory cross-currents in our breakdown of the [EU AI Act open-source exemption](/news/eu-ai-act-open-source-exemption-circus-2026.html). The short version: open weights stopped being the budget option and started being the default for anyone who wants control. ## The market read the memo You can argue about sovereignty in the abstract. The stock market settled it in an afternoon. When trading opened after the shutdown, the two newly listed Chinese open-weight labs jumped, and the reason was not patriotism. It was supply risk repricing in real time. Lab (ticker) | Move after the ban | What they were shipping that week Zhipu / Z.ai (2513.HK) | Up ~33% (intraday as much as ~47%) | GLM-5.2 open weights, MIT license, no usage restrictions MiniMax (0100.HK) | Up ~7.4% | M3, an open-weight sparse-attention coder, shipped June 1 Bank of America initiated coverage on both with "buy" ratings the same week, setting price targets of HK$1,250 for Zhipu and HK$500 for MiniMax, per [CNBC](https://www.cnbc.com/2026/06/15/china-ai-zhipu-minimax-artificial-intelligence-race-washington-trump-anthropic.html). The timing was almost theatrical: on roughly the same day Washington was pulling a closed model, Zhipu was putting GLM-5.2 weights on Hugging Face for anyone to download and keep. The New Stack [documented](https://thenewstack.io/fable-ban-open-weights/) four open models that teams swapped in before Anthropic could even restore access. The plug got pulled, and the network routed around it. That routing shows up in usage, not just share price. On [OpenRouter](https://openrouter.ai/rankings), the most-used models by token volume this month are led by DeepSeek V4 Flash, with Tencent's Hy3 going from zero to near-parity in a single month. Chinese open weights now hold roughly half of the platform's top-tier traffic. The reader takeaway is simple: the alternatives are not theoretical, they are already what a large slice of the market is running. ## Who is ready to catch the runoff If you need to de-risk a closed dependency this quarter, here is the open-weight bench. The column that matters for sovereignty is not the benchmark score, it is the license. MIT and Apache 2.0 mean you can download the weights, run them on your own metal, and nobody can revoke that later. A vendor "community" license means you need to read the fine print before you bet a product on it. Model | Lab | License | Size (MoE) | Why it matters DeepSeek V4 (Pro / Flash) | DeepSeek | MIT | 1.6T total / 49B active; 284B / 13B | 80.6% SWE-bench Verified on Pro-Max, the top open-weight score, with a 1M-token context GLM-5.2 | Z.ai (Zhipu) | MIT | ~744B total / ~40B active | Leads the Artificial Analysis Intelligence Index v4.1 at 51, ahead of every other open model; built for agentic coding MiniMax M3 | MiniMax | Community (check terms) | ~230B total / ~10B active | Sparse-attention coder tuned for cheap, long-horizon agent runs Tencent Hy3 | Tencent | Hunyuan Community (not OSI) | 295B total / 21B active | The fastest climber on OpenRouter token volume; strong agentic workflows MiMo V2.5 | Xiaomi | MIT | Omnimodal | Text, image, video, and audio in one model, tuned for efficient agent tasks Qwen3 series | Alibaba | Apache 2.0 | 235B / 22B flagship + dense 27B | The permissive workhorse, 201 languages, an open option at almost every size Two names on that list, DeepSeek and Z.ai, are MIT, which is about as close to "yours forever" as software gets. If you have written about these before on your own stack, you already have a head start: see our notes on [Z.ai GLM](/models/zai-glm-5-1.html) and [DeepSeek](/models/deepseek-v3-0324.html) for the earlier generations. The jump from API consumer to weight holder is smaller than it looks. ## The sovereignty checklist: de-risking a single-vendor dependency You do not need to rip out your closed provider. You need an exit that you can pull as fast as the government pulled Anthropic's. Five concrete moves, in order of payoff: - Inventory the chokepoints. List every feature that breaks if one closed API disappears tomorrow. That list is your actual risk, and it is usually shorter and scarier than people expect. - Pick one open-weight fallback per chokepoint. Match capability to need: DeepSeek V4 or GLM-5.2 for heavy coding and agents, Qwen3 for general work and many languages, a smaller MiMo or Qwen for on-device. - Abstract the provider. Put a thin interface between your app and the model so the provider is a config value, not a hard dependency wired through your codebase. - Keep a model warm. Run the fallback somewhere you control, even at low scale, so failover is a switch and not a weekend project. Our [local AI stack guide](/tutorials/run-full-local-ai-stack-with-local-ai-packaged.html) and [self-hosted tools roundup](/tutorials/top-free-self-hosted-ai-tools-for-builders-2026.html) cover the plumbing. - Audit licenses and data flow. Confirm the fallback's license actually permits your use, and confirm where prompts and data go. MIT and Apache pass cleanly. Community licenses need a read. Here is the ten-minute version. Most SDKs already speak the OpenAI-compatible protocol, so the entire failover can be one environment variable. ``` # 1) Keep an open-weight model warm locally (MIT-licensed, yours to run) ollama pull qwen3 # or any open model from ollama.com/library # 2) Abstract the provider: same OpenAI-compatible SDK, different base_url. # Swap a closed endpoint for one you control with one variable. export OPENAI_BASE_URL="http://localhost:11434/v1" # local Ollama # export OPENAI_BASE_URL="https://openrouter.ai/api/v1" # hosted fallback export OPENAI_API_KEY="ollama" # any string for local; a real key for OpenRouter # Your application code does not change. The provider does. ``` Do that once and you have moved the off switch back into your building. That is the whole point. ## The catch: this is not a free lunch Open weights solve the kill-switch problem. They do not solve every problem, and pretending otherwise would be its own kind of hype. Three honest caveats. First, self-hosting a frontier model is not cheap. GLM-5.2 wants roughly eight H100 GPUs to run well locally, which is a real bill, not a Raspberry Pi. For most teams the practical answer is a hosted open-weight endpoint plus a small local model kept warm, not a private datacenter. Second, "open weight" is not the same as "no questions asked." Some of the strongest models ship under vendor community licenses rather than OSI-approved ones, and several of the leaders are Chinese-origin, which carries its own data-governance and procurement scrutiny depending on your sector. Run the audit. Third, export policy is a moving target. The same machinery that reached a closed API could, in principle, reach open-weight distribution next. The difference is that weights you have already downloaded cannot be recalled. ## What to watch in the next 90 days - Restoration and precedent. Whether Fable 5 and Mythos 5 come back, and on what terms, sets the template for the next time a model is deemed a national-security concern. - More directives. One letter is an incident. A second one is a policy. Watch Commerce for follow-on action against other labs or capabilities. - Architecture shifts. Expect "open-weight fallback" to show up in enterprise reference architectures and vendor risk reviews as a default requirement. - The next open drops. DeepSeek, Z.ai, Alibaba, Tencent, and Xiaomi are shipping monthly. The bench gets deeper while the closed off switch stays exposed. - Scope creep. Whether the "foreign national" standard stays narrow or widens to other models and other vendors. The takeaway is not "closed models are bad." It is that access you do not control is a dependency you can lose without warning, and hundreds of millions of users just watched it happen in ninety minutes. Spend ten of your own minutes this week standing up a fallback you own. That is the cheapest insurance in AI right now. ## Sources and further reading - [Anthropic: Statement on the US government directive to suspend access to Fable 5 and Mythos 5](https://www.anthropic.com/news/fable-mythos-access) - [CNBC: Anthropic disables access to Fable 5 and Mythos 5 to comply with government directive](https://www.cnbc.com/2026/06/12/anthropic-disables-access-to-fable-5-and-mythos-5-to-comply-with-government-directive.html) - [Al Jazeera: US orders Anthropic to disable AI models for all foreign nationals](https://www.aljazeera.com/news/2026/6/13/us-orders-anthropic-to-disable-ai-models-for-all-foreign-nationals) - [TIME: Anthropic pulls its most powerful AI models after US bars foreign access](https://time.com/article/2026/06/13/anthropic-fable-mythos-ban-US-security/) - [CSIS: The Department of Commerce restricted access to Anthropic's latest models. What comes next?](https://www.csis.org/analysis/department-commerce-restricted-access-anthropics-latest-models-what-comes-next) - [CNBC: Zhipu surges as Wall Street raises bets on China AI after Anthropic curbs](https://www.cnbc.com/2026/06/15/china-ai-zhipu-minimax-artificial-intelligence-race-washington-trump-anthropic.html) - [The New Stack: Fable 5 ban, four open models responded before Anthropic could restore access](https://thenewstack.io/fable-ban-open-weights/) - [Hugging Face: GLM-5.2 open weights (zai-org/GLM-5.2)](https://huggingface.co/zai-org/GLM-5.2) - [Hugging Face: DeepSeek V4 Pro open weights](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro) - [OpenRouter: LLM usage rankings](https://openrouter.ai/rankings) --- # Xiaomi MiMo-V2-Pro: The Hunter Alpha Reveal Source: https://singularitybyte.com/news/xiaomi-mimo-v2-pro-hunter-alpha-stealth-launch.html On March 11, 2026, a model called "Hunter Alpha" appeared anonymously on OpenRouter with no lab name, no model card, and a 1-trillion-parameter budget. Within a week it had processed over a trillion tokens, topped OpenRouter's daily usage charts, climbed to #8 globally on the Artificial Analysis Intelligence Index, and convinced most of the local-LLM community it was an unannounced DeepSeek V4. On March 18, Xiaomi's MiMo team stepped forward and claimed it as [MiMo-V2-Pro](https://help.apiyi.com/en/xiaomi-mimo-v2-pro-omni-hunter-alpha-api-guide-en.html). The technical story is interesting. The bigger story is that Xiaomi just demonstrated it can stealth-launch a frontier model and have it taken seriously without leaning on its brand. ## The seven days nobody knew who built it Hunter Alpha showed up on OpenRouter on a Wednesday with a free tier and no documentation. The first thing the community noticed was that the system prompt identified it as "a Chinese AI model primarily trained in Chinese" with a May 2025 training cutoff, which matched DeepSeek's published cutoff exactly. Reverse-engineered parameter counts pegged it at roughly 1T total, again consistent with leaked V4 specs that had been circulating since February. A [community write-up on Medium](https://medium.com/@him2696/the-mystery-of-hunter-alpha-the-anonymous-1-trillion-parameter-ai-taking-over-openrouter-9e4e94dc0cb8) walked through the evidence and assigned DeepSeek as the most likely author. The benchmarks did not slow the speculation down. SWE-bench Verified scores landed near 78%. ClawEval came in at 61.5, putting it third worldwide behind Claude Opus 4.6 (66.3). PinchBench numbers were class-leading. The token volume on OpenRouter spoke even louder than the leaderboards: developers were paying for the calls and the dominant call patterns were coding agents and tool-use loops. ## The reveal On March 18, Luo Fuli, head of Xiaomi's MiMo division, claimed Hunter Alpha as an early internal build of [MiMo-V2-Pro](https://topaiproduct.com/2026/03/18/a-1-trillion-parameter-ai-model-appeared-on-openrouter-with-no-name-attached-so-who-built-hunter-alpha/). The MiMo team had been a quiet presence on HuggingFace for months: MiMo-7B (MIT-licensed, mobile-class), MiMo-VL-7B for vision-language, MiMo-Audio-7B, and a 309B-total / 15B-active MoE called MiMo-V2-Flash in December 2025. None of those releases got significant Western attention. The 1T MiMo-V2-Pro was a sharp step up the scale ladder, and the team's choice to debut it anonymously meant the community formed its opinion before the brand could prejudice the read. MiMo-V2-Pro spec | Value Architecture | Mixture of Experts, 1.02T total parameters Active parameters per token | 42B (7:1 hybrid ratio) Context length | 1,048,576 tokens (1M) Max output | 32,000 tokens License | Closed weights, API-only on OpenRouter Pricing | $1 / $3 per million in/out tokens Artificial Analysis rank | #8 global, #2 among Chinese LLMs at launch The pricing matters. At $1 / $3 per million tokens, MiMo-V2-Pro is roughly 5 to 7 times cheaper than Claude Opus 4.6 or GPT-5 at comparable benchmark tiers. That is the same cost-down playbook DeepSeek used in late 2024 to reset the API price floor; Xiaomi has just done it again at the frontier. ## Why the DeepSeek attribution stuck Three things made Hunter Alpha read as DeepSeek to the community. First, a "V4 Lite" variant had reportedly appeared on DeepSeek's own site days before, raising expectations. Second, the parameter count, context length, and training cutoff all matched leaked V4 specs. Third, the model's strengths (structured agentic workflows, tool-calling, code) and weaknesses (creative writing, hard math) lined up with DeepSeek's known training emphasis. A handful of analysts flagged GLM-6 from Zhipu AI as an alternative theory, since the same OpenRouter account had previously hosted GLM-5 as "Pony Alpha." Almost nobody guessed Xiaomi. ## Why Xiaomi at all Xiaomi's prior AI work is the missing context. MiLM-6B landed in August 2023 as the company's first public LLM. MiLM2-30B followed as a cloud-scale internal model powering HyperOS features. HyperOS 3.0 in 2026 layered visible AI features into the phone and IoT line. Then, in late 2025, [Luo Fuli joined Xiaomi from DeepSeek](https://www.digitimes.com/news/a20260428PD232/xiaomi-deepseek-competition-hardware.html), where she had been a key contributor to V2. That hire was the visible signal that Xiaomi intended to compete at frontier scale, not just at the on-device tier. MiMo-V2-Pro is the first product of that pivot. A faster follow-up, MiMo-V2.5-Pro, landed on April 22, 2026 and [matched frontier benchmarks at lower token cost](https://www.marktechpost.com/2026/04/22/xiaomi-releases-mimo-v2-5-pro-and-mimo-v2-5-matching-frontier-model-benchmarks-at-significantly-lower-token-cost/), suggesting the team is iterating quickly and is not running on one-shot luck. ## The stealth-launch pattern Anonymous OpenRouter releases are not new. Anthropic's "claude-3-opus-pre" leaked briefly in early 2024. Several smaller Chinese labs have used the same channel to dogfood unreleased models against real developer traffic. What makes the Xiaomi case different is the scale and the credibility: a 1T-parameter frontier candidate, deployed to production-grade load on an open marketplace, then claimed only after a week of independent benchmarks had already validated it. For practitioners, the lesson is not "Xiaomi is now in the top tier" (one frontier model does not equal a moat). The lesson is that the route from "we have a frontier model in the lab" to "the community treats it as real" can be one week and one OpenRouter account, with the lab name attached at the end of the validation cycle instead of the start. That changes how to read leaderboards. The next time an anonymous OpenRouter listing posts top-tier benchmark numbers, the default assumption should no longer be "leaked Anthropic or OpenAI checkpoint." It should be "another lab is testing the water." ## What to do with it today MiMo-V2-Pro is callable on [OpenRouter](https://openrouter.ai/) right now at the listed pricing. The model is API-only, no open weights, so this is not one for the local-LLM stack. If you build agent or coding workflows that already use Claude Opus or GPT-5 through OpenRouter, MiMo-V2-Pro is a straight drop-in to test against. Run it on your actual evals before forming an opinion. The benchmark profile says it will be excellent at structured tool-use loops and merely OK at long-form creative work; verify whether that matches your traffic. And keep half an eye on the OpenRouter homepage. The next anonymous 1T-parameter listing might be from Tencent, Bytedance, Baichuan, or someone you have not heard of yet. The stealth-launch playbook now has a working precedent. ## Sources and further reading - [VentureBeat: Xiaomi stuns with MiMo-V2-Pro](https://venturebeat.com/technology/xiaomi-stuns-with-new-mimo-v2-pro-llm-nearing-gpt-5-2-opus-4-6-performance/) - [TopAIProduct: 1T-parameter mystery model](https://topaiproduct.com/2026/03/18/a-1-trillion-parameter-ai-model-appeared-on-openrouter-with-no-name-attached-so-who-built-hunter-alpha/) - [Community analysis of the anonymous week](https://medium.com/@him2696/the-mystery-of-hunter-alpha-the-anonymous-1-trillion-parameter-ai-taking-over-openrouter-9e4e94dc0cb8) - [Apiyi MiMo-V2-Pro API guide and full benchmarks](https://help.apiyi.com/en/xiaomi-mimo-v2-pro-omni-hunter-alpha-api-guide-en.html) - [MarkTechPost on the V2.5 follow-up](https://www.marktechpost.com/2026/04/22/xiaomi-releases-mimo-v2-5-pro-and-mimo-v2-5-matching-frontier-model-benchmarks-at-significantly-lower-token-cost/) - [DIGITIMES on the Luo Fuli hire](https://www.digitimes.com/news/a20260428PD232/xiaomi-deepseek-competition-hardware.html) Timeline and benchmark numbers compiled from the linked sources. Not independently verified. Compiled 2026-05-19. --- # Tenstorrent TT-QuietBox 2 Source: https://singularitybyte.com/news/tenstorrent-tt-quietbox-2-risc-v-ai-workstation.html For 15 years, "buy a serious AI box" has meant "buy NVIDIA, run CUDA, hope the driver stack stays kind to you." Tenstorrent's [TT-QuietBox 2](https://www.tenstorrent.com/waitlist/tt-quietbox), announced at GDC 2026 and shipping Q2 2026 starting at $9,999, is the first credible desk-side workstation to break that pattern. Four Blackhole ASICs, 128 GB of GDDR6, RISC-V cores top to bottom, a fully Apache-2.0 software stack from MLIR compiler down to the kernel-level SDK, liquid cooling, and a standard wall outlet. It runs Llama 3.1 70B at 476 tokens per second on-device, no cloud round-trips and no CUDA black boxes. ## What's actually in the box The TT-QuietBox 2 packages four Blackhole accelerator cards in a single chassis. Per the [Tenstorrent Blackhole spec sheet](https://tenstorrent.com/en/hardware/blackhole), each chip has 120 Tensix cores, 16 "big" RISC-V 64-bit cores, and 752 "baby" RISC-V cores for control and movement. That works out to 480 Tensix cores and roughly 2,654 TFLOPS of BlockFP8 compute across the box. Memory is 128 GB of GDDR6 (32 GB per chip, around 512 GB/s per chip), with 256 GB of DDR5 system RAM behind the host CPU. The interconnect is Ethernet-based at 1 TB/s total, ten 400 Gbps QSFP-DD links per chip, which is the same plumbing Tenstorrent uses to scale up to their Galaxy systems. Power draw stays under 1.5 kW so the whole thing plugs into a standard 120 V wall outlet. The chassis is liquid-cooled by design rather than as an aftermarket bolt-on, and the announcement is explicit that Tenstorrent wanted the box "quiet enough to sit on a desk," not parked in a closet. Pricing starts at $9,999, with the [waitlist live now](https://www.tenstorrent.com/waitlist/tt-quietbox) and shipments planned for Q2 2026. ## The actual selling point: an open stack you can fork Hardware is the easy half. The interesting part is that every layer of the software stack is Apache 2.0 on GitHub. Tenstorrent didn't ship an open API on top of a proprietary blob; they shipped the compiler, the runtime, the kernel library, and the low-level kernels themselves. If a Blackhole op is slow or wrong on your model, you can read the code that emitted it, patch it, and rebuild. Layer | Repo | What it does TT-Forge | [tenstorrent/tt-forge](https://github.com/tenstorrent/tt-forge) | MLIR-based compiler. Takes PyTorch, ONNX, TensorFlow, JAX, PaddlePaddle and emits Blackhole code. TT-Metalium | [tenstorrent/tt-metal](https://github.com/tenstorrent/tt-metal) | Kernel-level SDK. Direct access to Tensix cores, the equivalent of CUDA C++ but readable. TT-NN | same repo | Higher-level operator library on top of TT-Metalium, PyTorch-friendly. TT-LLK | [tenstorrent/tt-llk](https://github.com/tenstorrent/tt-llk) | The low-level kernels themselves. Custom Tensix ISA. For comparison, CUDA is closed. Triton is open, but the NVIDIA driver and PTX backend it targets are not. With the TT-QuietBox 2 stack, a developer can trace a slow Llama matmul down to the actual Tensix instructions and either patch TT-LLK or hand-write a custom kernel through TT-Metalium. That degree of visibility doesn't exist on any NVIDIA box at any price. ## Performance vs. the H100, honestly The numbers Tenstorrent has published focus on single-box throughput. Llama 3.1 70B clocks 476.5 tokens/sec on the four-chip TT-QuietBox 2. [An 8 x H100 node serves Llama 3.3 70B at roughly 2,600 tokens/sec](https://dlewis.io/evaluating-llama-33-70b-inference-h100-a100/), so per-chip the H100 is still significantly faster on dense inference. A single Blackhole box also runs gpt-oss-120B fully on-device, and a Boltz-2 biomolecular workload that takes 45 minutes on a CPU finishes in 49 seconds on one Blackhole chip. What Blackhole genuinely wins on is openness, programmability, and price-per-token if you can stand the lower raw throughput. A four-chip H100 workstation does not exist at $9,999. The closest NVIDIA equivalent is a DGX Station at multiples of the price with a proprietary stack you cannot meaningfully fix when it misbehaves. ## Where it sits in the open-hardware spectrum The open-hardware AI compute spectrum now has a recognizable shape. At the budget end, Seeed Studio's [reComputer RK series](https://wiki.seeedstudio.com/object_detection_with_yolov11_on_recomputer_rk/) built on Rockchip RK3576 and RK3588 SoCs delivers around 6 TOPS of NPU compute in a small-board form factor for a few hundred dollars, aimed at edge computer vision and small-model inference. At the upper end, the TT-QuietBox 2 puts frontier-class inference on a desk for ten thousand. The middle (Jetson Orin, AMD AI accelerators, Apple Silicon Macs) is crowded but mostly proprietary in the parts that matter. Tenstorrent is currently the only player covering the high-end-and-open quadrant. ## Who actually wants this The TT-QuietBox 2 is not the right box if you need maximum raw inference per dollar today; an H100 cloud instance still wins on that metric. It is the right box if any of the following matter to you: - You need the model to never leave your building. Healthcare, defense, legal, financial back-office, anything where data residency is a hard requirement. - You want to fix the stack when it breaks. Researchers building custom transformer variants, kernel authors, anyone who has hit a CUDA bug and waited months for a driver fix. - You are betting on RISC-V. The Blackhole chip is RISC-V end-to-end and the software stack reflects that. Investing in skills here transfers. - You are tired of cloud bills. $9,999 amortizes against ongoing inference costs faster than most teams expect once usage is steady. ## What to watch in the next 90 days Three things will decide whether the TT-QuietBox 2 actually displaces NVIDIA on desks. First, ship dates: Q2 2026 ends June 30. If the box ships on time and at the announced price, that itself is news. Second, kernel maturity: independent benchmarks on Mixtral, DeepSeek-R1, Qwen 3, and Gemma 3 will tell us how Blackhole handles MoE and attention patterns that weren't in the launch demos. Third, the developer story: Tenstorrent has shipped [tt-metal](https://github.com/tenstorrent/tt-metal) on GitHub for years, but pull-request velocity from outside contributors is the real proxy for whether the open stack pulls a community. Get on the [waitlist](https://www.tenstorrent.com/waitlist/tt-quietbox) if you want to play. Even if you do not buy one, the existence of a full-stack open RISC-V workstation at $9,999 reshapes the conversation about what local AI hardware can look like. ## Sources and further reading - [Tenstorrent Blackhole official spec page](https://tenstorrent.com/en/hardware/blackhole) - [TT-QuietBox 2 announcement, March 11 2026](https://www.accessnewswire.com/newsroom/en/computers-technology-and-internet/tenstorrent-unveils-tt-quietboxtm-2-the-first-risc-v-ai-workstation-with-a-fully-open-source-stack-to-deliver-teraflop-class-inference-1145920) - [TT-Forge compiler on GitHub](https://github.com/tenstorrent/tt-forge) - [TT-Metalium SDK on GitHub](https://github.com/tenstorrent/tt-metal) - [TT-LLK low-level kernels on GitHub](https://github.com/tenstorrent/tt-llk) - [VideoCardz coverage with full spec table](https://videocardz.com/newz/tenstorrent-tt-quietbox-2-liquid-cooled-risc-v-workstation-starts-at-9999-with-four-blackhole-asics) - [Spheron comparison: Tenstorrent vs. NVIDIA](https://www.spheron.network/blog/tenstorrent-vs-nvidia-open-source-ai-hardware/) Benchmarks community-reported from Tenstorrent's announcement and the linked references. Not independently verified. Compiled 2026-05-19. --- # AI Cybersecurity Just Got Autonomous: Mythos, Glasswing, GPT-5.4-Cyber Source: https://singularitybyte.com/news/ai-cybersecurity-mythos-glasswing-gpt-cyber-agentic-2026.html Ten days, three shipped models, one $190 million check, and a stock chart that looks like a trapdoor. April 2026 is when AI in cybersecurity stopped being a conference keynote slide and started being the product cycle. If you ship software, defend a network, or run anything that touches the public internet, the threat model this month is not the one you planned for last quarter. Here is what actually happened, in order, with links. No "game changer" language, no breathless futurism, just what shipped and what it means for the people whose job is to ship code or keep it from being broken. ## Project Glasswing and Claude Mythos Preview On April 7, 2026, Anthropic published the [Mythos Preview announcement](https://red.anthropic.com/2026/mythos-preview/) and launched Project Glasswing. Mythos Preview is a new general-purpose Claude model. Glasswing is the private program Anthropic built around it because the company decided not to ship the model publicly. The model found zero-day vulnerabilities in every major operating system and every major web browser. Thousands of them, including high and critical severity bugs, some of them 27 years old. Anthropic reports that in comparable Firefox testing, Opus 4.6 had a near-zero autonomous exploit development rate. Mythos Preview hit 181 successful exploits in the same harness. Running the model at roughly $10,000 of API credits against FFmpeg and $20,000 against OpenBSD produced hundreds to thousands of runs, respectively, each capable of chaining multiple vulnerabilities, reverse-engineering closed-source binaries, and writing working exploits without a human in the loop. The Project Glasswing partner list reads like a Fortune 500 who-has-root list: Amazon Web Services, Apple, Broadcom, Cisco, CrowdStrike, Google, JPMorgan Chase, the Linux Foundation, Microsoft, NVIDIA, and Palo Alto Networks. Anthropic said around 40 additional critical-infrastructure organizations are also inside the program. The company is committing up to $100 million in Mythos Preview usage credits to Glasswing partners and another $4 million in direct donations to open-source security orgs. The position is explicit: Mythos Preview will not be generally available. Anthropic believes it is too dangerous to ship. That judgment alone is the story. A frontier lab shipped a model good enough at offensive security that it decided to give it only to the defenders of the most critical software on earth. US Treasury Secretary and Federal Reserve Chair reportedly [warned bank CEOs](https://www.sullcrom.com/insights/memo/2026/April/Treasury-Secretary-Federal-Reserve-Chair-Warn-Bank-CEOs-About-Cybersecurity-Risks-Posed-Anthropics-New-AI-Model) about the cyber risks the same week. ## Claude Opus 4.7 and the Cyber Verification Program On April 16, Anthropic shipped Claude Opus 4.7 for general use, and with it a new safeguard layer. Opus 4.7 ships with [real-time cyber safeguards](https://support.claude.com/en/articles/14604842-real-time-cyber-safeguards-on-claude) that automatically detect and block two kinds of requests: prohibited uses like ransomware code or mass data exfiltration, and high-risk dual-use activities like vulnerability testing and exploit development that also have legitimate defensive applications. If your day job is pentesting, red-teaming, or vulnerability research, your day job just got blocked by the model you were paying for. Anthropic's answer is the Cyber Verification Program. It is a free, application-based allowlist. You submit the [Cyber Use Case Form](https://claude.com/form/cyber-use-case), document your intent, and Anthropic reviews within two business days. Approved users get the safeguards relaxed for dual-use work. Unapproved users keep hitting the block. Access routes differ by platform. Direct Claude users (Claude.ai, Claude Code, the API) apply directly. Microsoft Foundry users select Azure and provide tenant and subscription IDs. Amazon Bedrock and Google Vertex AI do not currently support the program. Zero Data Retention customers have to go through their Anthropic sales rep. Per [The Register](https://www.theregister.com/2026/04/16/anthropic_claude_id_verification_persona/), ID verification for the program runs through Persona. The short version: Anthropic just built a registration gate around cyber capabilities. Legitimate security pros can still get through, but the vibes-coded weekend-hacker path is closed. ## OpenAI ships GPT-5.4-Cyber Two days before Opus 4.7, on April 14, OpenAI launched [GPT-5.4-Cyber](https://openai.com/index/scaling-trusted-access-for-cyber-defense/), a fine-tuned variant of GPT-5.4 with deliberately lowered refusal boundaries for defensive cybersecurity work. The model adds binary reverse engineering for malware analysis and vulnerability discovery, plus broader capability to reason about offensive techniques that the stock GPT-5.4 refuses outright. Like Anthropic's model, GPT-5.4-Cyber is not generally available. OpenAI scaled up its Trusted Access for Cyber (TAC) program to thousands of verified individual defenders and hundreds of teams, and gates the cyber model behind the highest TAC tiers. Two competing frontier labs, shipping two cyber-specialized models, inside one week, both gated behind verification programs. That is not a coincidence; that is an industry admitting that the unmodified models can cause too much trouble to leave open. ## The defender problem On March 10, Kevin Mandia, the founder of Mandiant (sold to Google for $5.4 billion in 2022), came out of retirement with [Armadin](https://techcrunch.com/2026/03/10/mandiants-founder-just-raised-190m-for-his-autonomous-ai-agent-security-startup/), an AI-native cybersecurity startup. Armadin closed $189.9 million in combined seed and Series A, led by Accel, with GV, Kleiner Perkins, Menlo Ventures, 8VC, Ballistic Ventures, and In-Q-Tel. The co-founders are Travis Lanham (ex-Google Cloud Security), Evan Pena (ex-Mandiant), and David Slater (ex-Google SecOps). The pitch is one sentence: autonomous cybersecurity agents that learn and respond to threats without a human in the middle. Mandia's framing of the threat model: "When you have AI on offense, what you are going to get is a technology that can think, can learn, can adapt." Attackers, he warned, will complete attacks in minutes that used to take days. Anthropic's own threat reporting backs this up. The [GTG-1002 campaign](https://www.anthropic.com/news/disrupting-AI-espionage) used autonomous agents to automate 90 percent of the intrusion lifecycle, collapsing the window between initial access and impact from hours to, in some measured cases, 22 seconds. A human SOC analyst reading a Slack alert cannot compete with that. An agent running a playbook can, which is why the defensive side of the market just raised a record seed round to build those agents. ## The CrowdStrike hedge Not every incumbent is buying it. CrowdStrike CEO George Kurtz argues that while AI agents can reason about code, they cannot manage the operational chaos of a live enterprise environment. Markets reportedly took that as a defensive statement more than a confident one; CrowdStrike's stock took a hit in February as investors [reassessed the moat](https://www.financialcontent.com/article/marketminute-2026-2-24-the-agentic-ai-crisis-crowdstrike-plummets-as-markets-reassess-the-cybersecurity-moat) around traditional endpoint-detection-and-response platforms. Kurtz is not wrong that agents can stumble in messy real-world environments. He is also in the position of a CEO whose product line was designed for an earlier threat model. The honest read is that some of the incumbents will pivot successfully and some will not, and the ones that move first on agentic defense have the better odds. ## What this means for open-source Nothing about the frontier closed-access play changes the open-source picture yet. SmolLM3, Qwen3.5, Gemma 4, and the rest of the fully open stack still run with standard safeguards. Community-maintained tooling like [Anthropic-Cybersecurity-Skills](https://github.com/mukul975/Anthropic-Cybersecurity-Skills), a 754-skill library mapped to MITRE ATT&CK, NIST CSF 2.0, D3FEND, and NIST AI RMF, is pushing structured cyber workflows into any agent framework that understands the agentskills.io standard. But the gap is widening. Mythos Preview and GPT-5.4-Cyber are qualitatively better at offense than anything you can download. If the pattern holds, future frontier cyber models will stay locked behind verification programs, which means open-source defense will have to ride on open-source models that lag the frontier by six to twelve months. Defenders running fully open stacks will need to invest harder in tooling, telemetry, and playbooks, because the raw model horsepower gap is not going to close itself. ## What to do this week Four concrete moves if you work in or around software security: - If you do legitimate cyber work with Claude, apply to the [Cyber Verification Program](https://claude.com/form/cyber-use-case) now. Two business days is the stated turnaround. Opus 4.7 will keep blocking your prompts until you are through. - If you ship critical software and are not already in Project Glasswing, watch for Anthropic's outreach. The company says there are about 40 organizations beyond the named partners already inside; the list is not closed. - If you run a SOC, start pressure-testing your response playbooks against a hypothetical 22-second agent-driven intrusion. Most existing playbooks assume human-in-the-loop containment at minutes-to-hours timescales. That assumption is obsolete for the new threat profile. - If you build with open-source models, assume you are six months behind the frontier on offensive capability and plan defense accordingly. Invest in behavioral detection, hardening, and response automation that does not depend on a frontier model behind a verification gate. ## The verdict The thing that makes this week different is not any one model. It is the combined signal. Two frontier labs shipped cyber-specialized models in seven days. Both gated them behind verification programs. The founder of Mandiant came back with nine figures to build autonomous defense agents. The federal government started calling bank CEOs. Anthropic found thousands of zero-days in software you use every day and decided it was safer not to ship the model that found them. The cybersecurity industry spent the last decade arguing about whether AI would matter for defense. That argument is over. Whether your stack is ready for the next six months of this is a different question, and one worth answering before something tests it for you. ## Sources - [Anthropic: Claude Mythos Preview announcement (April 7, 2026)](https://red.anthropic.com/2026/mythos-preview/) - [Anthropic: Project Glasswing](https://www.anthropic.com/project/glasswing) - [Anthropic: Zero-day findings from Mythos Preview](https://red.anthropic.com/2026/zero-days/) - [Anthropic: Real-time cyber safeguards on Claude](https://support.claude.com/en/articles/14604842-real-time-cyber-safeguards-on-claude) - [Help Net Security: Claude Opus 4.7 released with cyber safeguards](https://www.helpnetsecurity.com/2026/04/16/claude-opus-4-7-released/) - [OpenAI: Trusted access for the next era of cyber defense](https://openai.com/index/scaling-trusted-access-for-cyber-defense/) - [9to5Mac: OpenAI unveils GPT-5.4-Cyber](https://9to5mac.com/2026/04/14/openai-unveils-gpt-5-4-cyber-an-ai-model-for-defensive-cybersecurity/) - [TechCrunch: Kevin Mandia raises $190M for Armadin](https://techcrunch.com/2026/03/10/mandiants-founder-just-raised-190m-for-his-autonomous-ai-agent-security-startup/) - [IBM: Mythos and Glasswing raise the stakes](https://www.ibm.com/think/news/anthropic-claude-ai-mythos-project-glasswing-raises-stakes-cybersecurity) - [PostQuantum: Mythos Preview and the end of a twenty-year equilibrium](https://postquantum.com/security-pqc/anthropic-mythos-preview-ai-offensive-security/) --- # EU AI Act Open-Source Exemption: The Circus of Conditions Continues Source: https://singularitybyte.com/news/eu-ai-act-open-source-exemption-circus-2026.html Brussels has spoken. Again. On April 10, 2026, the Commission pushed out another round of clarifications around the AI Act's open-source exemption, and the message to anyone who actually builds models is: yes, you can release open weights, but only after you have read 40 pages of caveats, passed three tests, and filed a copyright policy with the AI Office. If this sounds like a lighter touch, that is because the European definition of "lighter" is carrying a filing cabinet up a spiral staircase instead of two filing cabinets. The headline version plays well. "Open-source models are exempt from certain GPAI obligations." The fine print version, once you read the actual guidance and the Code of Practice, makes clear the exemption applies to roughly three paragraphs of the Act, never applies to anything anyone would call a frontier model, and still leaves open-source providers on the hook for the two obligations that cost the most time: copyright compliance and training-data disclosure. Congratulations, Europe. You have invented the paperwork-optional paperwork track. ## What the exemption actually exempts Three obligations. That is the full list. If your model qualifies as "free and open-source" under the Commission's definition, you get a waiver from: - Article 53(1a): maintaining detailed technical documentation for authorities on request. - Article 53(1b): providing that same technical documentation to downstream providers who integrate your model. - Article 54: appointing an authorized representative physically present in the EU (applies to non-EU providers). Everything else in Article 53 still applies. You still write a copyright compliance policy. You still publish a sufficiently detailed summary of your training data using the AI Office's template. Miss either one and the fines scale to 35 million euro or 7 percent of worldwide turnover, whichever is larger. The exemption gets you out of paperwork for bureaucrats. It does not get you out of paperwork for lawyers. ## The three conditions you must pass To qualify as "truly free and open-source" per the guidance, a model must clear all three of these hurdles: - License. Apache 2.0, MIT, and OpenMDW are in. Any license with research-only clauses, non-commercial restrictions, or usage carve-outs that are not "specific, proportionate, and safety-oriented" is out. Llama's acceptable-use policy? Out. Most of the corporate "community" licenses? Out. - Transparency. Weights, architecture information, and usage information must be publicly available. No sneaking the good weights behind a gated form. - No monetization. You cannot charge for access, bundle the model with a paid service, run ads against it, or harvest user data as a condition of access. The Commission did at least carve out that hosting weights on an open repository like Hugging Face does not count as monetization. Thank you, Brussels, truly. The monetization clause is the quiet killer. It rules out almost every commercial "open-weights" release from the major labs. Your startup built a paid API around an open model you trained yourself? Not exempt. Your company releases a model and sells premium fine-tunes? Not exempt. The exemption is designed for researchers and hobbyists, then dressed up in press-release language that makes it sound like it applies to industry. ## The systemic-risk trap Here is where the circus gets loud. The exemption does not apply to general-purpose AI models with "systemic risk." Systemic risk is defined by a compute threshold: any model trained with more than 10 to the 25th floating-point operations of cumulative compute. For reference, that covers GPT-4o, Grok 4, Mistral 2 Large, and effectively any frontier-scale model shipping today. Cross the threshold, and you owe the Commission full Article 55 compliance regardless of your license: model evaluation, systemic risk assessment, adversarial testing, incident reporting, cybersecurity measures, and energy efficiency reporting. Open weights or not. The Act does leave a pressure valve. Under Recital 112, a developer can submit evidence arguing that "because of its specific characteristics, a general-purpose AI model exceptionally does not present systemic risks." Good luck with that. The subtext is that the Commission gets to decide what counts as exceptional, and the Commission has not traditionally treated "we trained it ourselves and shipped the weights for free" as an exceptional characteristic. ## The bar is lower than it looks The regular GPAI threshold sits at 10 to the 23rd FLOPs, roughly the compute it takes to train a one-billion-parameter model on a serious dataset. That puts SmolLM3-3B, Mistral 7B, Llama-3.2-3B, and most of what a mid-sized AI startup releases this year squarely inside the scope of the Act. Not exempt from GPAI status, not exempt from training-data summaries, not exempt from copyright policies. Only exempt from the three paperwork items listed above, and only if all three open-source conditions are cleanly met. Anyone who assumed "open-source" meant "exempt from the AI Act" in any general sense has spent the last year reading marketing summaries instead of the actual legislation. ## The finetuning loophole that might actually help One provision in the guidance is genuinely useful. If you finetune an existing GPAI model, you only become a "provider" under the Act if your finetuning compute exceeds one third of the original model's training compute. For a 10 to the 23rd FLOP base model, that is about 3.3 times 10 to the 22nd FLOPs of additional compute before you inherit full provider obligations. Below that, you are a downstream user, and your documentation duty is limited to the changes you made plus a pointer to the original model's training data summary. Translation for developers: LoRAs, QLoRAs, small SFT runs, and task-specific finetunes of open models almost never cross the one-third threshold. You inherit the base model's status, not a fresh provider obligation. That is the cleanest part of the guidance and the one piece that clearly helps the open-source ecosystem. ## The enforcement deadline is real The AI Office's full enforcement powers switch on August 2, 2026. At that point the Office can request information, order model recalls, mandate mitigations, and issue fines. Models placed on the EU market before August 2, 2025 have until August 2, 2027 to bring their training-data summary into compliance. Models placed on the market after August 2, 2025 owe the summary immediately. Most open-source releases from the second half of 2025 and early 2026 are already inside the stricter deadline and are at various stages of scrambling to publish the summaries. The big labs have legal teams that already filed the template. The small ones are reading it for the first time this month. ## The verdict: a better circus than before Credit where it is due. The April 2026 clarification is better than the original Act. The finetuning ratio is sensible. The Hugging-Face-hosting-is-not-monetization note is sensible. Excluding non-commercial research licenses from the definition of open-source is at least a defensible position. The guidance reads like the Commission finally talked to actual developers before shipping. But the overall structure is unchanged. Any model that matters commercially will be above 10 to the 25th FLOPs, which means the exemption does not apply, which means the rules kick in in full. Any model below that threshold still owes copyright policies and training summaries regardless of license. The "exemption" waives the smallest three items on the list. The rest stays, and the fines stay, and the August 2026 enforcement date stays. Europe's open-source developers get a slightly smaller filing cabinet to carry. They still carry it up the same staircase, and the ringmaster in Brussels is still selling tickets to the show. Meanwhile, across the Atlantic, a 19-year-old with a Colab notebook fine-tunes Qwen3 and ships it on Hugging Face the same afternoon. No template. No summary. No representative. No staircase. That is the competitive picture this guidance does not change. ## What to actually do If you are shipping open weights from Europe, three concrete moves: - Pick a real open-source license. Apache 2.0 or MIT. Avoid "community," "research-only," and anything with downstream-user restrictions. OpenMDW is explicitly on the Commission's accepted list if you want something AI-specific. - Publish the training-data summary using the [Commission's template](https://digital-strategy.ec.europa.eu/en/news/commission-presents-template-general-purpose-ai-model-providers-summarise-data-used-train-their). It is not optional for any model placed on the EU market after August 2, 2025. - Write a one-page copyright compliance policy, respect robots.txt and machine-readable opt-outs, and publish a complaint mechanism for rightsholders. This is the bare minimum under Article 53(1c), exemption or not. Then get back to work. The enforcement deadline is four months out, the filing cabinet is already on your back, and the circus in Brussels will still be running the same act this time next year. ## Sources - [European Commission: Guidelines for providers of general-purpose AI models](https://digital-strategy.ec.europa.eu/en/policies/guidelines-gpai-providers) - [European Commission FAQ: Obligations for GPAI providers](https://digital-strategy.ec.europa.eu/en/faqs/guidelines-obligations-general-purpose-ai-providers) - [Hugging Face: What Open-Source Developers Need to Know about the EU AI Act's Rules for GPAI Models](https://huggingface.co/blog/yjernite/eu-act-os-guideai) - [Hugging Face: Open Source Developers Guide to the EU AI Act](https://huggingface.co/blog/eu-ai-act-for-oss-developers) - [Artificial Intelligence Act portal: GPAI Guidelines overview](https://artificialintelligenceact.eu/gpai-guidelines-overview/) - [Linux Foundation Europe: AI Act Explainer for Open Source Developers](https://linuxfoundation.eu/newsroom/ai-act-explainer) ---