Run Kimi K3 Locally: Unsloth GGUFs Land at 594 GB

When the Kimi K3 weights landed on July 27, our take was blunt: 1,561 GB, multi-node or nothing, and any day-one community quant deserves scrutiny. Three days later, the scrutiny has a serious target. Unsloth published Dynamic GGUFs of Kimi K3 starting at 594 GB, plus a llama.cpp branch that runs the model with its vision tower intact. The largest open-weight model ever shipped is now a single-box problem. A very large box.

Run Kimi K3 locally: what shipped

• unsloth/Kimi-K3-GGUF holds six Dynamic quants of the 2.8T-parameter MoE, from a 594 GB 1-bit (UD-IQ1_S) to a near-lossless 1,560 GB Q8, shipped as split GGUF files that llama.cpp loads directly.
• Vision works. You build Unsloth's llama.cpp branch kimi-k3-fullsize-vision and pass the bundled mmproj-BF16.gguf projector file alongside the model.
• Moonshot's recommended settings carry over: temperature 1.0 with top_p 0.95, or top_p 1.0 for agent work. Thinking is always on, with low, high and max effort levels.

The hardware bill

Unsloth's rule of thumb: combined RAM plus VRAM roughly equal to the quant size. Less still works through disk offloading, just much slower. Here is the honest table.

So "locally" means an NVIDIA DGX Station, a fat EPYC box, or, per Unsloth, "a Mac Studio connected to a 128GB RAM device". That last one adds up: a 512 GB M3 Ultra plus one 128 GB machine totals 640 GB, which clears the 1-bit line. On B200-class hardware that fits the whole quant, Unsloth reports about 20 tokens per second generation and over 120 tokens per second throughput. Your 4090 is not in this story, and we respect you too much to pretend otherwise.

Run it today

Build the branch, then pull the 1-bit quant and the vision projector. The 1-bit alone is 14 split files, so budget the disk and the weekend.

git clone https://github.com/unslothai/llama.cpp
cd llama.cpp
git fetch origin pull/48/head:kimi-k3-fullsize-vision
git checkout kimi-k3-fullsize-vision
cd ..
cmake llama.cpp -B llama.cpp/build -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first

pip install -U "huggingface_hub[cli]"
hf download unsloth/Kimi-K3-GGUF --local-dir unsloth/Kimi-K3-GGUF \
    --include "*mmproj-BF16*" --include "*UD-IQ1_S*"

Then point llama-cli at the first shard and the projector:

./llama.cpp/build/bin/llama-cli \
    --model unsloth/Kimi-K3-GGUF/UD-IQ1_S/Kimi-K3-UD-IQ1_S-00001-of-00014.gguf \
    --mmproj unsloth/Kimi-K3-GGUF/mmproj-BF16.gguf \
    --temp 1.0 --top-p 0.95

One gotcha for agent builders, straight from the model card: K3 always thinks and returns reasoning_content, and multi-turn calls must feed the complete assistant message back into history, reasoning and tool calls included, not just the final text.

Why this matters, and the caveat

Three days ago, self-hosting K3 meant roughly 19 H100s just to load the raw MXFP4 weights. A 610 GB RAM+VRAM target moves that to one serious workstation, and K3's linear-attention design keeps the KV cache tame if you push the 1M context. The caveat: a Dynamic 1-bit is not uniformly 1-bit. Unsloth, the two-person outfit we profiled, mixes bit-widths per tensor, and nobody has published quality numbers for these exact files yet. The Hugging Face counter already shows 12,178 downloads, which is a lot of terabytes pulled on faith.

Your under-10-minutes move: check whether any machine you own clears the line before you burn 594 GB of disk finding out.

# Your RAM + VRAM in GB, against the 610 GB the 1-bit wants.
echo "$(( $(free -g | awk '/Mem:/ {print $2}') + $(nvidia-smi --query-gpu=memory.total --format=csv,noheader,nounits 2>/dev/null | awk '{s+=$1} END {print int(s/1024)}') )) GB available"

If the number is a few hundred GB short, the hosted API remains the sane path, and Kimi K2.7 Code still covers the single-node crowd.

Sources and further reading

• Unsloth: Kimi K3 local deployment guide (sizes, settings, commands)
• unsloth/Kimi-K3-GGUF on Hugging Face (all six quants + mmproj)
• unslothai/llama.cpp, branch kimi-k3-fullsize-vision
• moonshotai/Kimi-K3 (original weights) and the Kimi K3 License, which applies to these quants too
• SingularityByte: what actually shipped in the K3 weights drop
• SingularityByte: Kimi K3 model breakdown

Tested on: not independently tested. The smallest Kimi K3 quant needs about 610 GB of combined RAM and VRAM, which is beyond our bench. Quant sizes, hardware guidance and throughput figures are Unsloth-reported; sampling settings and the multi-turn rule are from the model card; file sizes and download counts were read from the Hugging Face repo.
Date checked: 2026-07-30