Pi runs git status and git diff fine. You still lose the commit graph, the staged file list, inline diffs, and hunk buttons when you review a refactor. I kept Alt-Tabbing to Fork or squinting at git diff output. I wanted Sourcetree in the same cmux grid as the agent.

So I built gitgui, one Rust binary that paints a GUI into your terminal pane with the kitty graphics protocol. Pixels, not a TUI. Graph, sidebar, diff viewer, commit box, editor, conflict resolver. cmux, Ghostty, kitty, WezTerm. Over SSH it sends zlib plus base64 frames.

Updated 2026-09-07 for v0.6.0: the UI moved from egui to iced, the panes drag, resize and hide, merge conflicts get a three-way resolver, the layout is remembered per repository, and the same binary opens a desktop window in terminals without kitty graphics. Plus a folder switcher, zoom keys and SHA-256 repositories.

This is my daily layout: the agent on the left, gitgui v0.6.0 on the right, same cmux workspace, same checkout.

The agent edits the repo. gitgui shows branch lanes, unstaged files, and hunk buttons. One workspace. One checkout. cmux + agent + gitgui = 😍

On a repo with branches, tags, a stash and a merge stopped on a conflict it looks like this:

Why stay inside the terminal?

cmux is a pane grid. You already split shell, Pi, and sometimes a browser. Fork or GitKraken pull you out: different font, different shortcuts, another app fighting for focus.

lazygit and gitui help for quick commits. They fall short when you read a three-file refactor with colored hunks, click Stage hunk on one block, and scan merge lanes on a graph. I wanted mouse clicks and a graph without leaving the grid.

terminal-browser and terminal-code proved the pattern: render pixels in the terminal, read pixel mouse events back. gitgui uses iced instead of Chromium, driven without a window.

Three steps:

1. Build the iced UI, draw it with the tiny-skia software renderer straight into an RGBA framebuffer
2. Send the framebuffer as a kitty graphics image each frame
3. Map kitty keyboard and SGR mouse events into iced events

Your terminal never prints UI text. It shows a picture. Locally frames go through POSIX shared memory. On my Mac a 1600×1000 frame at 2x costs about 6 ms.

What you get in v0.6.0

• Panes you arrange: repository, files, commits, changes and diff on an iced pane grid. Drag a title bar to move a pane, drag the gaps to resize, maximize with the arrows or 1 to 5, hide a pane with its x and bring it back from the footer. The commit list’s columns resize from the header. The layout, hidden panes, collapsed sections and zoom are saved per repository in .git/gitgui.json and come back on the next start.
• Commit graph with branch lanes, ref pills, filter by summary, author or hash.
• Staging by file, hunk or line: click, Shift+click or drag lines in the diff, then Stage or Discard just those. Search in the diff, adjust context, ignore whitespace, wrap.
• Three-way conflict resolver: ours, result, theirs side by side with per-conflict take-left, take-right, keep-both, drop buttons, accept-all, edit the result, apply and mark resolved. Conflicted files also get a banner with whole-file ours / theirs.

• History rewriting from the commit menu: reword, squash, fixup, drop, move up / down, edit, autosquash. gitgui runs git rebase with itself as the sequence editor, so no editor pops up in your pane.
• Branches, remotes, tags, stashes in a collapsible sidebar: checkout, create, rename, delete, merge, rebase onto, fast-forward, upstream, open pull request, delete on remote, fetch one remote, push a tag, apply / pop / drop / branch from stash.
• File tree of the whole working tree, folders listed on demand, changed files colored. Click a file and it opens in the built-in editor.
• Editor on iced’s text editor with syntax colors for the common languages, undo, Ctrl+S. Shift+E opens the file in your own editor in a new cmux split (--editor, git config gitgui.editor, $EDITOR; GUI editors like code open detached). Shift+O opens cmux’s file preview.

• Merge and rebase state in the footer with continue / abort / skip.
• Change folder from the footer or Ctrl+O: a folder dialog with git repositories marked, so one gitgui pane follows you across projects. Zoom with Ctrl+= / Ctrl+-.
• SHA-256 repositories (git init --object-format=sha256) open, diff and commit like any other; libgit2 is built with its SHA-256 support.
• Branch switcher, publish to GitHub through gh, initialize a non-git folder, auto refresh every 2 s when the repo changes.
• Agent socket: gitgui ls, gitgui action '{"cmd":"status"}', so Pi or Cursor query status, select commits, stage paths, fetch, push, or save a PNG screenshot.

Keys are single letters and Ctrl combinations terminals do not steal. cmux keeps Cmd+*. ? lists them all. Quit with q or Ctrl+C, or click Quit in the footer.

And outside cmux: a desktop window

Not every terminal speaks kitty graphics. Terminal.app, iTerm2, the VS Code terminal, tmux. Since v0.6.0 the same binary notices that and opens a native window instead, still the same iced UI and the same tiny-skia renderer, through winit and softbuffer, no GPU. gitgui --window forces it, and scripts/bundle-macos.sh wraps it as gitgui.app with the logo as its icon, so it also launches from Finder. Started outside a repository it shows the logo and a Change folder button.

The terminal split, cmux’s file preview and agent screenshots stay terminal-only; keys, panes, the state file and the agent socket work the same.

Install

You need macOS or Linux. In a kitty-graphics terminal gitgui draws into the pane; anywhere else it opens a desktop window.

Quick start:

curl -fsSL https://raw.githubusercontent.com/antonellof/gitgui/main/scripts/install.sh | bash

That one-liner works on public repos. Private repo? Clone with gh and run the script locally:

gh repo clone antonellof/gitgui
cd gitgui
bash scripts/install.sh

Pin a release:

GITGUI_VERSION=0.6.0 bash scripts/install.sh

The script pulls a release binary when GitHub has one. Otherwise it builds from source with cargo (Rust 1.95+). Success looks like:

installed: ~/.local/bin/gitgui

run in a kitty-graphics terminal (cmux, Ghostty, kitty):
  gitgui                  open repo in current directory
  gitgui /path/to/repo    open a specific repo
  gitgui --split right .  open in a new terminal split

quit with q or Ctrl+C

Check terminal support:

gitgui --probe

cmux + Pi + gitgui

Same workspace as the trio post. One extra split.

Pane A, Pi:

cd ~/Projects/my-app
pi

Pane B, gitgui, from the agent pane:

gitgui --split right .

cmux runs new-split right and send with the gitgui command. Or split by hand and type gitgui in the new pane.

Give your agent the skill file:

mkdir -p ~/.cursor/skills/gitgui
ln -sf ~/path/to/gitgui/skill/SKILL.md ~/.cursor/skills/gitgui/SKILL.md

Drive the GUI from the shell:

gitgui ls
gitgui action '{"cmd":"status"}'
gitgui action '{"cmd":"stage","paths":["src/main.rs"]}'
gitgui action '{"cmd":"fetch"}'
gitgui action '{"cmd":"commit_and_push","message":"fix layout"}'
gitgui action '{"cmd":"screenshot","path":"/tmp/frame.png"}'

Run action from the pane where gitgui lives and it finds the instance via the controlling tty. Writes queue on a background git thread. Poll status until busy hits zero.

A typical session: Pi edits and runs tests on the left. gitgui auto-refreshes on the right. You stage hunks, click Commit or Commit & Push, or use Ctrl+Enter / Ctrl+Shift+Enter. Switch branches from the status bar when Pi opens a feature branch. Or Pi stages through gitgui action and you type the commit message yourself.

Inside the binary

One Rust process. Three threads.

stdin reader blocks on terminal bytes and feeds a parser for kitty keys, SGR mouse, paste, resize.

main loop runs iced without a window: it builds the UI, feeds it the terminal events, applies the messages, and lets tiny-skia draw straight into the framebuffer that goes out as kitty graphics. The commit log, the diff and the conflict resolver are custom widgets that draw only their visible rows.

git worker uses libgit2 for reads and index writes. fetch, pull, and push shell out to git so your credential helper and SSH agent stay untouched. A background poll every 2 s refreshes the snapshot when files change on disk, so Pi edits show up without a manual refresh.

The UI reads an immutable RepoSnapshot. The worker swaps in a new snapshot after each command. Rendering never calls git.

iced 0.14 (iced_core, iced_runtime, iced_widget, iced_renderer on tiny-skia; the iced umbrella crate with winit only for the window mode), git2 with libgit2’s SHA-256 support, libc for termios and shm, serde for the agent API. No Electron. No GPU backend. No tokio. Spec and protocol bytes live in docs/SPEC.md and docs/PROTOCOLS.md.

One refactor, start to finish

Pi touches three files. You want the graph to update, one hunk staged and one left alone, and a commit message you wrote yourself. All inside cmux.

Before gitgui: git diff in the shell for small diffs, Alt-Tab to Fork for a graph, lazygit when you want speed over pixels.

Now: gitgui --split right . once per session. Tab between Pi and gitgui. Edits from Pi land in the unstaged list within a couple of seconds. Stage one hunk, leave another, or select three lines and stage just those. Type a message, hit Commit & Push, or fetch first from the footer. Need a feature branch? Click the branch name, pick from the list, stash if you have WIP. Merge came back with conflicts? Click Resolve, take a side per conflict, Apply, Continue. Want to fix a typo in a file Pi touched? Click it in the tree, edit, Ctrl+S.

cmux still rings when Pi waits on you. Mouse clicks hit the right widgets. SSH sessions use the direct transport and the same UI.

What it still skips

cmux and Ghostty are the main targets. kitty works. tmux and Zellij do not pass kitty graphics through; there, and in any terminal without them, gitgui opens its desktop window instead of drawing into the pane.

Ghostty on macOS has no stable split-from-child API. gitgui --split prints a keybind hint and runs in the current pane when needed.

The editor has no line-number gutter yet, and the resolver’s result column is not editable in place: press Edit result to hand-edit the merged file.

If you already run cmux and Pi, gitgui is the git pane I wished existed. Install it, split right, keep the graph beside the harness.

gitgui: github.com/antonellof/gitgui. cmux: cmux.dev. Pi: pi.dev. Related: The Local Agent Trio: cmux + Pi + Unsloth Studio.

Ferrox is a pure-Rust inference engine for GGUF models. Dense and mixture-of-experts, on CPU, Apple Metal, or CUDA. No bindings to llama.cpp, no wrapper around ggml. You get a CLI with llama.cpp-style flags and an OpenAI-compatible HTTP server you point your existing tools at.

I wrote it for one reason. I want mixture-of-experts models running on machines too small to hold them in VRAM, with expert-level residency rather than layer offload: watch which experts fire, keep those resident, evict the rest. Doing that well means designing the router, the KV cache, and the memory manager together. Every speed claim gets pinned against llama.cpp on the same machine, same file, same backend.

The first post covers the design. This one covers what changed since.

Kept current with the latest release. Since this went up: Metal passed llama.cpp on decode as well as prefill, speculative decoding stays lossless at any temperature, streams survive a dropped connection, and one binary now does every job. Those are at the end.

MoE prefill on Metal got 2.4x faster

OLMoE-1B-7B went from 587 tok/s to 1402 tok/s on 512-token prefill. Against llama.cpp on the same machine and the same file, the gap closed from 2.62x behind to 1.11x.

Where it stands at v0.12.0, re-measured in one session with a warmup rep: OLMoE prefill is 1.09x and its decode is at parity.

Ferrox runs a whole transformer layer inside one Metal command encoder. Dense layers already joined this fused stack. MoE layers did not, so each one paid host-side projections, a separate command buffer for attention, a round trip to the CPU to pick experts, and another command buffer to run them. Roughly 112 command buffers per prefill.

MoE layers now live in the same stack. Two kernels were missing and are new: a router GEMM for F32 weights, which every MoE GGUF ships, and a batched top-k softmax. The old one handled a single token at a time, which suits decode and wastes a 512-token prompt.

Decode improved too. Profiling found the expert router taking 3.6 ms of 8.5 ms per token, more than the experts it selects. Two of my own kernels ran serial work on a chip with thousands of lanes. Rewriting them dropped GPU time per token by 25 percent.

The same work extended the flash-attention kernel to 256-dimension heads. The biggest single winner was Qwen3-0.6B: 1936 tok/s to 3400 tok/s.

The measurement I was most wrong about

The decode work left a note in my plan saying the remaining 2.4 ms per token went on barriers, roughly eight per layer. So I replaced the blanket barriers with a hazard tracker that synchronises only the memory ranges a kernel touches, ported from ggml.

Worth about 1 percent. The reason is the useful part.

Barrier counts came out at 0.99 per tracked operation before and after. A single-token decode layer is a strict dependency chain, so a range tracker has nothing to overlap and only narrows each barrier’s scope. The arithmetic agrees: Llama-3.2-1B at Q4_K_M is 0.8 GB, and 6.1 ms per token works out to 130 GB/s against the M2 Pro’s 200 GB/s. Decode is already limited by weight bandwidth, not by synchronisation.

My own earlier note was wrong. I wrote that into the plan so the next attempt captures the attention kernel instead of touching synchronisation again.

Verifying the change took 54 runs of the CPU-versus-Metal check across dense, MoE, sliding-window and QK-norm models at four prompt lengths. A missing barrier is a race. One green run proves nothing.

Try it

The only build flag you need is your GPU. Everything else is in the binary, and that binary is 19 MB with no interpreter or wheels under it.

cargo install ferrox-cli --features metal

# Same argument shape as `hf download`, and no Python involved.
ferrox download bartowski/Llama-3.2-3B-Instruct-GGUF \
  Llama-3.2-3B-Instruct-Q4_K_M.gguf --local-dir models

# Chat model. Ferrox applies the checkpoint's own chat template.
ferrox -m models/Llama-3.2-3B-Instruct-Q4_K_M.gguf \
  -p "Explain quantization in two sentences" -n 128 -dev metal -ngl all

# Raw completion, no chat wrapping.
ferrox -m models/Llama-3.2-3B-Instruct-Q4_K_M.gguf \
  -p "The capital of France is" -n 32 --temp 0 --no-cnv

# OpenAI-compatible server on 127.0.0.1:8383.
ferrox serve -m models/Llama-3.2-3B-Instruct-Q4_K_M.gguf -dev metal -ngl all

# Benchmark against llama-bench, side by side.
ferrox bench -m models/Llama-3.2-3B-Instruct-Q4_K_M.gguf -p 512 -n 128 -r 3 --compare

Point any OpenAI client at http://127.0.0.1:8383/v1 and it works. Anthropic Messages and Responses live on the same server, so codex and Anthropic-shaped clients work without a shim.

The engine publishes as ferrox-inference, since the name ferrox belongs to an unrelated crate. Twelve crates, all live, all sharing one version. Take the facade for the whole stack, or ferrox-gguf alone if you only want to read GGUF files.

[dependencies]
ferrox-inference = "0.13"

Where Ferrox beats llama.cpp

Decode on Metal at v0.12.0, same M2 Pro, same GGUF, both engines measured in one session with a warmup rep and the host load recorded at both ends. Ratio is llama divided by ferrox, so under 1.0 means Ferrox is faster.

Eight of the twelve Metal decode rows are ahead of llama.cpp and the other four sit within 3 percent of it, so no Metal row is red any more. Qwen2.5-0.5B runs 62 percent faster. Small models gain most, because the per-token fixed cost dominates and that is where the fused Metal stack pays off. Prefill closed too: every dense pp512 row lands between 0.98x and 1.10x.

CPU is the other story, and Ferrox loses across it: all 16 comparable rows land between 1.41x and 5.06x. The cause is measured and the fix is written, waiting on a clean benchmark window. Full table either way in benchmarks/RESULTS.md, generated from checked-in raw timings rather than typed by hand.

New model support

gpt-oss runs on CPU, with attention sinks, alternating sliding-window attention, a biased router, and the swiglu clamp. I checked every layer against reference logits from llama.cpp rather than against my reading of the spec.

The low-bit quantization tiers landed: IQ2_XS, IQ2_S, IQ3_S, and IQ1_M. IQ3_S matters most, since IQ3_M mixes are largely made of it. All four decode bit-exact against llama.cpp’s own dequantization. F16 checkpoints load now as well, which they never did before.

Phi-4-mini runs correctly on CPU and Metal. Partial rotary, LongRoPE factor sets and rope attention scaling all work, and the Metal RoPE kernels took rot_dim and mscale uniforms so the GPU path stopped being refused. A second bug turned up on the way: rope_freqs was sized by the head width instead of by n_rot.

Rotary embeddings now use the right layout for 24 more architectures, including exaone, nemotron, starcoder2, minicpm3, openelm, plamo, and dots1. Getting this wrong produces plausible text with subtly wrong attention, so I pinned the whole table against llama.cpp’s reference with a test. Measured against golden logits, error dropped from 1.1e-2 to 1e-6.

Chat templates come from the checkpoint

Ferrox used to sniff a model into one of six families and render an approximation. Anything outside those six fell back to plain text, and tool-calling formats were unreachable.

Every GGUF already carries its chat template as a Jinja string, so Ferrox evaluates that instead. Four real templates lifted from actual checkpoints are pinned in tests with exact expected output.

Update: this section originally said half of it shipped, with the evaluator written and no call site using it. Both front ends run it now. A checkpoint whose family nobody hand-wrote a renderer for is framed the way it was trained, and chat_template_kwargs and reasoning_effort pass through to it. The effort is quantized onto the gears that checkpoint actually grades, probed from its own template at load rather than read from a table keyed by model name.

One consequence I had not thought about: plenty of templates emit the BOS token themselves, and Unsloth strips it from the ones it bakes into a GGUF. Whether a loader double-adds BOS becomes a property of the checkpoint rather than of the code. I measured it across 26 local checkpoints instead of reasoning about it. Zero doubled today, but only because every prepend site had a guard. Remove the guard and 6 of the 26 double.

Ferrox Studio

A web UI ships alongside the server, rebuilt this week on React, Tailwind, and assistant-ui: streaming chat with markdown and code blocks, a model manager, a live request log, and a Connect screen generating curl and Python snippets from your running server.

It runs as its own app rather than inside the server binary. npm run dev in ui/ proxies to the API, so the server serves the API and nothing else.

Every screen goes through the public HTTP API, so anything the UI does, your own client does too. The stats under each answer come from the server’s own timings rather than a browser stopwatch, so time-to-first-token and decode rate are the real numbers. That is the grey line under the answer above: TTFT 889 ms · prefill 17 tok · 19.2 tok/s · decode 99 tok · 47.2 tok/s, from a 3B on Metal.

Model output never becomes markup. I seeded an answer containing a script tag, an image with an onerror handler, and a javascript: link, then checked the DOM: zero script or image elements, no globals set, the link href emptied, all three shown as text.

Serving under real load

The parts deciding whether a server behaves with several clients at once:

• Chunked prefill as a resumable state machine, so a long prompt stops stalling replies already in flight.
• Admission on an integer KV block budget instead of a byte watermark, with strict FIFO so a large request cannot starve behind small ones.
• Cancellation drained at a step boundary, so a cancelled row leaves through the same exit every row uses rather than being ripped out mid-batch.
• Stop sequences in two layers, matching single tokens by id before detokenization.
• A radix tree over reference-counted KV pages, so two conversations sharing a system prompt share its pages instead of copying them per request.
• Queue depth capped with a 503 and a Retry-After, so retry storms cannot grow memory without bound.

Models load, unload, and swap at runtime. A request already running finishes against the weights it started with.

Two tools worth stealing

ferrox parity compares first-token logit distributions against llama.cpp’s own library. Greedy text comparison does not work for this, because ordinary floating-point drift flips one near-tied token and everything after it differs. Comparing distributions separates a wrong graph from a rounding difference.

ferrox bench refuses to time a run on a busy machine. Known-good rows read 25 to 45 percent low under load, wider than most gaps worth chasing. The load average now goes into every receipt, and a run above the bar stops rather than publishing noise.

The loader also refuses checkpoints it cannot compute. It records every tensor name a loader reads and fails the load on leftovers, so a model needing a graph feature Ferrox lacks gets a clear refusal instead of confident wrong tokens.

What is next

Ranked, with measurements attached, in docs/plans:

• CPU decode, the widest remaining gap. The cause is measured: Rayon forks and joins per operation while llama.cpp runs a persistent thread pool. A replacement pool is written and waiting on a clean benchmark window.
• CPU prefill. The blocked attention kernel is 53 percent of non-idle samples on SmolLM2 against roughly 8 percent of the model’s FLOPs, because pass one does a dot product per query per KV position with no K reuse across the query block.
• dFlash speculative decoding, where a drafter emits a whole block of tokens in one pass instead of one at a time.
• AMD Strix Halo, where a large unified memory pool suits keeping MoE experts resident.

Every speed number gets measured against llama.cpp on the same host, same file, same backend. If there is no receipt, the table says so. I do not invent numbers.

Shipped since this was written

Six things landed after this post went up.

Speculative decoding is lossless at any temperature. It used to accept a draft token only when the target’s argmax matched, which is correct at temperature 0 and quietly wrong above it. Verification now uses the speculative-sampling rejection rule. Drafters plug in through a Drafter trait, and acceptance length plus the per-position accept rate are reported, so a drafter that decays toward the end of a block shows up instead of being averaged away.

Streams survive a dropped connection. GET /v1/stream/{request_id} replays with Last-Event-ID, and there is a polling sibling for proxies that buffer SSE. Two features collided here and only one of them wins: an orphan deadline cancels generation when nobody has read for 30 seconds, and a resumable stream exists precisely so a dropped socket does not cancel. The deadline still detects and logs, and only cancels streams that are not resumable.

The server prices its KV budget before loading. Weights plus context times per-token KV against the device budget, with FERROX_CB_MAX_CONTEXT and FERROX_CB_KV_BLOCKS derived when unset. A request that cannot fit gets a typed 400 naming which ceiling binds, rather than an OOM later.

One binary does every job. cargo install ferrox-cli --features metal now gives you completions, ferrox serve, ferrox download, ferrox bench and ferrox verify. There is no feature flag to discover beyond your GPU. I had kept serve behind a flag to spare a completion-only user 98 crates and a C crypto library, and that rule was costing more than it saved: it had already put a false claim in the README, and it was hiding a headline capability where nobody would find it. ferrox-server stays published for anyone who wants the server alone.

Fetching a model needs no Python. ferrox download takes hf download’s exact argument shape, so a command copied off a model card runs unchanged. It resolves IPv4 first, because Hugging Face publishes AAAA records that black-hole on some networks, reads HF_TOKEN for gated repos, and asks for a byte range so an interrupted download resumes. Bytes land on a .partial name and are renamed only after the last one arrives, so the loader never opens a truncated file as a whole GGUF.

A model too large for the machine is refused before it loads, naming the checkpoint’s size, the memory available, and what expert streaming would cost. I shipped the opposite first: streaming turned itself on when the weights did not fit. Then OLMoE answered “Paris.” resident and “amongst amongst, and of” streamed, deterministically, at temperature 0. Auto-enabling was withdrawn the same day. Streaming stays an explicit opt-in until it returns the right tokens, because “your model does not fit” is a bad outcome and “your model answers nonsense” is a much worse one.

AI full disclosure

This software is developed with strong assistance from Cursor, Grok 4.5, GPT 5.6, and Claude Fable 5, with humans leading the ideas, testing, and debugging. We say this openly because it shaped how the project was built. If you are not happy with AI-developed code, this software is not for you. The acknowledgement below is equally important: this would not exist without llama.cpp and GGML, largely written by hand.

Acknowledgements to llama.cpp and GGML

Ferrox does not link against GGML, but exists thanks to the path opened by the llama.cpp project and the kernels, quantization formats, GGUF ecosystem, and hard-won engineering knowledge developed there. Half of this work is “I read how llama.cpp did it and ported the idea”, which is deliberate and is the project’s default method. We are thankful and indebted to llama.cpp and its contributors. Some source-level pieces are retained or adapted here under the MIT license, notably IQ quantization codebook tables, and many other pieces were written independently against that public design. We keep the GGML authors’ copyright notice in docs/THIRD_PARTY_NOTICES.md.

Closing

Ferrox is Apache-2.0. Try it, and open an issue with the GGUF name, the backend, and your ferrox bench output if something runs slow or wrong.

• github.com/antonellof/ferrox
• latest release
• ferrox-inference on crates.io
• benchmarks/RESULTS.md
• the first post, on why the engine exists

Written August 2026, at v0.4. Ferrox has moved a long way since, so the sections below carry update notes where the numbers changed.

I built Ferrox over the last couple of weeks. It is a pure-Rust inference engine for GGUF models: dense and MoE, on CPU, Apple Metal, or CUDA. No bindings to llama.cpp. No wrapper around ggml.

Where Ferrox is now

Many releases later, the current one is published on crates.io. The short version:

• Metal caught llama.cpp and went past it. Every dense pp512 row sits between 0.98× and 1.10×, and 8 of the 12 tg128 rows are faster. CPU is still behind on all 16 comparable rows, 1.41× to 5.06×.
• One binary does everything: completions, ferrox serve for the OpenAI and Anthropic compatible API, ferrox download, ferrox bench, ferrox verify. The only build flag you need is your GPU.
• ferrox download fetches a model with hf download’s exact syntax, so a command copied off a model card runs unchanged. No Python.
• Twelve real checkpoints answer correctly on Metal, two of them MoE.
• A web UI, Ferrox Studio, running as its own app against the same public API any other client uses.
• Published as ferrox-inference, with every layer available on its own if you want the GGUF reader and nothing else.

The middle of that stretch has its own post: Ferrox on Metal.

Why another engine

llama.cpp is excellent. I am not trying to catch up with every feature it has.

What I want is MoE on machines that cannot fit the whole model in VRAM. Not layer offload. Expert-level residency: watch which experts fire during decode, keep those resident, evict the rest. To do that well the router, the KV cache and the memory manager have to be designed together. That is about the only good reason to write a runtime from scratch. If you only need “run a Llama GGUF on my Mac”, use llama.cpp.

What it does today

One binary with flags close to llama.cpp (-m, -p, -n, -ngl, --ctk, …). ferrox serve starts the HTTP server from the same executable, and ferrox-server still ships on its own for anyone who prefers two.

Update: this originally listed two binaries. serve is a default feature now, so one cargo install gets completions, the server, the downloader, bench and verify. The server answers Anthropic Messages and Responses beside the OpenAI routes.

That binary is 19 MB, or 14 MB stripped, with no runtime under it. No interpreter, no virtualenv, no wheels to resolve, nothing to activate before you can run a model. It sits in the same ballpark as llama.cpp’s own core, which is 14 MB of executables and shared libraries on this Mac. Against the Python serving stacks it is a different category: PyTorch alone measures 402 MB here, and vLLM sits on top of that.

Weights stay quantized on mmap. Dequant happens inside the matmul, not as a separate pass. Same idea as llama.cpp, which is why an 8B model fits on a laptop without thirty gigabytes of RAM.

Verified families, meaning a real checkpoint answered correctly and stopped, not merely “it loads”: Llama 3.x, TinyLlama, SmolLM2, Qwen2.5/Qwen3, Gemma-2/3/4, Phi-4-mini, Mistral-7B, Yi, DeepSeek-R1 distills, OLMoE and Qwen1.5-MoE. gpt-oss runs on CPU. MLA / DeepSeek-style stacks are partial.

That is about twelve architectures with evidence behind them, out of 150 rows in the catalog. The rest refuse to load, naming the tensor or the hyperparameter they need, or are marked unproven. I chose that. llama.cpp hand-writes 140 per-architecture graphs, and that hand-written work is the whole reason its coverage is wider than mine. Until Ferrox has done the same work for a family, it stops with an error rather than running a graph that is merely close enough to compile and handing back fluent text computed the wrong way. A refusal you can read is coverage. Confident wrong tokens are not.

Build, download, run

# --features metal on Apple silicon, cuda on Linux + NVIDIA, nothing on CPU.
cargo install ferrox-cli --features metal

# Same argument shape as `hf download`, without the Python.
ferrox download bartowski/Llama-3.2-1B-Instruct-GGUF \
  Llama-3.2-1B-Instruct-Q4_K_M.gguf --local-dir models

# Ferrox wraps the prompt in the checkpoint's own chat template.
ferrox -m models/Llama-3.2-1B-Instruct-Q4_K_M.gguf \
  -p "What is 2+2?" -n 64 --temp 0 -dev metal -ngl all

# Serve on 127.0.0.1:8383 and point any OpenAI client at /v1.
ferrox serve -m models/Llama-3.2-1B-Instruct-Q4_K_M.gguf -dev metal -ngl all

The README lists known-good GGUFs to start from. Prefer Q4_K_M day to day, Q8_0 for tiny tests. On Metal, --ctk q8_0 shrinks the KV cache if you need longer context.

A model too large for the machine is refused before it loads, naming the checkpoint’s size, the memory you have, and what expert streaming would cost. Streaming is opt-in: it is slower than keeping experts resident, and it still returns wrong tokens on real MoE checkpoints. Running out of memory is a bad outcome. Answering nonsense is a worse one.

Numbers, with receipts

I do not trust speed claims without a method. ferrox bench runs the same pp512 prefill and tg128 decode workloads as llama-bench, on the same Mac (M2 Pro), the same GGUF and the same backend. Every run writes a JSON file of raw timings, those files are checked into the repo, and benchmarks/RESULTS.md is generated from them rather than typed by hand. Gap = llama / ferrox. Under 1.0 means Ferrox is faster.

Engine decode on Metal, M2 Pro, measured at v0.12.0 in a single session:

Metal is at or past llama.cpp on both tests. Every dense pp512 row lands between 0.98× and 1.10×, 8 of the 12 tg128 rows are ahead, and no Metal row is red. OLMoE reaches parity on prefill and decode as well, so MoE is no longer the column I did not want to publish.

CPU is the whole remaining gap, and Ferrox loses across it: all 16 comparable rows land between 1.41× and 5.06×. The cause is measured. Rayon forks and joins per operation while llama.cpp runs a persistent thread pool, and Ferrox is ahead at a single thread on Mistral-7B, so it scales badly rather than computing slowly. A replacement pool is written and waiting on a clean benchmark window.

Re-run it yourself:

ferrox bench --suite --fit-host --skip-missing

It stops rather than time a run on a machine too busy, too hot, or too short on memory for the number to mean anything. If there is no receipt, the table says so. I do not invent numbers.

What I learned so far

A few things that stuck, after many evenings staring at tok/s:

• Decode and prefill are different problems. Getting tg128 to parity did not make pp512 follow. Prefill needed its own answer: simdgroup-MMA flash attention, batched GEMM, and a fused command encoder that runs a whole layer without returning to the host.
• MoE Metal needs its own path. A dense kernel wearing a MoE costume loses. Expert placement, mul_mm_id and fused encode groups took OLMoE to parity on both tests.
• Fair comparison is harder than it looks. Forcing both engines to the same thread count made llama.cpp look worse on this Mac, because it already prefers performance cores. The suite no longer forces -t.
• CUDA was quietly broken. --features cuda on the CLI did not enable CUDA in ferrox-core. Prefill never hit the GPU.

Update: CUDA compiles and runs, and nobody has benchmarked it. No pinned host, no receipts, so treat a Windows or Linux install as CPU-only in practice. Saying that plainly in the docs was easier than pretending otherwise.

None of this is glamorous. It is the work.

What is next

From the roadmap:

1. CPU, the widest remaining gap, with the persistent thread pool ready to measure.
2. Run bigger models on the same hardware (Qwen3 35B-A3B-class on a box that today only likes 8B). Expert streaming has to return the right tokens first.
3. Hybrid CPU/GPU expert placement, and tighter KV.
4. CUDA tuning. It builds and runs, with no serious pass yet.
5. Grammar and JSON-schema constrained decoding, Docker images.

If you want to point an IDE or agent at the local server, there is a short cookbook: docs/AGENTS_COOKBOOK.md.

AI full disclosure

This software is developed with strong assistance from Cursor, Grok 4.5, GPT 5.6, and Claude Fable 5, with humans leading the ideas, testing, and debugging. We say this openly because it shaped how the project was built. If you are not happy with AI-developed code, this software is not for you.

The acknowledgement below is equally important: this would not exist without llama.cpp and GGML, largely written by hand.

Acknowledgements to llama.cpp and GGML

Ferrox does not link against GGML, but it exists thanks to the path opened by the llama.cpp project and the kernels, quantization formats, GGUF ecosystem, and hard-won engineering knowledge developed there. We are thankful and indebted to llama.cpp and its contributors. Their implementation, kernels, tests, and design choices were an essential reference while building this pure-Rust GGUF / MoE inference path.

Some source-level pieces are retained or adapted here under the MIT license, notably IQ quantization codebook tables, and many other pieces (GGUF layouts, quant/dot semantics, CLI and server conventions) were written independently against that public design. For this reason, and because we are genuinely grateful, we keep the GGML authors’ copyright notice in docs/THIRD_PARTY_NOTICES.md.

Closing

Ferrox is Apache-2.0. Stars are nice. PRs and failed pins are more useful. If you try it and something is slow or wrong, open an issue with the GGUF name, backend, and the ferrox bench output.

Links

• github.com/antonellof/ferrox
• ferrox-inference on crates.io
• benchmarks/RESULTS.md
• docs/FEATURES.md
• docs/MODELS.md
• docs/THIRD_PARTY_NOTICES.md
• HN discussion

Those choices change with every machine. A good setup for a 24 GB MacBook is not a good setup for a 256 GB Mac Studio. An RTX 4090 should not be configured like an H100, and a DGX Spark has its own Blackwell-specific path.

Rondine is an open-source, hardware-aware control plane for this problem. It detects the machine, recommends models that fit, builds an optimized serving plan, downloads the selected weights, and starts an OpenAI-compatible local server.

It does not implement another inference engine. Rondine drives the mature engines that already exist:

• MLX-LM or llama.cpp on Apple Silicon
• llama.cpp or vLLM on discrete NVIDIA GPUs
• vLLM or llama.cpp on DGX Spark / GB10
• Native engine launchers for homogeneous multi-node experiments

The goal is simple: go from “which model and which flags?” to a repeatable local endpoint without hiding the decisions being made.

The workflow

A complete setup is a short sequence of commands:

git clone https://github.com/antonellof/rondine.git
cd rondine
uv tool install .

rondine doctor
rondine suggest --profile coding
rondine suggest --configure 1 --save-as coding
rondine setup
rondine pull
rondine serve --preset coding
rondine verify --profile coding

doctor probes the host: operating system, architecture, RAM, GPU VRAM, CUDA capability, and installed engines. suggest combines that inventory with Rondine’s model catalog and hardware profiles. The selected plan can be saved as a named preset, so restarting the same configuration later is one command:

rondine serve --preset coding

Every suggestion includes the resolved engine settings. Rondine is automation, not a black box: the model, quantization, context, batch sizes, memory settings, and launch command remain inspectable.

Recommendations based on the hardware you actually have

Model selection starts with fit. On NVIDIA systems, Rondine sizes against GPU VRAM, not total system RAM. On Apple Silicon, it accounts for unified memory. It then ranks viable variants using the requested profile, provider preference, quantization quality, and available headroom.

The current catalog includes a practical range of coding models:

These are starting points rather than universal declarations of the “best” model. A coding workload, an interactive chat session, and a long-context document task have different latency and memory requirements. Rondine makes that trade-off explicit through profiles and dry runs.

You can inspect a launch without downloading or starting anything:

rondine serve qwen3.6-27b --profile coding --dry-run

If the curated catalog does not contain what you need, Hub discovery is built into the workflow:

rondine search "Qwen3.6 35B GGUF"
rondine inspect org/model-repo
rondine plan org/model-repo --quant Q4_K_M --save-as custom-model

GLM-5.2: an example where configuration matters

GLM-5.2 illustrates why a hardware-aware launcher is useful. It is a frontier-scale Mixture-of-Experts coding model: roughly 744B total parameters with about 40B active per token. The active parameter count helps compute efficiency, but the complete quantized weights still have to live somewhere.

Rondine’s preferred single-machine variant is Unsloth’s UD-IQ2_M GGUF:

• Approximately 239 GB of model weights
• At least 245 GB of usable unified memory recommended
• llama.cpp as the serving engine
• A practical 32K coding context by default
• Thinking enabled with model-specific sampling
• Intended for a 256 GB Mac Studio-class machine

There is also a smaller UD-IQ1_S option at roughly 223 GB, but the more aggressive quantization is a quality trade-off. Official BF16 weights belong on a large multi-GPU or multi-node system, not a single workstation.

Rondine marks GLM-5.2 as opt-in. A 48 GB Mac will not receive a recommendation merely because the model is fashionable, and a plan that does not fit is rejected rather than launched into an avoidable out-of-memory failure.

The same logic applies at smaller scales. On a 24 GB machine, choosing a model with enough headroom for context and cache is usually more useful than loading the largest possible file and leaving no memory for real work.

Engine-specific tuning

The user-facing plan is consistent, but the settings under it are specific to each backend.

For llama.cpp, Rondine can configure:

• GPU layer offload
• Flash attention
• Batch and micro-batch sizes
• KV-cache quantization
• Parallel request slots
• Context length and sampling options

For MLX-LM, it selects MLX model variants and applies Apple Silicon-specific runtime settings, including Metal synchronization behavior.

For vLLM, it manages settings such as:

• GPU memory utilization
• Maximum model length
• Prefix caching
• Tensor parallelism when explicitly enabled
• Blackwell / DGX Spark-oriented model formats such as NVFP4

Configuration is resolved from layered templates: engine defaults, the requested usage profile, and the detected hardware class. This keeps the policy understandable while avoiding a separate hand-written command for every model and machine combination.

A local OpenAI-compatible endpoint

Once rondine serve starts the backend, clients connect through the familiar OpenAI API format:

http://127.0.0.1:8080/v1

That makes the server usable from existing applications and coding clients without introducing a Rondine-specific protocol. A basic request looks like this:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "rondine/qwen3.6-35b-a3b",
    "messages": [
      {"role": "user", "content": "Review this Python function for race conditions."}
    ]
  }'

You can point Cursor, Continue, Aider, Codex CLI, Claude Code with a custom base URL, or another OpenAI-compatible tool at the same endpoint. Rondine owns the model-serving lifecycle; the coding client remains replaceable.

Binding to 127.0.0.1 also keeps the default exposure local. If you deliberately make an inference server reachable over a LAN, treat it as a network service: use a trusted network, authentication or a reverse proxy, and an appropriate firewall policy.

A coding profile, not a coding-agent lock-in

Rondine’s coding profile is scoped to inference. It applies a practical context length and model-specific sampling, enables reasoning where supported, and tunes the engine for sustained code-oriented requests.

The profile currently includes model-aware behavior such as:

• Qwen3.6 thinking mode and coding sampling defaults
• GLM-5.2 maximum reasoning effort
• DeepSeek-V4-Flash high reasoning effort
• Engine-level batching, cache, and memory settings appropriate to the host

After launch, rondine verify --profile coding checks server health and runs coding-oriented smoke tests. This catches the frustrating class of failure where a process is listening on a port but the loaded model, chat template, or generation path is not actually usable.

Rondine deliberately does not ship a proprietary agent loop. Repository access, tool execution, approvals, edits, and planning belong to the coding client. The server remains a standard local model endpoint rather than coupling inference to one editor or one agent framework.

Presets make experiments repeatable

Local inference experiments often end with an excellent command buried in shell history. Rondine presets preserve the complete plan:

rondine preset list
rondine preset show coding
rondine preset serve coding

That matters when comparing models. A coding preset can favor reasoning and a larger context, while a chat preset can use a smaller context and disable thinking for lower latency. Both remain named, inspectable, and reproducible.

It also makes the setup easier to explain to another developer. Instead of sharing a multi-line backend command with hardware assumptions embedded in it, you can share the model plan and let Rondine resolve the appropriate engine configuration on the target machine.

What Rondine is—and is not

Rondine is intentionally a thin layer:

1. Detect the host.
2. Match it to a hardware profile.
3. Rank compatible model variants.
4. Resolve engine-specific settings.
5. Download, serve, save, and verify the plan.

It is not a replacement for llama.cpp, MLX-LM, or vLLM. It is not a model manager that pretends every backend has identical capabilities. It is not a distributed inference orchestrator for a heterogeneous datacenter, and it is not another coding agent.

That limited scope is the design. Local inference already has excellent engines and clients. The missing layer is often the small, boring, hardware-specific control plane between them.

Try it

Rondine requires Python 3.11+ and is released under Apache-2.0:

git clone https://github.com/antonellof/rondine.git
cd rondine
uv tool install .
rondine doctor
rondine suggest --profile coding

The repository includes the model catalog, hardware templates, coding-client documentation, and engine-tuning notes:

• Rondine on GitHub
• Coding client guide
• Engine tuning

I am particularly interested in feedback from people running unusual configurations: high-memory Macs, 24–80 GB NVIDIA workstations, DGX Spark, and homogeneous two-node setups. Model catalogs age quickly; transparent hardware profiles and reproducible serving plans are the parts intended to last.

1. cmux for the terminal workspace
2. Unsloth Studio for inference
3. Pi for the coding-agent loop

That architecture still makes sense. But there is another way to handle the inference layer—especially when you want the configuration to come from the hardware rather than a model picker.

That is what Rondine does.

Rondine detects your machine, checks whether a model fits, selects an inference engine and quantization, applies hardware-specific settings, downloads the weights, and starts an OpenAI-compatible server.

For this setup, the stack becomes:

The interesting target is GLM-5.2: a frontier-scale Mixture-of-Experts coding model that requires approximately 245GB of memory even at 2-bit quantization.

The GLM-5.2 hardware reality

Rondine’s catalog describes GLM-5.2 as a 744B-parameter MoE model with approximately 40B active parameters per token and support for a context window of up to one million tokens.

The active parameter count reduces computation, but it does not eliminate the need to store the complete model.

Rondine currently provides these configurations:

For a single machine, Rondine prefers the Unsloth UD-IQ2_M GGUF and requires at least 245GB of usable memory.

In practice, this means a 256GB Mac Studio-class machine. Even then, context length matters. A model may fit at a small context and fail once the KV cache, batching, and operating-system memory are included.

For that reason, Rondine uses a practical 32K coding context instead of blindly enabling the model’s theoretical maximum.

Why use Rondine?

A raw GLM-5.2 llama.cpp launch requires several decisions:

• Which quantization should I download?
• Does it fit in unified memory?
• How much memory should remain available to macOS?
• What context length is realistic?
• Should the KV cache use F16, Q8, or something smaller?
• Should all layers be offloaded to Metal?
• Which batch and micro-batch sizes should I use?
• Should flash attention be enabled?
• How many parallel slots should the server expose?

Rondine turns those decisions into a reproducible plan.

It is not another inference engine. It installs and drives llama.cpp, MLX-LM, or vLLM while keeping the generated command visible.

Install Rondine

Rondine requires Python 3.11+ and uv:

git clone https://github.com/antonellof/rondine.git
cd rondine

uv tool install .

Start by inspecting the machine:

rondine doctor

On a compatible high-memory Mac, the output should identify Apple Silicon, unified memory, and available engines.

You can then ask Rondine to plan GLM-5.2 directly:

rondine plan glm-5.2 \
  --profile coding \
  --save-as glm-coding

GLM-5.2 is marked as opt-in because of its size. Explicitly naming it opts into planning it, but Rondine still rejects the plan if the detected hardware cannot satisfy the memory estimate.

Before downloading hundreds of gigabytes, inspect the generated launch:

rondine serve --preset glm-coding --dry-run

This prints the resolved llama.cpp command without starting it.

What the coding configuration contains

For GLM-5.2, Rondine’s coding profile applies:

context:           32768
temperature:       1.0
top_p:             0.95
min_p:             0.01
thinking:          enabled
reasoning effort:  max
parallel slots:    1

On Apple Silicon, the llama.cpp template also enables:

• Full Metal offload with -ngl 99
• Flash attention
• Continuous batching
• Q8 KV caches
• Large batch and micro-batch sizes where memory permits
• mlock to reduce unwanted swapping
• A single coding slot so context is not divided among concurrent clients

The effective command remains visible through the dry run, which is useful when experimenting or reporting a problem upstream.

Download GLM-5.2

Once the plan looks correct:

rondine setup --engine llama.cpp
rondine pull glm-5.2

The preferred quantization is downloaded from:

unsloth/GLM-5.2-GGUF

with the pattern:

*UD-IQ2_M*

This is a multi-hundred-gigabyte download. Plan for enough disk space beyond the model itself, use a stable connection, and remember that loading it will also require memory for context, cache, and the operating system.

If 239GB is too large, UD-IQ1_S reduces the weights to approximately 223GB, but that is a more aggressive quality compromise.

Start the OpenAI-compatible server

Launch the saved preset:

rondine serve \
  --preset glm-coding \
  --name glm

Rondine starts llama.cpp on:

http://127.0.0.1:8080/v1

The model alias exposed to clients is:

rondine/glm-5.2

Confirm it through the models endpoint:

curl http://127.0.0.1:8080/v1/models

Then run Rondine’s coding verification:

rondine verify \
  --name glm \
  --profile coding

The verification checks:

• Server readiness
• The /v1/models endpoint
• A short code-generation request
• A best-effort OpenAI-format tool-call request

This distinction matters: a process listening on port 8080 does not necessarily mean the model, chat template, and tool-calling path are working correctly.

Test GLM-5.2 directly

Before adding Pi, make one direct request:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "rondine/glm-5.2",
    "messages": [
      {
        "role": "user",
        "content": "Write a typed Python binary search function. Return code only."
      }
    ],
    "max_tokens": 512
  }'

If this works, the inference layer is ready. Pi becomes a client of the same endpoint.

Install Pi

Pi is a minimal coding agent with four core tools:

• read
• write
• edit
• bash

Install it with:

npm install -g --ignore-scripts \
  @earendil-works/pi-coding-agent

Pi can connect to custom OpenAI-compatible providers through:

~/.pi/agent/models.json

Add Rondine as a provider:

{
  "providers": {
    "rondine": {
      "baseUrl": "http://127.0.0.1:8080/v1",
      "api": "openai-completions",
      "apiKey": "rondine",
      "models": [
        {
          "id": "rondine/glm-5.2",
          "name": "GLM-5.2 local via Rondine",
          "contextWindow": 32768
        }
      ]
    }
  }
}

Most local llama.cpp servers do not validate the API key, but Pi expects a non-empty value.

Use the exact model identifier returned by:

curl http://127.0.0.1:8080/v1/models

Start coding with Pi

Move into a repository and launch Pi:

cd ~/Projects/my-application
pi

Inside Pi, open the model selector:

/model

Choose:

GLM-5.2 local via Rondine

Pi now supplies the agent loop while Rondine owns the inference lifecycle.

A useful first prompt is something constrained and verifiable:

Read the project structure and test configuration.

Find one untested error path in the HTTP client, add a focused test,
run only the relevant test suite, and show me the resulting diff.
Do not modify production code unless the test reveals a real defect.

This exercises reading, editing, shell execution, and multi-step reasoning without giving the model permission to refactor the entire repository.

Add cmux if you want the complete stack

cmux remains optional, but it makes the workflow easier to observe.

A practical layout is:

• Pane 1: Rondine server logs
• Pane 2: Pi running in the target repository
• Pane 3: Shell for health checks, Git, and memory monitoring

Start Rondine in the foreground if you want logs attached to a pane:

rondine serve \
  --preset glm-coding \
  --name glm \
  --foreground

Start Pi in another:

cd ~/Projects/my-application
pi

The architecture remains modular:

cmux
  ├── Rondine
  │     └── llama.cpp
  │           └── GLM-5.2 UD-IQ2_M
  └── Pi
        └── http://127.0.0.1:8080/v1

Swap Pi for another OpenAI-compatible coding client and the inference layer stays unchanged. Swap GLM-5.2 for a smaller model and Pi’s workflow stays unchanged.

If GLM-5.2 does not fit

Most developers do not have 256GB of unified memory. Rondine should reject GLM-5.2 on those machines—that is better than discovering the limitation after a 239GB download.

Use the same workflow with a smaller model:

rondine suggest --profile coding

Typical alternatives include:

• Qwen3.6 27B for 24–48GB Macs
• Qwen3.6 35B-A3B for larger Macs or 24GB NVIDIA GPUs
• Gemma 4 12B for smaller systems
• DeepSeek-V4-Flash at 3-bit for approximately 128GB systems

For a quick end-to-end test, Rondine also includes Qwen2.5-Coder 3B:

rondine plan qwen2.5-coder-3b \
  --context 4096 \
  --save-as small-coder

rondine pull qwen2.5-coder-3b
rondine serve --preset small-coder
rondine verify --name small-coder

The Pi configuration is identical apart from the model ID.

This is the main advantage of separating the agent from inference: the workflow does not depend on one model fitting forever.

Stopping and restarting

Stop the managed GLM server with:

rondine stop --name glm

Restart the saved configuration later:

rondine serve \
  --preset glm-coding \
  --name glm

Inspect it at any time:

rondine preset show glm-coding

The tuned command is no longer something you have to recover from shell history.

Honest limitations

This setup does not make GLM-5.2 small.

The recommended quant still occupies approximately 239GB, and a 256GB Mac leaves limited headroom. Large contexts may require reducing batches, using a smaller context, or moving to hardware with more memory.

Rondine currently provides a hardware-aware configuration, not a published GLM-5.2 performance guarantee. Its documented benchmark is a smaller Qwen2.5-Coder run on an M2 Pro; that validates the plan-to-API path but should not be presented as evidence of GLM-5.2 throughput.

Tool calling also depends on the model, llama.cpp version, chat template, and streaming compatibility. Run rondine verify, test Pi on a disposable branch, and inspect the server logs before trusting a long autonomous session.

Bottom line

The local coding stack works best when its layers remain replaceable:

• Rondine chooses, configures, downloads, and serves the model.
• llama.cpp performs inference.
• Pi reads, edits, and runs commands in the repository.
• cmux, optionally, keeps the sessions visible.

GLM-5.2 is the demanding case that makes hardware-aware planning valuable. On a compatible 256GB machine, Rondine turns a 239GB sharded GGUF and a page of llama.cpp flags into a named, inspectable preset exposed at a standard OpenAI URL.

Pi does not need to know how any of that works. It only needs:

http://127.0.0.1:8080/v1

That is the useful abstraction: a very large local model underneath, a very small coding agent on top, and a standard API boundary between them.

Links

• Rondine on GitHub
• The Local Agent Trio: cmux + Pi + Unsloth Studio
• Pi coding agent
• GLM-5.2 GGUF weights

After a fair amount of trial and error, this is the trio I keep coming back to:

1. cmux — native macOS terminal (Ghostty-based) built for juggling multiple AI agents in parallel
2. Unsloth Studio — local web UI + OpenAI/Anthropic-compatible API over llama.cpp
3. Pi — minimal open-source coding agent in the terminal

Point Pi at Unsloth. Run Unsloth in one cmux pane. Run Pi in another. When GLM-5.2 fits your hardware, you get frontier-class open weights at home. When it does not — and it often does not — the same stack still works with smaller GGUFs while you plan your RAM upgrade.

Why three tools instead of one?

Monolithic “local AI” apps try to be terminal + model server + agent + UI. They work until you want to swap the agent, swap the model, or run two agents side by side without two Electron apps eating 4GB each.

Each piece is replaceable. Swap Pi for Claude Code pointed at the same Unsloth endpoint. Swap Unsloth for raw llama-server. Keep cmux either way — that is the whole point.

Layer 1: cmux — the agent-aware terminal

cmux is an open-source (GPL-3.0), native macOS terminal from Manaflow (a YC S24 company). It is not a fork of Ghostty — it embeds libghostty as a rendering library inside a Swift/AppKit app, the same way other apps embed WebKit, and even reads your existing ~/.config/ghostty/config. It was built for one specific itch: running a pile of coding agents in parallel without losing track of which one is blocked on you. The author got tired of staring at a wall of Ghostty splits where every Claude Code notification just said “Claude is waiting for your input” with no idea which Claude. That frustration is the whole product, and it shows.

What matters for agent work:

• Vertical tab sidebar — git branch, working directory, ports, latest notification text per workspace
• Notification rings — blue halo on panes when an agent needs input (OSC 9/99/777 or cmux notify)
• Cmd+Shift+U — jump to the most recent unread notification
• Split panes — shell + Pi + browser without nested tmux config archaeology

cmux docs list Pi alongside Claude Code, Codex, OpenCode, and Gemini CLI as agents that work out of the box — because cmux is a terminal, not a vendor lock-in product. Notification hooks wire into agent lifecycle with:

cmux notify --title "Pi" --body "Waiting for your input"

Compared to tmux: tmux multiplexes inside a terminal; cmux is the terminal with GUI affordances agents actually need. Compared to Warp: cmux is native Swift, not Electron, and does not force one agent workflow.

Install from cmux.com or build from github.com/manaflow-ai/cmux.

Layer 2: Unsloth Studio — inference without the YAML archaeology

Unsloth Studio is an open-source, browser-based local GUI for running (and training) open models. Under the hood: llama.cpp, with extras — self-healing tool calling, code execution, web search, automatic inference tuning, Hugging Face Hub search built in.

Install:

curl -fsSL https://unsloth.ai/install.sh | sh
unsloth studio -p 8888

Open http://127.0.0.1:8888, set a password on first run, then head to Chat → Select model. Studio exposes:

• GET /api/health — health check
• POST /v1/chat/completions — OpenAI-compatible (Pi, Cursor, Continue, Cline…)
• POST /v1/messages — Anthropic-compatible (Claude Code, OpenClaw…)
• GET /v1/models — whatever is loaded in memory

API key lives under Settings → API (printed once at startup as sk-unsloth-...). Full reference: Unsloth API docs.

GLM-5.2 on the model picker — and the OOM badge

GLM-5.2 is Z.ai’s open Mixture-of-Experts model. The official model card lists it at 753B parameters with a 1M-token context under an MIT license (Unsloth’s docs round it to ~744B; the Studio picker in my screenshot reads 753.9B). Being an MoE, only a slice of those weights fire per token — but you still have to hold the entire thing in memory. Unsloth’s UD-IQ2_M 2-bit quant needs roughly 240–245 GB of RAM or unified memory — realistic on a 256 GB Mac Studio, not on a laptop without serious SSD offloading.

Here is what my Studio instance looked like on launch day:

GLM-5.2-GGUF at 753.9B with a red OOM pill is Studio being honest: your machine cannot load that variant right now. The trio setup still works — load Qwen3.5-9B-GGUF, gemma-4-12b, or another On Device model from the same picker while you plan for GLM-scale RAM.

When hardware cooperates, search GLM-5.2 in Studio, pick UD-IQ2_M (or higher quant if you have headroom), download the shards from unsloth/GLM-5.2-GGUF, and chat with High/Max thinking toggles in the UI. CLI alternative:

hf download unsloth/GLM-5.2-GGUF \
  --local-dir unsloth/GLM-5.2-GGUF \
  --include "*UD-IQ2_M*"

Terminal logs worth knowing:

• Studio binds to 127.0.0.1:8888 by default; use -H 0.0.0.0 for LAN access (Pi on another machine, phone on same network)
• llama.cpp prebuilt lacks MTP support — run unsloth studio update if you want speculative decoding on MTP-tagged GGUFs; models still load without it

Layer 3: Pi — the minimal coding harness

Pi (package @earendil-works/pi-coding-agent) is a minimal terminal coding agent by Mario Zechner — yes, the libGDX guy. It ships with exactly four tools (read, write, edit, bash), the shortest system prompt of any serious agent, multi-provider LLM support, and no baked-in sub-agent circus. The philosophy is “adapt the harness to your workflow, not the other way around” — anything the bigger agents bundle, you add yourself as a TypeScript extension or a Pi package. It is MIT, now stewarded by Earendil, and it is the agent core underneath OpenClaw, so it is battle-tested despite the minimalism.

Install:

npm install -g --ignore-scripts @earendil-works/pi-coding-agent
pi

Pi supports Anthropic, OpenAI, OpenRouter, Ollama, and any OpenAI-compatible local server via ~/.pi/agent/models.json. Docs: pi models.md.

For llama.cpp servers specifically, Hugging Face ships pi-llama — auto-discovers models from a running server:

pi install git:github.com/huggingface/pi-llama
# LLAMA_BASE_URL defaults to http://localhost:8080/v1

Unsloth wraps llama.cpp on port 8888 with its own API surface — so the direct path is OpenAI-compat config:

{
  "providers": {
    "unsloth": {
      "baseUrl": "http://127.0.0.1:8888/v1",
      "api": "openai-completions",
      "apiKey": "sk-unsloth-your-key-from-settings",
      "models": [
        {
          "id": "unsloth/GLM-5.2-GGUF",
          "name": "GLM-5.2 local (Unsloth)",
          "contextWindow": 131072
        }
      ]
    }
  }
}

Use the exact model id from curl http://127.0.0.1:8888/v1/models after loading in Studio. Then inside Pi:

/model

Select your Unsloth provider model. Pi’s status bar shows provider, model, token budget, and cost (zero for local).

Pi inside cmux

My cmux sidebar with shell, π, and Paperclip tabs:

The screenshot shows Pi on OpenRouter / GPT-5.5 — useful when GLM-5.2 OOMs or you need a quick cloud fallback. Same pane, swap model via /model to your Unsloth local entry when the GGUF is loaded. That is the trio’s flexibility: one terminal layout, two inference backends.

Pi v0.80.x changelog visible in the pane (June 2026) — active project, weekly releases, pi-ai provider auth improvements. Mirror line 192.168.1.8:3001 in the status bar is Pi’s optional session mirror for secondary clients — handy if you want phone/tablet observation without disturbing the agent pane.

Wiring the trio: step by step

Pane A — Unsloth (inference server)

unsloth studio -p 8888
# Load a model in the browser (Chat → Select model)
# Copy API key from Settings → API

Pane B — Pi (coding agent)

export PI_MODEL=unsloth/your-loaded-model-id   # optional shortcut
pi
/model   # pick Unsloth provider

cmux layout

• Tab 1 shell: Studio server logs, curl health checks, hf download progress
• Tab 2 π: Pi session on your repo (cd ~/Projects/my-app && pi)
• Optional Tab 3 Paperclip: docs, PRDs, attachments Pi reads via read

When Pi blocks on a tool approval or finishes a long task, wire a hook:

cmux notify --title "Pi" --subtitle "my-app" --body "Ready for review"

GLM-5.2: when the trio earns its name

GLM-5.2 is why you bother with Unsloth at all instead of pulling a 7B model from Ollama and calling it a day. The MoE design means the model carries frontier-scale knowledge while only activating a fraction of its parameters per token — you get agentic coding, long-horizon reasoning, and a 1M-token context out of weights that, quantized, fit in a (very large) workstation. Unsloth’s Dynamic GGUF quants are the part doing the heavy lifting: they compress aggressively while trying hard to preserve accuracy.

Realistic hardware map (from Unsloth docs):

No 256 GB Mac? Run the trio with Qwen3.5-9B, gemma-4-12b, or Qwen3.6-27B locally — same cmux layout, same Pi config, same API wiring — and treat GLM-5.2 as the target state when RAM catches up. The OOM badge in Studio is a feature, not a failure.

Troubleshooting the obvious failures

Pi cannot see tools / agent stalls after first bash

OpenAI-compat shims sometimes drop streamed tool_calls. Pi’s models.json compat flags help; for Ollama specifically use pi-ollama native provider. Unsloth’s self-healing tool calling is designed for this — if issues persist, check Studio logs while Pi runs.

Studio port conflict

Default is 8888; docs also mention 8000 depending on version. Terminal prints the actual URL on boot — use that in baseUrl.

GLM download is 6 shard files, hundreds of GB

Plan disk and time. hf download with --include "*UD-IQ2_M*" pulls only the 2-bit shards. Resume supported.

MTP warning in shell

Update Studio: unsloth studio update. MTP speculative decoding is optional; chat works without it.

Want Claude Code instead of Pi

Same Unsloth server, Anthropic route:

export ANTHROPIC_BASE_URL="http://127.0.0.1:8888"
export ANTHROPIC_API_KEY="sk-unsloth-your-key"
export ANTHROPIC_MODEL="unsloth/gemma-4-26B-A4B-it-GGUF"
claude --model "$ANTHROPIC_MODEL"

See Unsloth’s Claude Code guide. cmux does not care which agent occupies the pane.

Bottom line

The trio is not “install one app and pray.” It is a deliberate split:

• cmux — see every agent, know which one needs you
• Unsloth — search, download, serve, and API-wrap open models including GLM-5.2
• Pi — minimal harness that respects your repo and your keys

GLM-5.2 on Unsloth is the headline. The OOM badge in my screenshot is the honest footnote: frontier-class local AI genuinely exists now, but RAM is still the gate, and most of us are not sitting on 256 GB of unified memory yet. That is fine. The setup earns its keep today with a model that fits, and the day you do upgrade, the path is “pick a bigger GGUF in the same picker” — not “rewrite your whole workflow.”

So: start cmux, start Studio, load whatever your machine can hold, and point Pi at localhost:8888/v1. Then go refactor something you would rather not have shipped off to a cloud provider in the first place.

cmux: cmux.com · github.com/manaflow-ai/cmux. Pi: pi.dev · github.com/earendil-works/pi. Unsloth Studio: unsloth.ai/docs/new/studio. GLM-5.2: unsloth.ai/docs/models/glm-5.2.

No event log. No checkpoint. No way to pick up where it left off. The framework handled the reasoning loop beautifully, but nobody handled the boring part: execution durability.

In 2026, Google open-sourced AX (Agent Executor) — a distributed agent runtime that coordinates agentic loops, logs every step, recovers from failures, and audits tool calls through a single controller. It is explicitly not an agent framework, not a managed service, and not tied to one model. It is the orchestration layer — how execution proceeds, resumes, and gets inspected.

For where agent processes physically run at scale (suspend, multiplex, teleport across Pods), see the companion post on Agent Substrate — the other open-source project Google’s GKE team shipped at the same time, and the recommended Kubernetes target for AX.

Why agents need a runtime, not just a framework

LangGraph, ADK, AutoGen — excellent for structuring agent logic. They do not give you:

• Durable execution across crashes and deploys
• Connection recovery when the client disconnects mid-stream
• Trajectory branching to explore alternatives without destroying history
• Centralized auditing of every skill, tool, and sub-agent call
• Isolated actors for tools, sandboxes, and remote agents in distributed harnesses

As agents evolve from chat assistants to autonomous long-running workers, those gaps stop being edge cases. Google’s announcement blog frames AX as the open foundation every sophisticated agentic application will need — built in public so design decisions get validated before APIs freeze.

AX is in active early development. PRs are temporarily paused. Protocols will break. Read the warnings; the architecture still matters.

What AX provides

AX ships as a Go runtime plus an ax CLI:

• Single-writer controller — one source of truth for execution state
• Durable event log — SQLite by default; replay on recovery
• Resumable streams — clients reconnect with --last-seq and catch up
• Trajectory forking — ax fork branches at any checkpoint
• Isolated actors — agents, tools, skills, sandboxes as separate processes

  Client
    │
    │  resumable stream
    ▼
  Router ──► AX Controller ──┬──► Remote Agent (isolated actor)
              (event log,     ├──► Tool / MCP server (isolated actor)
               registry)      └──► Environment / skills (isolated actor)

What AX is NOT

Google is explicit — and the discipline is refreshing:

• Not a managed service — self-hosted; you operate it
• Not an agent framework — bring LangGraph, ADK, whatever
• Not a coding harness — Antigravity integration is roadmap, not product
• Not model-specific — built-in Gemini agent included, not required

The industry keeps conflating “agent product” with “agent runtime.” AX draws the line.

The CLI: flight recorder for agentic execution

go install github.com/google/ax/cmd/ax@latest

# Local execution with built-in planner + bash tool
ax exec --input "List files in this directory"

# Server mode
ax serve   # :8494 by default

# Client disconnected? Catch up from sequence 12
ax exec \
  --conversation d85a4b4e-c53b-4c84-b879-f10d905bce40 \
  --last-seq 12 \
  --resume

# Branch exploration without destroying source history
ax fork \
  --src-conversation 38460323-9a78-41cb-8991-022b0ff2c19c \
  --dest-conversation e5e26e38-53a2-4f22-b1cb-ae867357df83 \
  --src-seq 12

# Visualize execution in browser
ax trace --conversation 1a6e0b29-87c2-4af0-81ac-0c73bf8fa293

ax trace tells you the audience — engineers who wanted a flight recorder, not another chat UI.

Configuration (ax.yaml)

server:
  address: ":8494"

eventlog:
  sqlite:
    filename: "eventlog/log.sqlite"

planner:
  gemini:
    model: "gemini-3.5-flash"
    timeout: "60s"
    skills_dir: "./examples/skills"

registry:
  remote_agents:
    - id: "medical-deep-researcher"
      name: "Medical Deep Researcher"
      description: "Deep medical research via pubmed and clinicaltrials.gov"
      address: "localhost:50051"

Remote agents implement AgentService gRPC (proto/ax.proto). Examples ship for native remote agents, ADK (Python), A2A protocol bridges, and experimental Colab agents.

The built-in planner includes a bash tool with explicit user approval before execution — small detail, production thinking.

Custom agents in three terminals

Terminal 1 — remote agent server:

go run examples/remote_agent/main.go   # :50051

Terminal 2 — AX controller:

ax serve

Terminal 3 — execute:

ax exec --server localhost:8494 \
  --input "HELLO, CAN YOU LOWERCASE WHAT I JUST SAID?"

Register remote agents in ax.yaml under registry.remote_agents. Point ax exec --agent coding at any registered ID.

Three “resume” problems — and who solves what

“Resume” means different things. AX and Agent Substrate solve different ones:

AX owns execution semantics — which agent spoke, which tool fired, what was the plan.

Substrate owns compute state — RAM, filesystem, Pod placement.

Long-running agents need both. AX on bare metal or plain Kubernetes works; on Kubernetes, Google recommends deploying AX on Substrate for actor resumption at the compute layer. Watch the combined demo.

Deployment on Kubernetes

AX is compute-agnostic but aims for the best experience on Kubernetes:

# Install CLI
go install github.com/google/ax/cmd/ax@latest

# Production path: AX + Agent Substrate
# See manifests/README.md in the ax repo

Google’s GKE blog covers Agent Sandbox on GKE plus Substrate as the agent-first compute layer — with AX as the runtime on top. Both AX and Substrate came out of Google’s GKE/agent infrastructure work and were announced together; Substrate carries the standard “not an officially supported Google product” disclaimer, but it is a Google-originated open-source project, not a third-party one.

Roadmap

From the AX README:

1. Antigravity as built-in harness
2. Bring Your Own Harness (BYOH)
3. Suspension/resumption of subagents
4. Tool call approvals in subagents
5. Resumption protocol improvements

Resumable streaming and agent communication protocols are actively evolving — plan for breaking changes.

Where AX sits in the stack

Most teams have models and frameworks. Almost nobody has the runtime and compute rows. That is why demos feel magical and production feels haunted.

Honest limitations

• Active early development — PRs temporarily paused
• Resumable streaming protocols will break before stable release
• Self-hosted only — you operate clusters, logs, backups
• Collaboration via ax-dev@google.com; file issues freely

If you need something Monday, use Postgres for session state and pray. If you are building agent infrastructure for the next few years, install ax, run ax exec, kill your terminal, --resume with --last-seq, then ax trace what happened. An hour well spent.

Federating deployment models

Google positions AX as a bridge between deployment models — on-prem sovereignty, managed frontier agents, custom LangGraph/ADK builds, A2A-connected agents. You own models, harnesses, and compute; AX provides the execution contract: event log, resumption, audit, isolation.

Enterprise adoption of agents needs that contract open-sourced. Google shipped it. Whether it becomes the Linux of agent runtimes or a reference implementation depends on the next twelve months of API stability and community adoption.

AX: github.com/google/ax, agentexecutor.io. Announcement: Agent Executor blog post. Companion post: Agent Substrate — zero-idle Kubernetes.

The reason is boring once you see it. The demo keeps one process warm on a laptop. Production maps one user session to one Kubernetes Pod, bills you for RAM while the agent sits waiting on a model, a tool, or a human, and still loses its context the moment you scale to zero. Agents spend most of their life idle — waiting for an event, a token, a click — yet the standard setup forces a bad trade: pay for memory that does nothing, or cold-boot and throw away volatile state.

Agent Substrate is an open-source project aimed squarely at that problem. It was introduced in 2026 by Google’s GKE team — announced alongside Agent Sandbox on GKE and Agent Executor (AX) — and developed in the open with the community (Solo.io’s kagent team contributed work from a near-identical project). One caveat worth stating up front, because the README states it: this is not an officially supported Google product. It is a Google-led open-source effort, not a GCP service with an SLA.

What it gives you is a session-centric, zero-idle architecture on Kubernetes: decouple logical actors from physical workers, suspend and resume process state in well under a second, and multiplex many stateful sessions onto a small warm pool of pods.

This is the compute layer of the agent stack — where processes live, hibernate, and teleport. For the orchestration layer (event logs, trajectory forking, audit trails), see the companion post on Google AX.

The stateful gap Kubernetes does not solve

Agents are not web servers. A web server handles thousands of short requests. An agent handles one long request, goes idle, wakes up, calls tools, blocks on MCP, and dies mid-thought because the node OOM’d.

Traditional Kubernetes optimizes for thousands of long-running services. Agent workloads look like millions of sub-second activations with long idle tails. One Pod per session is economically absurd. One process per laptop does not survive a deploy.

Substrate targets three operational needs at the hardware level:

1. Suspend and resume RAM + filesystem in milliseconds — not “restart and replay the prompt”
2. Multiplex idle sessions onto shared workers without cross-talk
3. Isolate arbitrary code execution via sandboxed OCI containers

Frameworks solve reasoning. Substrate solves density and session mobility.

What Agent Substrate is (and is not)

Agent Substrate is not an SDK for building agents. It is infrastructure for running them — a control plane on top of Kubernetes that maps many actors onto fewer workers (Pods).

A note on what it is not, because there has been some confusion: it is not a managed, officially supported Google product. The README is explicit about that. But it is a Google-originated open-source project — the GKE team introduced it, it builds directly on Agent Sandbox’s secure runtime and snapshotting, and it lives in its own agent-substrate org with an open community. So “Google built it in the open, but does not support it as a product” is the accurate reading — not “independent project Google merely partners with.”

Two vocabulary words appear in every demo and doc:

• Actor — a logical session; a private instance of an agent or any OCI workload
• Worker — physical compute; typically a pre-initialized Pod in a warm pool

Actors are often suspended upon creation. They exist logically, cost nothing while idle, and hydrate when traffic arrives. Substrate’s core innovation is this decoupled lifecycle — rapid suspend/resume on any worker, state persisting independent of underlying hardware.

Watch the launch demo

The Agent Substrate OSS Launch Demo (~8 minutes) walks through the counter demo, a “secret agent” zero-idle pattern, and a boardroom UI scaling to ~250 concurrent agents on eight GKE worker pods — call it ~30:1 oversubscription, which lines up with the project’s own “30× more actors than pods” framing.

What the demo actually proves

The video has three acts. Each maps to a repo demo.

Act 1 — Counter: session teleport. A Go HTTP server keeps an in-memory counter. Create an actor (suspended), resume, increment: 1, 2. Manually suspend, flood the cluster until the original Pod is occupied. Substrate resumes on a new Pod — IP changes — counter reads 3. Living memory, new hardware.

Act 2 — Secret Agent: zero-idle by default. A toy agent returns a volatile secret from RAM, then self-suspends via the Substrate API. While waiting, it costs nothing. The gateway resumes on inbound traffic in milliseconds; the worker frees automatically after work. 24 agents on eight Pods — 3:1 oversubscription — multiplexed in a parallel pulse.

Act 3 — Boardroom UI: swarm at 30× density. Visual layer on the same APIs as kubectl-ate. A lead architect actor spawns sub-agents; contention steals warm Pods without leaking context. The architect hibernates to GCS when done; a reviewer recalls it on a different Pod (.117 → .121) with state intact. Finale: ~250 agents, eight slots. Demo claims ~45-second cold boots replaced by sub-second rehydration — treat “97% efficiency” as launch marketing; the snapshot mechanism is the real story.

gVisor checkpoint/restore handles RAM and filesystem snapshots. “Instant session teleport” is fair after you watch the counter survive a Pod swap.

Architecture

                    ┌─────────────────────────────────┐
  HTTP / gRPC       │         atenet-router           │
  requests    ───►  │   (DNS + Envoy routing)         │
                    └──────────────┬──────────────────┘
                                   │
                    ┌──────────────▼──────────────────┐
                    │         ate-apiserver           │
                    │   (actor lifecycle, scheduling) │
                    └──────────────┬──────────────────┘
                                   │
         ┌─────────────────────────┼─────────────────────────┐
    ┌────▼────┐              ┌─────▼─────┐             ┌─────▼─────┐
    │ Worker  │              │  Worker   │             │  Worker   │
    │  Pod A  │              │  Pod B    │             │  Pod C    │
    │ actor 1 │◄─suspend──►  │ actor 47  │             │ actor 12  │
    └─────────┘              └───────────┘             └───────────┘

Kubernetes provisions nodes and networking. Substrate takes the Kubernetes control plane out of the hot path for actor scheduling — critical when wake/sleep cycles happen constantly.

Framework-agnostic by design

Substrate manages OCI containers via gVisor — it does not care what framework built the agent:

• ADK — session identity and persistent working memory
• LangChain — long-running stateful agents with sandboxed tools
• Claude Code / Codex — multiplexed coding environments with terminal state
• MCP servers — durable, sandboxed tool actors

Demos worth running: Counter, Sandbox (Antigravity), Claude Code Multiplex, Secret Agent.

Google AX lists Substrate as its recommended Kubernetes deployment target — AX coordinates agentic loops on top; Substrate provides the dense, resumable compute underneath. The AX-on-Substrate demo shows the full stack.

Quickstart on kind

git clone https://github.com/agent-substrate/substrate.git
cd substrate

hack/create-kind-cluster.sh
hack/install-ate-kind.sh --deploy-ate-system
hack/install-ate-kind.sh --deploy-demo-counter

go install ./cmd/kubectl-ate
kubectl ate create actor my-counter-1 --template ate-demo-counter/counter

kubectl port-forward -n ate-system svc/atenet-router 8000:80

curl -X POST \
  -H "Host: my-counter-1.actors.resources.substrate.ate.dev" \
  -i http://localhost:8000/

Suspend, resume on another worker, counter still increments. One curl captures the value proposition.

GKE quickstart: go run ./tools/setup-gcp --all then ./hack/install-ate.sh --deploy-ate-system. See the README for teardown and partial deploy flags.

Where Substrate sits in the stack

This is not inference orchestration (Cognitora) or local inference (DwarfStar 4). Substrate is purely session mobility and hardware efficiency for agent-shaped workloads.

Honest limitations

• VERY early development — APIs guaranteed to change
• Not an officially supported Google product
• gVisor snapshot path is complex; distributed recovery has edge cases
• PRs may not merge unless aligned with core roadmap

If you need production tomorrow, keep plain Kubernetes Jobs. If you are designing agent infrastructure for 2027, run the counter demo on kind this weekend — suspend an actor, steal its Pod, resume elsewhere, watch the counter keep counting.

Agent Substrate: github.com/agent-substrate/substrate. Community: ate-dev Google Group, CNCF Slack #substrate-users. Companion post: Google AX — distributed agent runtime.

I did not expect it to blow past 12,000 GitHub stars within its first few weeks either (it has kept climbing since). But after reading antirez’s own words on the launch and his follow-up on distributed inference, the hype makes sense. Three things converged at once: a quasi-frontier open-weights model fast enough to matter on a laptop, an asymmetric 2/8-bit quantization recipe that actually works, and a decade of local-AI experimentation finally paying off — accelerated, as antirez openly admits, by heavy use of GPT 5.5 during development.

This is not a generic GGUF runner. It is a deliberate bet that local inference should feel finished, not merely possible.

The problem DS4 refuses to solve

The local LLM landscape is a graveyard of half-integrated projects. A new model drops. Someone ports it to llama.cpp. Tool calling breaks. Context windows shrink mysteriously. KV cache eats all your RAM. You wire up Ollama, then Open WebUI, then a coding agent, then wonder why the 284B-parameter MoE feels dumber than the 27B dense model you replaced.

DS4 takes the opposite approach, stated plainly in the README:

Not a generic GGUF runner, not a wrapper around another runtime — completely self-contained.

The vision is three pieces working together out of the box:

1. Inference engine with HTTP API and CLI
2. GGUF files crafted for that engine — validated against official logits, not “close enough”
3. Testing and agent integration so you know it works before you trust it with your codebase

The model may change over time — antirez expects future DeepSeek checkpoints, maybe ds4-coding, ds4-legal, ds4-medical variants — but the constraint stays: one best open-weights model at a time, practically fast on high-end personal hardware.

That narrowness is the feature. Redis did not try to be a general-purpose database. DS4 does not try to be a general-purpose inference server.

Why DeepSeek V4 Flash, specifically?

After weeks of comparisons, the DS4 team argues Flash deserves its own engine for reasons that matter in daily use, not just on benchmarks:

• Thinking mode that scales with problem complexity. Enable thinking on many models and watch them monologue for 8,000 tokens about a two-line bug. Flash’s thinking section is often ~1/5 the length of competitors and proportional to actual difficulty — making thinking mode usable locally where it was previously a curiosity.
• A 1-million-token context window with aggressively compressed KV cache — and DS4 treats that cache as a first-class disk citizen, not something that must live in RAM until your MacBook fans sound like a jet engine.
• Edge-of-knowledge sampling. Ask niche political or cultural questions and 284B routed MoE parameters show up. Dense 27B–35B models feel smaller in ways that are hard to benchmark but obvious in conversation.
• Asymmetric 2-bit quantization that is not a joke. Only routed MoE experts get crushed to 2-bit (IQ2_XXS up/gate, Q2_K down). Shared experts, projections, routing, and attention stay at higher precision. Result: Flash runs on 96–128 GB MacBooks; PRO fits on 512 GB Mac Studio class machines.

antirez’s blunt assessment in news/165: if you imagine local models as experience A and frontier cloud models as B, DS4 is a lot more B than A. For the first time in his years of local inference experiments, he uses it for work he’d normally send to Claude or GPT.

I have been running similar experiments on my own hardware. The gap between “toy local model” and “I would actually ship code reviewed by this” has never been smaller.

What’s in the box

DS4 ships as a handful of native binaries — no Python runtime holding the critical path hostage:

Backends: Metal is the primary target (MacBook Pro/Studio, from 96 GB up). CUDA builds target the NVIDIA DGX Spark (GB10) and generic Linux GPUs. The README also lists Strix Halo (ROCm) — Framework Desktop and similar unified-memory AMD systems — as a backend target, so this is no longer a Mac-only story.

The engine borrows quant layouts, GGUF ecosystem knowledge, and kernel ideas from llama.cpp/GGML (acknowledged prominently in the LICENSE), but ds4.c is its own inference path — not a fork you ./llama-cli your way through.

Getting started

Download the imatrix-tuned quant that matches your RAM, build, run:

git clone https://github.com/antirez/ds4.git
cd ds4

# 96/128 GB machines — imatrix-tuned 2-bit (recommended starting point)
./download_model.sh q2-imatrix

# macOS Metal build
make

# Interactive session
./ds4 -m ds4flash.gguf --ctx 32768

# OpenAI-compatible server for Cursor, Cline, etc.
./ds4-server --ctx 100000 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192

Weights live on Hugging Face (antirez/deepseek-v4-gguf). The download script resumes partial transfers and symlinks ./ds4flash.gguf to your chosen variant. Do not feed it arbitrary GGUF files — tensor layout, quant mix, and optional MTP state are all DS4-specific.

Status: beta. antirez worked ~14 hours/day the first week (reminiscent of early Redis, he notes). The ds4-agent is explicitly alpha. Use --trace and file issues with full session logs when something breaks.

Speed: the numbers that matter locally

Published Metal benchmarks (greedy decode, --nothink, 32K context) tell a clear hardware story:

The M5 Max numbers align with antirez’s hardware take: right now, the best local-inference deal might literally be a laptop — ~500 t/s prefill and ~35–40 t/s decode on 2-bit Flash for $6–7K. A Mac Studio M3 Ultra with 512 GB runs PRO at usable (if not thrilling) ~10–13 t/s decode and ~150 t/s prefill — frontier-class weights at home for ~$12K total spend.

Not cloud-fast. But no API bill, no data leaving your machine, and no “sorry, we’re rate-limiting Pro users again” email.

For thermal sanity on long runs, --power N throttles GPU duty cycle (50 = half speed, less fan scream). Available on CLI, server, agent, and bench tools.

Disk-backed KV cache: RAM is not the only tier

Most inference stacks assume KV lives in memory or dies. DS4’s design thesis — spelled out in the README — is that modern NVMe is fast enough to change the equation, especially with DeepSeek’s compressed cache format.

Practical effect:

./ds4-server \
  --ctx 100000 \
  --kv-disk-dir /tmp/ds4-kv \
  --kv-disk-space-mb 8192

You can run contexts that would evict you from RAM on lesser setups. The native agent stores sessions under ~/.ds4/kvcache with /save, /list, /switch — resuming a saved session skips prefill entirely because the KV state is the session.

This is the kind of architectural choice that sounds incremental and feels revolutionary the first time you /switch back to a 200K-token debugging session instantly.

The native coding agent

Most coding agents treat inference as a black-box HTTP call. DS4’s agent inverts that: inference is controlled from inside the agent, no socket boundary on the hot path, tools and system prompt designed vertically for DeepSeek V4.

What that buys you:

• Instant tool calling — no DSML conversion layer; native model format end to end
• KV truth — session state cannot drift from cache state; they are the same object
• Live prefill progress bar — sounds cosmetic, matters when you’re waiting on 100K tokens of repo context
• Session switching without re-prefill — /switch to another saved conversation and keep going

antirez plans to eventually split client/server with a stateful protocol once the agent matures. Today it is alpha — usable, opinionated, not yet “install and forget.”

Pair it with directional steering (dir-steering/) and you get something antirez highlights in news/165: the first local setup where vector steering makes the model feel less constrained, not more gimmicky.

Distributed inference: when one MacBook is not enough

The distributed inference docs are where DS4 stops being a laptop toy and starts looking like infrastructure. antirez’s news/167 frames the macro picture: NVIDIA clusters are not getting cheaper, RAM shortages may delay the next Mac Studio Ultra, and tensor parallelism over Thunderbolt is a non-starter (go read NVLink speeds and weep).

DS4 implements pipeline parallelism today — split layers across machines, ship activations (small), keep KV shards local:

# Machine A — coordinator, layers 0–30
./ds4 \
  -m gguf/DeepSeek-V4-Pro-Q4K-Layers00-30.gguf \
  --role coordinator \
  --layers 0:30 \
  --listen 169.254.43.68 1234

# Machine B — worker, layers 31 through output head
./ds4 \
  -m gguf/DeepSeek-V4-Pro-Q4K-Layers-31-output.gguf \
  --role worker \
  --layers 31:output \
  --coordinator 169.254.43.68 1234

Prefill pipelines across the cluster like an assembly line — measured 1.38×–1.85× speedup on two M5 Max MacBooks over Thunderbolt 5 for long prompts. Decode cannot pipeline (autoregression is sequential); expect ~15–20% slowdown vs single machine due to per-token network hops. Distributed inference is for fitting bigger models and faster long prefills, not faster token generation.

Real-world link comparison from the README (same two hosts, 8K prompt):

WiFi works. It is not fun. VPN across continents is for “I need to run the model at all,” not daily driving.

What comes next: three paths, not one

antirez outlines three distributed strategies in news/167:

1. Pipeline / layer split (shipped) — duplicate effective memory, accelerate prefill, accept slower decode.
2. Expert-parallel vertical split (experimental) — both machines load full 2-bit weights; route half the MoE experts to each box via Apple RDMA; viable for PRO’s larger routed experts where activation traffic stays tiny.
3. Model ensemble (research) — completely shared-nothing: run different models on different machines, combine logits or pick the lower-perplexity continuation. Recent work shows ensembles can outperform either model alone — like an implicit two-expert MoE where routing is “who is more confident about the next token.”

Tensor parallelism? antirez bets no on Apple Thunderbolt vs NVLink. The winning patterns all minimize bytes on the wire.

Two Mac Studio 512 GB machines could run full-size PRO Q4 today with the split GGUF workflow — antirez demoed ~11.5 t/s generation with balanced ~40 ms local / ~47 ms remote layer times. Frontier weights, no datacenter lease.

Honest caveats (because antirez is honest)

A few things the README says out loud that most launch posts bury:

• AI-assisted development. Built with heavy GPT 5.5 help. If that offends you, DS4 is not your project. The ideas, testing, and debugging are human-led; the typing is not all hand-crafted C.
• CPU path is for diagnostics only. On macOS, running CPU inference can kernel-panic the machine due to a virtual memory bug. antirez’s comment: “Software sucks.” Metal or CUDA for real work.
• PRO support is experimental — naturally limited to 512 GB hardware unicorns.
• Distributed protocol has no encryption or auth — trusted network, same git commit on all nodes.
• MTP speculative decoding exists but is correctness-gated and currently a slight speedup at best.

This transparency is refreshing. Beta software that admits beta beats production software pretending it is done.

How DS4 fits the wider local-AI stack

If you are running datacenter-scale orchestration — KV-aware routing across dozens of H100s, prefill/decode disaggregation, etcd cluster state — projects like Cognitora (which I have written about separately) occupy that lane.

DS4 occupies a different lane entirely: personal sovereignty. One developer, one (or two) Apple Silicon boxes, one model family, everything from GGUF validation to coding agent in a single repo. The philosophical overlap with early Redis is striking — optimize one workload brutally well, ship the complete experience, ignore the rest.

Neither replaces the other. DS4 is the integrated appliance. The generic stack is the swiss army knife.

What antirez is building toward

From news/165, the roadmap reads like a product plan, not a science project:

• Quality benchmarks and regression gates (official continuation vectors, ds4-eval, ds4-bench)
• A coding agent that graduates from alpha to daily-driver
• Home hardware CI so releases do not depend on “works on Salvatore’s MacBook”
• More ports and distributed modes — serial and parallel
• Model churn as DeepSeek ships new checkpoints

The closing line stuck with me: “AI is too critical to be just a provided service.”

That is the whole argument for DS4 in one sentence. Cloud APIs are convenient. They are also someone else’s computer, someone else’s retention policy, and someone else’s idea of what your prompt should cost this quarter.

Bottom line

DwarfStar 4 is not the inference engine for everyone. It is the inference engine for people who looked at DeepSeek V4 Flash, checked their MacBook’s RAM, and thought: I want this to actually work — tools, cache, agent, benchmarks, distributed PRO across two Studios — without spending six weekends gluing pieces together.

antirez built Redis because developers deserved a better in-memory data store. DS4 comes from the same instinct applied to local LLMs: stop shipping half-finished runtimes and call it progress.

Clone it. Download the imatrix quant. Run ./ds4. See if you, too, end up asking a local model what you used to send to Claude — and getting an answer you can use.

DwarfStar 4 is under active development. Pin a commit or watch releases if you are deploying beyond weekend experiments. Primary references: antirez/ds4 on GitHub, A few words on DS4, Distributing LLM inference in DwarfStar.

The two reference points the rest of the industry agrees on are NVIDIA Dynamo (Rust core, Python frontend, Kubernetes-first) and a long tail of generic ML serving stacks: Ray Serve, KServe, NVIDIA Triton Inference Server, BentoML, and the vLLM Production Stack. Each makes a different tradeoff between “specialized for LLM inference” and “general purpose,” between “one click on a managed cloud” and “I can run it on bare metal in a datacenter I own.”

Cognitora is an open-source LLM inference orchestration layer that lands in a deliberately specific spot in that design space: bare-metal-first, Rust-only, engine-agnostic, KV-cache-aware. It does not replace vLLM or SGLang — it coordinates them into a cluster. It is distributed as six statically-linked binaries with no Python control plane, no JVM operator, and no hard Kubernetes dependency. The same artifacts run as systemd units on a rack of servers, as recipes or docker compose on a single host, via a Helm chart on Kubernetes, or as Terraform-provisioned VMs across AWS, GCP, Azure, and Hetzner.

As of v0.3.0 (May 2026), the OpenAI-compatible surface includes /v1/chat/completions, /v1/completions, /v1/embeddings (real round-trip to the engine, not synthetic vectors), and /v1/models. The admin CLI cgn-ctl reads and writes etcd for cluster state (cluster nodes, cordon/drain, model load/unload), and cgn-ctl install --target single-node --apply can render cognitora.toml plus compose.yaml and bring the stack up with one command. cgn-metrics exposes Prometheus federation at /federate so upstream Prometheus scrapes a single endpoint with per-component labels. There is also a single-manifest Kubernetes quickstart (etcd + llama.cpp engine + router + agent + metrics in one Pod, LoadBalancer on port 80) that has been exercised end-to-end on GKE Autopilot—details in the “Try it” section below.

This post is the long-form version of why that combination of choices, and what falls out of them.

# One-line install — six static binaries, no runtime deps (pin a release for reproducibility)
curl -fsSL https://raw.githubusercontent.com/antonellof/cognitora-inference/main/deploy/installer/install.sh | CGN_VERSION=v0.3.0 sh

# Bring up Llama-3.1 8B on a single GPU with vLLM
bash recipes/llama3-8b/vllm/agg/up.sh
# Equivalent:
# cgn-ctl recipe up llama3-8b/vllm/agg

# Same model, prefill/decode disaggregated across two GPUs
bash recipes/llama3-8b/vllm/disagg-single-node/up.sh

# Single-node Docker install (writes cognitora.toml + compose.yaml, optional --apply)
cgn-ctl install --target single-node --model llama3-8b --engine vllm --apply

The thing inference engines don’t do

A modern inference engine is a token factory. Hand it a request, get tokens back. The contract is “one process, one model, one node.” Everything outside that contract — which replica should serve this request, which replica already has the system prompt cached on-GPU, when to spill cold KV blocks to RAM or SSD, when to migrate a long-context request from a prefill-optimized box to a decode-optimized one, how to weight a thermally-throttled GPU in the routing decision — is, from the engine’s point of view, somebody else’s problem.

Historically somebody else was a stack of glue: an Nginx in front, a Redis for KV metadata, a Python scheduler reading Prometheus, a Kubernetes operator reconciling deployments, a custom autoscaler. That stack works. It is also five processes in five languages with five failure modes, and it is the part of the system that has the worst observability story precisely when an SRE needs it most — at 03:00 on the first day a new model is in production.

The Cognitora bet is that the orchestration layer should be one runtime, in Rust, with a small surface area. Six binaries, each one statically linked, each one with a well-defined responsibility:

The cgn-operator is optional on purpose. If you run on bare metal, the systemd path is a first-class citizen rather than the “deprecated, please use the Helm chart” path that most cloud-native projects eventually push you toward.

KV-aware routing: the routing decision is the product

The single biggest lever in multi-node LLM inference is not token throughput. It is KV cache reuse. If a request shares a 4,000-token system prompt with a request that finished 200 ms ago on replica B, sending the new request to replica B saves 4,000 tokens of prefill compute. Sending it to replica A — because round-robin said so — burns a GPU-second to recompute something that already exists in HBM somewhere else in the cluster. At fleet scale that decision dominates everything else.

There are two common ways to encode “which replica has which prefix”:

1. Radix trees over chained block hashes. This is what Dynamo’s KV-aware router uses. Each block of KV is hashed; the cluster maintains a radix tree keyed on those hashes; the router descends the tree to find the deepest match. Fast, memory-efficient, the canonical structure.
2. Sequence-chained BLAKE3 digests with longest-prefix overlap. This is Cognitora’s choice. Each block’s digest is chained from the previous block’s digest, so the digest at position i summarizes the entire prefix [0..i]. Routing becomes “which replica reports the deepest prefix match against this digest sequence?”

The two approaches are close cousins. The motivating difference for Cognitora is positional correctness on interleaved requests. With a chained digest, two requests that share tokens out of order — same content, different positions — produce different chains, so the router does not falsely claim a cache hit that would force the engine to recompute or, worse, return a positionally incorrect KV. On real-world traces with heavy system-prompt sharing the practical hit ratio sits at ≥ 0.55, and the routing decision itself is sub-millisecond:

Worth flagging the obvious caveat: those are the project’s stated targets, not numbers measured on your traffic. The shape of the metrics matters more than the absolute values — sub-millisecond routing, single-digit-millisecond HTTP overhead, sub-200-µs warm hits. If any of those numbers grew by an order of magnitude the architecture would fall apart, so they are useful as a sanity envelope.

Disaggregation: prefill and decode want different hardware

Prefill is compute-bound. Decode is memory-bandwidth-bound. Running both on the same SKU is a compromise — you either over-provision compute for the decode phase or starve memory bandwidth for the prefill phase. The fix, popularized by DistServe and now standard in production stacks, is disaggregated inference: prefill on one pool of GPUs, decode on another, with the KV blocks streamed between them.

Cognitora handles disaggregation through the NIXL connector — the same NVIDIA-developed transport library Dynamo uses — and exposes the choice as a single TOML knob:

[engine]
kv_offload = "nixl"   # one of: none | nixl | lmcache | hicache | kvbm

That single knob renders the right engine argv for vLLM, SGLang, or TensorRT-LLM. The recipe folders ship the topologies most people actually want — recipes/llama3-8b/vllm/agg/, recipes/llama3-8b/vllm/disagg-single-node/, recipes/llama3-70b/vllm/agg/ — so you do not have to translate “I want 70B FP8 on 4×H100 with TP=4 and disaggregation off” into engine-specific flags.

Engine support matrix:

llama.cpp and OpenAI-compatible servers as first-class engines — not “we tolerate them, here is a config flag” — is a genuine differentiator. It means the same router that fronts your H100 fleet can also fan requests out to a developer’s Ollama on a Mac mini, or to a hosted Anthropic / OpenAI / Together endpoint when on-prem capacity is saturated. That changes what cluster topologies are reasonable to consider.

Tiered KV cache + cross-cluster federation

The KV cache is a hierarchy in any non-trivial deployment: GPU HBM is hot and small, system RAM is warm and bigger, SSD is cold and effectively unbounded. cgn-kvcached materializes that hierarchy as one daemon with explicit tier latencies (sub-200 µs warm, sub-5 ms cold) and a QUIC peer-fetch path between nodes — so a cache miss on node A that lives in node B’s RAM does not become a recompute, it becomes a 12 ms cross-node fetch.

The federation piece is the part that surprised me on first read. Cognitora’s router can form a federation across clusters, not just nodes — meaning a hot prefix that exists in your Frankfurt region can serve a request that landed on your Virginia router, if the prefill cost amortized across the network round-trip beats recomputing locally. Most production stacks do not even attempt this; they treat each cluster as an island. Whether that capability is worth the operational complexity depends entirely on your traffic shape, and Cognitora makes the right call by leaving it off by default.

Separately, cgn-metrics solves a smaller but universal ops problem: in-cluster Prometheus federation. Configure scrape targets in TOML and the daemon unions every target’s /metrics text, injects a cgn_target="<name>" label on each line, and serves the combined exposition at /federate — one scrape for your central Prometheus, without parsing the full metric stream twice. That is observability plumbing, not cross-region routing; both use the word “federation” but they are different mechanisms.

Energy-aware scheduling

The bit of the design I like most aesthetically is also the one with the least proven impact: routing decisions can incorporate power telemetry from Redfish, IPMI, and DCGM. A GPU that is thermally throttled, or a node whose PSU is drawing closer to its budget than its neighbors, gets weighted down in admission control. The stated efficiency target — ≥ 1.4× over a round-robin baseline — is plausible on workloads where the cluster is power-limited rather than compute-limited, which is increasingly the situation in modern racks where power per rack-U is the binding constraint.

I would not buy a system because of energy-aware scheduling alone. I would treat it as a strong tiebreaker if it is otherwise the right shape — and it is one of the explicit gaps in Dynamo today, which doesn’t surface power telemetry into routing.

Cognitora vs NVIDIA Dynamo

Dynamo is the obvious comparison and the most capable alternative. The two projects share a lot of DNA — Rust core, KV-aware routing, NIXL-based disaggregation, Prometheus telemetry — and disagree on a small number of important things.

Reading that table honestly: if your workload is multimodal, video, or NVL72-shaped, pick Dynamo today. That is where NVIDIA’s investment is showing through. If your workload is text-only LLM serving on heterogeneous hardware (mix of H100 / L40S / older Ampere / on-prem llama.cpp / hosted API fallback), if you do not want a Python control plane in the hot path, if you care about cross-cluster federation, or if you operate a power-constrained rack and want telemetry to feed the scheduler — Cognitora is the closer fit.

The llama.cpp + OpenAI-compat line is the one I would emphasize most to anyone considering this for a real deployment. It changes what “the cluster” can include. A company-internal cluster that is allowed to burst to a hosted API during a traffic spike has a very different cost curve than a cluster that has to provision for peak.

Cognitora vs the rest of the field

Dynamo is the closest comparison; the broader field is worth a paragraph each because the alternatives genuinely have different jobs.

vLLM Production Stack is the most natural alternative if you are vLLM-only and Kubernetes-native. It ships a router, autoscaler, and observability stack tuned for vLLM. Cognitora is the right pick if “vLLM-only” is not a constraint you want to commit to — most production fleets end up running at least two engines (vLLM + SGLang for structured output, or vLLM + TRT-LLM for the largest models) and the multi-engine story is easier on Cognitora’s side.

Ray Serve is the right answer if you already run Ray, or if your inference workload is genuinely heterogeneous (LLM + classical ML + Python preprocessing + tool calls all in one DAG). Ray’s strength is composability across arbitrary Python workloads. Cognitora’s strength is being narrowly excellent at the LLM-serving slice — no Python in the data path, no Ray cluster to operate, no actor model to reason about.

KServe is the Kubernetes-native, model-serving-CRD answer. It is the right fit when “this is one of forty model deployments my platform team manages, and they all need to look the same in the cluster.” If LLM inference is the workload your platform team primarily exists to serve, the abstraction layer KServe imposes starts costing more than it saves.

NVIDIA Triton Inference Server is still excellent for non-LLM inference — vision, audio, classical models — and increasingly for LLMs via the TensorRT-LLM backend. If your fleet is mostly non-LLM with LLM as a side workload, Triton is the centerpiece. If LLM is the workload, the LLM-specific systems (Cognitora, Dynamo, vLLM Production Stack) are a better starting point because the things they specialize in — KV routing, prefill/decode disaggregation, prefix sharing — are not a thing Triton optimizes for at the platform level.

BentoML lives at a different altitude. It is excellent at “package this Python model + preprocessing + business logic into a deployable artifact.” It is not, and does not try to be, a multi-node KV-aware orchestrator. The two compose: BentoML for service packaging, Cognitora (or Dynamo) for cluster-level orchestration of LLM-specific concerns.

The honest summary is that LLM serving has bifurcated into two layers that used to be one. The lower layer is “given a request and a replica, generate tokens efficiently” — vLLM, SGLang, TRT-LLM, llama.cpp own this. The upper layer is “given a fleet, route requests so KV is reused and disaggregation pays off” — Dynamo and Cognitora are the two open-source projects that take this layer seriously as a standalone product. Generic ML serving stacks (KServe, Triton, Ray Serve, BentoML) cover the upper layer for general workloads but do not optimize for the LLM-specific signals that turn out to dominate cost.

Multi-model cascades

One smaller capability worth calling out because it has outsized cost impact: multi-model cascades with logprob gating. The idea is old — route easy queries to a small model, fall back to a large model only when the small one is uncertain — but the orchestrator has to support it natively or it becomes a Python-in-the-hot-path workaround. Cognitora exposes it as a routing policy:

small_model = "qwen3-7b"
large_model = "llama3-70b"
gate        = "logprob"     # escalate when small-model logprob < threshold

For workloads where ~70% of requests are genuinely simple (classification-shaped, lookup-shaped, short-answer chat), this cuts cost dramatically without touching tail quality. Dynamo has partial support; in Cognitora it is a first-class router policy.

Honest limits

A few things I would want to know before betting a production deployment on this:

• Pre-1.0. The OpenAI-compatible HTTP surface is stable; internal gRPC APIs and the TOML config surface may still shift in minor releases. Pin CGN_VERSION=v0.3.0 (or newer) on the install script and read the changelog.
• Helm chart maturity. The chart under deploy/kubernetes/helm/cognitora/ exists and helm lint passes in CI, but there is no published OCI chart at oci://ghcr.io/… yet—you install from a local chart path or use the quickstart manifest below until the chart ships optional engine sidecars and a simpler dev-default TLS story. Terraform modules for cloud VMs are still thin stubs; the credible cloud path today is bring your own cluster + quickstart or Helm from a git checkout.
• No multimodal/video. If your roadmap includes image generation or video diffusion serving, Dynamo is ahead. The Cognitora architecture has no in-principle obstacle here, but the engine integrations are not shipped today.
• Gang scheduling is basic. Node selectors, not Grove-style NVL72-aware co-scheduling. If you operate NVL72 racks and need topology-aware placement, Dynamo is the better fit until this lands.
• The performance numbers are targets, not benchmarks on your traffic. The architecture supports them; whether your specific workload realizes them depends on prefix sharing, request shape, and hardware mix. The right move on a new deployment is to A/B against round-robin on a slice of real traffic and measure. CI runs a soft perf gate (cargo bench on routing/prefix paths, non-blocking on PRs); hard regression gating is planned once baselines stabilize.
• The cross-cluster federation story is powerful and operationally heavy. Turn it on only when you actually have multi-region traffic that benefits from it. The defaults are sensibly conservative.

Try it

# Install — six static binaries, no runtime deps (pin the release)
curl -fsSL https://raw.githubusercontent.com/antonellof/cognitora-inference/main/deploy/installer/install.sh | CGN_VERSION=v0.3.0 sh

# Bring up Llama-3.1 8B on a single GPU with vLLM
bash recipes/llama3-8b/vllm/agg/up.sh

# Disaggregated prefill/decode on two GPUs in one node
bash recipes/llama3-8b/vllm/disagg-single-node/up.sh

# Llama-3.3 70B FP8 on 4×H100 with TP=4
HF_TOKEN=… bash recipes/llama3-70b/vllm/agg/up.sh

# Admin: inspect nodes / desired models in etcd (needs etcd endpoints in cognitora.toml)
cgn-ctl cluster nodes
cgn-ctl model ls

Kubernetes — fastest path to a public URL (CPU demo: TinyLlama via llama.cpp in-cluster; no GPU quota required). Validated on GKE Autopilot; same manifest works on other clouds or local clusters (use port-forward if LoadBalancer stays pending):

kubectl apply -f https://raw.githubusercontent.com/antonellof/cognitora-inference/main/deploy/kubernetes/quickstart/cognitora-cpu.yaml
kubectl -n cognitora wait --for=condition=ready pod -l app=cognitora --timeout=10m
IP=$(kubectl -n cognitora get svc cognitora-router -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
curl -sS "http://$IP/v1/chat/completions" \
  -H 'Content-Type: application/json' \
  -d '{"model":"tinyllama","messages":[{"role":"user","content":"What is 2+2?"}]}'

Kubernetes — Helm from a git checkout (production-shaped chart; wire your own engine / GPU pool—the chart assumes mTLS material unless you adjust values):

git clone https://github.com/antonellof/cognitora-inference.git && cd cognitora-inference
helm install cognitora ./deploy/kubernetes/helm/cognitora \
  --namespace cognitora --create-namespace \
  --set router.replicas=2 \
  --set models.llama3-70b.tp=4

An oci://ghcr.io/antonellof/charts/cognitora one-liner is not published yet; track it in the repo’s plan.md. Until then, local chart path or the quickstart manifest above.

From source (if you want to read the routing code, which I recommend — it is the most interesting part):

git clone https://github.com/antonellof/cognitora-inference.git
cd cognitora-inference
cargo build --release --no-default-features \
  -p cgn-router -p cgn-agent -p cgn-kvcached \
  -p cgn-metrics -p cgn-ctl -p cgn-operator

Once a router is up, point any OpenAI-compatible client at it. The wire protocol is the lingua franca, so existing application code does not change. Use /v1/embeddings only when the loaded model is an embedding model—chat checkpoints will correctly surface as errors through the stack rather than fake vectors.

Why this shape of system, now

The closing observation is meta. For two years the LLM-inference field has tolerated a stack where Python is in the request path, Kubernetes is the only first-class deployment target, and “the orchestrator” is whatever combination of Nginx, Redis, and homegrown schedulers a given team has glued together. That stack works at startup scale. It does not work at datacenter scale, where a 1% efficiency gain is worth more than a feature, where the difference between sub-500-µs and 5-ms routing decisions shows up as a line item, and where the operations team would prefer a single binary they can strace.

Cognitora is one answer to “what would the orchestrator look like if it were designed today, for that scale, in one language, with KV cache reuse as the centerpiece rather than an afterthought?” NVIDIA Dynamo is another. Both are credible; they make different bets on the runtime shape (six small Rust binaries vs Rust+Python), the deployment surface (bare-metal-first vs Kubernetes-first), and the engine ecosystem (broad including llama.cpp/OpenAI-compat vs the three industrial engines). Which one fits depends on what your fleet actually looks like — and the fact that there are two well-engineered open-source choices in this layer at all is a meaningful change from where the field was twelve months ago.

Links:

• Cognitora repository — Apache-2.0, Rust 1.89+
• CHANGELOG · v0.3.0 release
• NVIDIA Dynamo — the closest comparable system
• vLLM Production Stack — vLLM-only alternative
• NIXL — the disaggregation transport both projects build on
• DistServe — the original prefill/decode disaggregation paper
• Ray Serve · KServe · Triton Inference Server · BentoML — generic ML-serving alternatives