A 27B model in 6 GB: Bonsai ternary on Ferrox, locally

Ferrox is a pure-Rust inference engine for GGUF models, with a llama.cpp-shaped CLI, an OpenAI-compatible server, and a small web UI called Studio. The first post covers the design, the second the Metal backend.

This one is about a single model: Ternary-Bonsai-2-27B from PrismML. 27 billion parameters at 1.75 bits each, 5.95 GB on disk, running on a 16 GB laptop with room to spare.

What Bonsai is

Every language weight is one of three values, -1, 0 or +1, with one 16-bit scale per group of 128. Five trits pack into a byte in base 3, which is where 1.75 bits per weight comes from; that packing is the GGUF type PTQ1_0.

Ternary alone would wreck a 27B model. What makes it work is a rotation: each weight matrix is transformed blockwise by a Walsh-Hadamard matrix with fixed signs, which spreads outliers across a block so a three-level grid can hold them. The rotation is folded into the stored weights, so the runtime has to apply the matching transform to the activations before every matmul, and undo it on the embedding table after each lookup. Get that wrong and the model loads and talks nonsense.

Ferrox v0.23.1 adds the packing (a CPU dot, a Metal matvec, a Metal GEMM) and the fold, read from the checkpoint’s own prism.hadamard.* metadata. Anything outside the configuration that has been verified is refused by name rather than guessed at.

PrismML’s llama.cpp fork is the reference implementation for the format, so that is what Ferrox is checked against. ferrox parity feeds both engines identical token ids and compares the full first-token logit distribution: a KL divergence of 2e-5 on CPU, 2e-5 on the Metal decode kernel and 2e-6 through the Metal prefill GEMM, with the same ten top tokens in the same order. The tokenizer matches on 1198 tokens across 21 test strings.

Download and run

You need Rust and, on a Mac, nothing else. The only build flag is your GPU.

cargo install ferrox-cli --features metal      # or --features cuda

# Same argument shape as `hf download`, no Python.
ferrox download prism-ml/Ternary-Bonsai-2-27B-gguf \
  Ternary-Bonsai-2-27B-PTQ1_0.gguf --local-dir models

# Chat. Ferrox applies the checkpoint's own template.
ferrox -m models/Ternary-Bonsai-2-27B-PTQ1_0.gguf \
  -p "Explain ternary quantization in three sentences" -n 400 -ngl 99

On an M2 Pro it decodes at about 7.7 tokens per second: reading speed rather than skimming speed, with most of the machine’s memory still free.

The server and Studio

ferrox serve -m models/Ternary-Bonsai-2-27B-PTQ1_0.gguf \
  -ngl 99 --alias bonsai-2-27b --port 8383

That is an OpenAI-compatible API on http://127.0.0.1:8383/v1: chat completions, completions, embeddings and models, with Anthropic Messages and Responses on the same port. --alias names the model in /v1/models and in every response; without it you get the checkpoint’s own general.name, which for this file is the unhelpful string Hf.

Note what is missing from that command: -c. With no context flag the server prices the checkpoint against the device and derives both the per-request ceiling and the KV block budget from the same arithmetic, so the context it advertises is one it can actually serve.

Studio is a separate app that talks to that API over HTTP. From a checkout:

cd ui && npm install && npm run dev   # http://localhost:5173/ui/

Its Models page lists every GGUF in your models directory with its quant, architecture, context and size, and loads one without restarting the server:

And a chat, at the real speed:

The finished turn carries its own numbers underneath: time to first token, prefill and decode rates.

The model thinks before it answers. Studio folds the thinking into a collapsible block; over the API it arrives in reasoning_content, separated from the answer, and reasoning_effort: "medium" shortens it while "none" turns it off.

Point a coding agent at it

Any tool that speaks the OpenAI API can use the server. Here is Pi, the minimal coding agent I covered in an earlier post: four tools, one loop, one provider file.

npm install -g --ignore-scripts @earendil-works/pi-coding-agent

Check the server answers first, using the id you gave --alias:

curl http://127.0.0.1:8383/v1/models

Then add Ferrox as a provider in ~/.pi/agent/models.json:

{
  "providers": {
    "ferrox": {
      "baseUrl": "http://127.0.0.1:8383/v1",
      "api": "openai-completions",
      "apiKey": "ferrox",
      "models": [
        {
          "id": "bonsai-2-27b",
          "name": "Bonsai 2 27B ternary, local via Ferrox",
          "contextWindow": 16384
        }
      ]
    }
  }
}

Ferrox does not validate the key, but Pi wants a non-empty one. Then cd into a repository, run pi, pick the Ferrox entry with /model, and start with something small and checkable. Two practical notes: keep max_tokens generous, because a thinking model spends part of the budget before it writes any code, and a few tokens per second suits reviewing each step rather than firing and forgetting.

Links

• Ferrox on GitHub, v0.23.1 release
• Ternary-Bonsai-2-27B GGUF and PrismML’s llama.cpp fork
• Pi coding agent

AI full disclosure

This software is developed with strong assistance from Cursor, Grok 4.5, GPT 5.6, and Claude Fable 5, with humans leading the ideas, testing, and debugging. We say this openly because it shaped how the project was built. If you are not happy with AI-developed code, this software is not for you.

Acknowledgements

Ferrox does not link against GGML, but exists thanks to the path opened by the llama.cpp project and the kernels, quantization formats, GGUF ecosystem, and hard-won engineering knowledge developed there. The ternary format, the Hadamard fold and the model itself are PrismML’s, and the Metal kernel in this release is a port of the design in their fork. We keep the GGML authors’ copyright notice in docs/THIRD_PARTY_NOTICES.md.