2026-08-20building a 1-bit llm, part 1

What one bit actually buys you

A 1-bit LLM does not store 1 bit per weight. It stores three states — −1, 0, +1 — which is log₂(3) ≈ 1.58 bits, and the name stuck anyway. Activations stay at 8 bits, because language models produce enormous outliers at a handful of positions and low-bit weights interact badly with them.

The reason to care is not compression. It is that a normal matrix multiply is a pile of multiply-accumulates, and when every weight is −1, 0 or +1, the multiplication disappears entirely:

• w = +1 → add the input
• w = −1 → subtract it
• w = 0 → skip

The entire forward pass becomes integer accumulation. On a GPU this barely helps — GPUs are drowning in multipliers. In silicon, a multiplier costs roughly ten to twenty times the area and energy of an adder. So 1-bit LLMs are an argument about hardware that happens to be expressible in PyTorch.

I found that out the hard way. My packed ternary model runs 1.7× slower than full precision on a T4, and 97% of that overhead is activation quantisation, not the weights. The multiplier elimination — the whole thesis — addresses 3% of the actual cost when you implement it in a framework built for float matmuls.

The comparison people quote is also the wrong one. At equal parameter count ternary loses: my 11M-parameter model scored 0.12 nats worse than the identical architecture in fp32. That is unavoidable — 1.58 bits carries less information than 32.

But nothing deploys on a parameter budget. Things deploy on a byte budget, and 2.31 MB buys you either 11.1 million ternary parameters or 1.1 million fp16 ones. At equal memory, ternary wins by about 0.17 nats — consistently, across a 2× range of budgets. Report only the first comparison and you have understated it; report only the second and you have oversold it. The pair is the finding.

One more thing, which is the most useful number I have from any of this. There are two ways to get ternary weights: quantise during training, or quantise a finished model afterwards. Same architecture, same weights, same forward pass — the only difference is when.

Quantised during training: 2.18 nats. Quantised afterwards: 5.02. That gap is 2.85 nats and a 17× jump in perplexity, from scheduling alone. A post-quantised eight-layer transformer scores worse than a single full-precision attention layer.

The reason is measurable. Ternarising trained weights changes every weight matrix by 53%, while leaving 88% of its direction intact. Each matrix keeps most of what it was pointing at — and composed through 48 matrices across 8 layers, the model keeps 25% of what it learned. Training through the quantiser lets the network adapt around it instead of having it imposed on a solution that assumed precision it no longer has.

Next: why every genuinely interesting dataset I wanted to use turned out to be too small, and the number I should have checked first.