~building a 1-bit llm

A series about making a language model as small and as stupid as possible, then making it very fast. Ternary weights, 2.31 MB, ~1,045 tokens a second in a browser tab.

You can play with the thing itself while you wait for the rest of these.

parts

• ·what one bit actually buys you2026-08-20
• ·the token floor — why every interesting dataset was too smallsoon
• ·seven models instead of one — making every layer earn its placesoon
• ·one line of .detach() and why the whole thing workssoon
• ·11 million parameters into 2.31 megabytessoon
• ·20 to 1,045 tokens a second, and two wrong guessessoon

why bother

Because I wanted to know what is actually inside one of these, and reading a finished implementation teaches you nothing — correct code looks inevitable. So I built it seven times, each version one component bigger, and measured what each component was worth. Attention turned out to be 37% of everything gained, for 3.7% of the parameters. Normalisation — 2,560 numbers, 0.1% of the model — was worth more than the entire feed-forward network.