Research / Efficiency
Efficiency, measured in joules
Parameter count says how big a model is, not what it costs to use or to teach. The measure we care about is capability per unit of computation, as the machine itself measures it. So every result here comes with a cost.
How we count
The code has a ledger that counts every multiply-add and every 4-byte memory access. It prices them with the widely cited 45 nm figures from Horowitz (ISSCC 2014): about 3.7 pJ for a 32-bit float multiply, 0.9 pJ for an add, and 640 pJ for a 32-bit read from DRAM. A modern chip spends less in absolute terms, so treat the joules as an estimate and trust the ratios.
What counting showed early on, with the first small neural network: answering one question cost about 14,000 multiply-adds, and about 91% of the estimated energy was moving memory, not arithmetic. Even a 63 KB network is a data-movement problem. A 7-billion-parameter model does roughly 500,000× more arithmetic per token and cannot keep its weights in cache at all.
The same counting showed that implementation is a real lever: writing a dot product as eight independent sums instead of one let the compiler vectorise it with no unsafe code and no special flags, taking a question from 32 µs to 13 µs with identical answers.
Answering: fewer operations, same answers
Using the mathematics of the computation, the prototype now finds the right kind of lesson with a precomputed table of how telling each word's shape is for each unit, instead of comparing the question with every lesson it was ever taught. Remembered lessons are found through an index of their content words, and networks are no longer trained where cheaper mechanisms already work.
Caveat
Honesty about this figure. These are counted operations, not a meter. The "slow way" is our own first design, which compared against everything, so a saving of this size comes from an inefficient baseline. The reach check that guards extrapolation is not yet counted in the 57. "One small network" is our earlier single network, not a clever alternative.
Learning: light enough for one laptop core
The claim worth testing is that learning should not need a data centre. On one CPU core of an ordinary laptop, with no GPU:
| What | Cost |
|---|---|
| Teach 792 lessons in 13 units, from nothing | 0.34 s (was 3.8 s), about 12 MB peak |
| A lesson of a kind it already understands | 0.19 ms, not stored |
| A new way of saying something it knows | 0.4 ms |
| A new kind of lesson, no unit named | 1 ms (a unit opens at the sixth) |
| Fit a rule: exact solve vs 800-pass gradient descent | 392× fewer operations, about 60× faster |
| Scale test: 25,776 lessons, 600 units, 10,000-word vocabulary | 2.65 s, 56 MB peak |
| Learn one more fact, from 20 to 600 units | 0.4 to 0.6 ms at every size |
| Reorganise (merge and split) at 600 units | 1.35 s |
Teaching dropped from 3.8 s to 0.34 s mostly because networks stopped being trained for plain tables, and the scale test only stayed flat after two fixes: a per-lesson table that was rebuilt from scratch each time (about 40 ms at 200 units) became incremental, and routing became sparse.
What we have not done yet
- Real joules. The CPU's own energy counter is now readable on the development machine, but it is not yet wired into the ledger. Everything above is time, memory and counted operations.
- The controlled comparison. A matched small transformer against
baby-style systems, at three or more sizes, on quality at equal joules, retention after new learning, adaptation speed and scaling slope. A rented GPU is planned for it. It is the experiment that would say whether any of this matters beyond tidy tasks. - Real text. The numbers come from a world of small numbers and a handful of phrasings. Nothing here says how cost grows on millions of real sentences.
The full derivations and the mistakes made along the way are in the repository's experiment log.