Research / Method

How we judge results, and what we got wrong

A system that bluffs can look excellent on accuracy alone. So we report what matters first: when it was wrong, separately from when it declined to answer.

The rules we hold ourselves to

WRONG is reported apart from refused. A refusal is an acceptable answer; a stated falsehood is not. The CI check fails the build if the scorecard's WRONG column rises above 0.0%.
Never silently wrong. A guess may be wrong, if it is labelled as a guess and its track record is stated. A fact stated and false is the failure that matters most.
Adversarial questions after every new capability. We build the harder test before the feature where we can, and keep batteries of must-refuse questions in the test suite.
Deceptive cases reported apart. Kinds of lesson built to look like one rule and secretly change behaviour are never folded into the headline.
Estimates are labelled as estimates. Energy is counted and priced, not metered, until it is metered.
Mistakes and rejected hypotheses are written down as they happen, in the repository's experiment log.

Standing caveats

The lessons are synthetic and tidy: made-up words, small numbers, a handful of phrasings.
The mechanisms, the expression language and the thresholds are ours, chosen by hand. The system selects among them; it did not discover them.
The experiments use a simulated teacher that is always right.
"0% wrong" holds on in-style questions plus our must-refuse batteries. It is a spot check, not a proof.
Some constants rest on thin evidence. The 0.8 floor of the reliability bar rests on one 72% failure and six clean seeds.

A list of mistakes

Each of these was found by looking at the cases behind a number, not by reasoning about it. Most of our first explanations were wrong; the experiment log says so.

What went wrong	How it was found	Outcome
The same ambiguity bit four times. "how far apart", "minus" and "ten minus n" agree on every lesson taught, so "three minus five" was answered as a distance.	Asking adversarial questions the scorecard had not asked.	Ambiguity is now defined by behaviour (expressions that differ somewhere), not by taught examples; operand order is kept; merges are checked per group.
A bluff the scorecard missed. "what is the code for five" was answered with the code for "one".	Asking for every code from zero to ten.	The word that fills the blank must now match.
Inflated cost numbers. A lesson's identity was its sorted words, so "five minus two" and "two minus five" were the same lesson; 39 of 60 lessons were wrongly "corrected", costing 93 ms each.	Counting what was kept versus skipped.	Identity is words in order; the same lessons cost 0.18 ms.
A coincidence explained a tens digit (60 wrong answers), and primality was "explained" by chance (a multiple-comparisons problem: search enough expressions and one fits).	The invention battery's harder kinds.	A uniqueness rule, and a check that the lessons carry more bits than the search spent.
A rule that changes regime outside what was taught (a sum that caps or wraps) is exact on all lessons and answered confidently wrong: 53 wrong answers per seed.	Building the deceptive kinds before building the feature.	Careful mode, later the default, plus labelled guesses.
A box-shaped "reach" let two confirmed edges vouch for an unchecked corner: 4 wrong facts of 9,309.	The calibration run, listing the wrong facts.	The reach is now a staircase. It costs some coverage (never-taught 96.9% to 94.9% at 60 units), measured and stated.
A correction widened the reach of the rule it had just contradicted. Two wrong facts after one correction.	The baseline of the correction experiment.	Where a rule is refuted, what the teacher said wins.
A cross-validation flaw let memory peek at swapped twins of commutative facts.	While building the scale test.	Facts that share a bag of words now stay in one fold.
A library rotation bug. The function library held 8 entries for about 14 program units; each refit evicted another, and entries grew into nested equivalents.	Logging every library addition and eviction after a third round made things worse.	A unit contributes a given behaviour once. My first guess at the cause was a different, real, smaller bug.
Flipping the default made a battery read "0 of 32 worked out" with no failure anywhere: it graded beyond-range questions that careful mode now refuses.	A number that looked wrong.	The capability batteries now run permissive explicitly and say so.
A commit was pushed with a failing test, because a command chain did not gate on the test result.	The next run.	Commits are now gated on the tests passing.

Dead ends are recorded too. For example, the search budget cannot be cut by watching partial progress, because doomed and deep searches look the same; and an early version of the rule-versus-memory cross-check, which treated a neighbouring lesson as a witness, cut the clean scorecard's never-taught score from 95.1% to 75.6% before the scorecard caught it. The complete record, with the numbers, is the experiment log.