A spec I had written carried a performance target: first token in 1.5 seconds. It looked like the sort of number a spec should contain. Specific, testable, ambitious but not absurd.
Measured on the machine the thing actually runs on, cold, the real figure was somewhere between sixty and a hundred and ten seconds.
Not fifty per cent out. Not double. Two orders of magnitude, in a number I had written down myself and then reasoned from for weeks.
Where a number like that comes from
I didn't invent 1.5 seconds arbitrarily. It's a defensible target for the class of thing being built, it's the kind of latency users tolerate, and it's roughly what you would expect from a hosted service doing similar work.
That's exactly the problem. It was a number imported from a similar-sounding context, not a measurement of this one. Local inference on a laptop, loading a model into memory from cold, isn't a hosted service, and no amount of the target being reasonable in general made it reasonable here.
Once measured, the number wasn't the only thing that changed. A memory cap in the same spec, four gigabytes, turned out to be enforcing something the underlying runtime doesn't implement. There's no such cap to set. That target wasn't wrong so much as unimplementable, and it survived review because it read like a sensible constraint. It was replaced with a calculated budget derived from how the runtime actually allocates, which is a less tidy number and an actual one.
A third figure, how many rows fit inside a context budget, was simply miscounted. Not wrong in kind, just wrong, and wrong in the direction that made the design look more comfortable than it was.
The acceptance criteria were worse
Wrong targets can at least be measured. The acceptance criteria included things that could never have passed.
One criterion required zero outgoing HTTP requests. The intent is clear and good: this thing shouldn't phone home. But taken literally it's unsatisfiable, because the application talks to a local inference server over HTTP on loopback. Any correct implementation fails the criterion as written. So either you fail forever, or you quietly reinterpret it, and quietly reinterpreting acceptance criteria is how they stop meaning anything. It became "no request to a non-loopback host", which is the thing I actually cared about and is checkable.
Another required a full-text search test that couldn't pass, because the search engine matches on AND across terms and the example in the spec assumed OR. The criterion was describing a behaviour the engine doesn't have, using an example that demonstrated the misunderstanding. Fixing it meant fixing my model of the engine first, and that turned into a real design change: search moved to quoted, OR-joined terms ranked by relevance, because recall is what a question needs and exact conjunction isn't.
Both unsatisfiable criteria were mismatches between the spec and reality that hadn't surfaced yet. The criterion is the first place the mismatch becomes visible, and it's tempting to file the edges off rather than ask what it's telling you.
Two things marked satisfied that weren't
The same pass turned up a database journal mode recorded as already configured, which wasn't actually in effect. Worse, the setting I had credited with protecting reads during background writes turns out not to be the one doing that job at all. Two errors stacked: the state was wrong, and my model of why it mattered was wrong.
And two guard tests, both green, both protecting nothing, because the values they inserted to represent "bad input" were tokenised identically to the values representing good input. They asserted a difference that the engine couldn't express. I deleted them rather than repair them, on the grounds that a test asserting nothing is worse than no test: it occupies the space where a real test would go and reports success while doing it.
What I do differently now
Before a number goes into a spec, I try to say where it came from. Measured on what, or taken from where. If the answer is "it seemed about right", it's a hypothesis and should be labelled as one, because a number in a spec gets treated as a constraint by everyone downstream, including me a fortnight later.
And when an acceptance criterion can't pass, my first move is no longer to soften it. It's to ask whether it has found a real gap between what the spec assumes and how the system behaves. Twice out of two, in this case, it had.
The part I haven't solved is the review problem. Every one of these numbers went past me more than once and read as fine each time, because plausible numbers don't attract scrutiny. The only thing that caught them was going and measuring, which is exactly the work a specification is supposed to let you defer. I don't know how to get the deferral without the risk.