Two background jobs that marked themselves done too early
Two background jobs stopped early and recorded themselves as complete, and once recorded, nothing in the system could reopen them.
26notes from production work
I build software end to end. I'm Scott Mallinson, a software engineer at Cognizant in Barcelona, Spain. I write about API design, type systems, guardrails for AI features, and the maintenance that keeps a codebase navigable.
Two background jobs stopped early and recorded themselves as complete, and once recorded, nothing in the system could reopen them.
Chief's daily brief runs on a small local model. Getting it to behave meant moving a bullet-count limit out of the prompt and into code, then measuring what a repeat penalty setting actually cost in accuracy.
Two defects were cancelling each other out. Fixing the real one on its own would have woken a dormant guard with the wrong logic and made the page worse.
The tests for a desktop app's auto-scrolling passed the whole time the scrolling was broken: jsdom has no layout engine, so every height it reports is zero.
Validating user input is the most familiar guardrail and the smallest part of the problem. The model's own output, what it hands to tools, and what gets rendered afterwards all need the same fail-closed treatment.
For an AI feature with guardrails, releasing it carefully requires two disciplines: coordinating multi-service promotions with evidence at each step, and keeping the validation environment faithful enough that human feedback can be trusted.