Try to Break It First
The Other Dimension introduced the four readiness components - ownership clarity, failure-mode awareness, confidence calibration, and recovery design. The Invisible Line focused on ownership. This post builds the first of the three that remained: failure-mode awareness - not as a testing checkbox, but as an organizational habit.
"It passed CI" is not the same as "we understand how it lies."
The product gate is how it fails, not how it demos
At Esco Micro, building an ultra-low temperature freezer with mechanical, electronics, and firmware engineers in the room, we learned a blunt lesson: a prototype that hits its spec sheet is not a product. The product begins when you have deliberately taken the system to places it does not want to go - thermal cycling, seal degradation, fault injection - and you know how it behaves when it misbehaves. Performance is a snapshot. Failure behavior is the story your customer lives in.
The product gate was never performance alone. It was validated confidence in how the system fails.
Generative systems hide the failure surface
Software teams talk about shift-left testing and chaos engineering. AI-augmented development adds a sharper version of the same question: the failure surface is partly unknown because the system is partly generative. If you have never tried to make the tool produce wrong outputs - confidently wrong, plausibly wrong - you do not know where the cliffs are. "It passed CI" is not the same as "we understand how it lies."
Automated tests catch regressions when you know what to assert. Failure-mode awareness is about building a shared mental model of the system outside the assertions - the behaviors that are not worth automating but are still catastrophic, the interactions that only show up under load, the classes of inputs that make the model sound authoritative when it should not. That is why the habit is organizational, not a QA ticket. Everyone who can ship has to care.
Physical consequence trains the habit early
At ION Mobility, the firmware path carried physical consequence. We did not treat "works on the bench" as equivalent to "safe on the road." The mental model we needed was not "find bugs" but map the boundary - what happens under load, under packet loss, under ambiguous sensor input? That habit is transferable to AI-generated pipelines: before you widen scope, you stress the path where the model touches money, safety, or irreversible data.
Plausible-but-wrong and tacit failure modes
At GoNetZero, the scary class of errors was not syntax. It was plausible-but-wrong - numbers that looked audit-ready but rested on a misread document, a shifted boundary condition, or a silent assumption in the chain. The organization that ships AI assistance into that domain needs a culture where people spend time trying to break the happy path before they bless the path as production. Not because people are adversarial by nature - because the failure mode that will hurt you is rarely the one that throws an exception.
In my institutional work at NUS, I have seen how failure modes stay tacit. A source system hard-deleted a record; downstream master data did not follow the expected path because the standard operating picture for that scenario lived in someone's head, not in a process the system could enforce. The failure mode was knowable in principle - it had happened before in human memory - but it was not institutionalized. That is low failure-mode awareness in another costume: the organization knew informally and learned too late.
The workflow is part of the attack surface
Security teams have a vocabulary for this - threat modeling, abuse cases - but the same discipline applies to ordinary wrongness when AI is involved. Prompt injection, supply-chain suggestions, and dependency confusion are not exotic when agents read repositories and comments as instructions. Failure-mode awareness for AI-augmented development includes asking: what is the attack surface of the workflow - not only the network perimeter, but the places where a human could paste production data into a tool that should not see it, or where an agent could interpret a string as a command. If you have not asked those questions in a room with the people who ship, you are discovering failure modes in production traffic.
What to do before you call it production
Pre-deployment failure hour
First, run a pre-deployment failure scenario for anything where an AI touches a consequential output - not a thirty-minute security theater, a real hour where the team's job is to make the system fail in interesting ways. Can you get a confident summary that inverts the meaning of the source? Can you get two incompatible answers from two phrasings of the same request? If you cannot break it, that is not always good news - it may mean you have not tried hard enough.
Integrations and seams
Bring the same mindset to integrations. Agents do not fail only inside the IDE; they fail at boundaries - where retrieved context is incomplete, where an API's semantics changed, where two tools disagree. Failure-mode awareness means mapping those seams explicitly: what happens when retrieval returns nothing useful, when the tool times out, when the human accepts a suggestion without reading it? Those are not edge cases in a mature workflow - they are the center of mass for operational surprise.
Plausible-but-wrong as a first-class test
Second, treat plausible-but-wrong as a first-class test category. The dangerous output is rarely gibberish. It is the email that almost matches policy, the refactor that almost preserves behavior, the figure that almost matches expectations. Your review discipline has to include "would I bet my reputation on this being right?" - not "did the tests pass?"
Document the failure surface before deploy
Third, document the failure surface before deployment - not only the bugs you find after launch. What modes did you surface in rehearsal? What inputs caused degradation? What is explicitly out of scope? That document is the beginning of institutional memory. Without it, every incident feels novel.
None of this replaces ownership. The Invisible Line argued that someone has to stand behind the system - failure-mode work is how that person earns the right to stand there without bluffing. It also pairs with calibration: the next post discusses confidence when the system is not failing obviously - the hardest case.
Failure-mode awareness tells you where the system can go wrong. The next post asks a different question: when it is working, how confident should you be in what it produces - and how do you communicate that confidence to someone who was not in the room when it was built?
The Other Dimension · Part 3 of 6 · Previous: The Other Dimension · Next: How Confident Should You Be?