The system can work and still be wrong

This morning, during my regular workday, my boss called me into a meeting to talk about verification and validation.

It is one of those distinctions I have known for years. Anyone who has spent enough time around software engineering, systems engineering, or quality assurance has probably encountered some version of it:

Verification asks whether we built the system right. Validation asks whether we built the right system.

I knew the definition.

What I had forgotten was how consequential the difference becomes when software can now be produced at a speed that would have seemed absurd only a few years ago.

During the conversation, my boss told us two stories. One involved artificial intelligence and the U.S. Army. The other went much further back, to work he had done around the Space Shuttle program after Challenger.

They appeared to be completely different stories.

They were actually about the same problem.

A system can work exactly as designed and still be wrong.

And AI is about to make that problem much more important.

01 The Dangerous Kind of Correct

The first story involved an Army selection process.

Every year, senior officers had to evaluate a large number of candidates for leadership positions. The process involved assessments, interviews, reports from peers and subordinates, cognitive evaluations, and other written material.

It was expensive in the currency organizations tend to underestimate most: human attention. The entire process could take six to eight weeks.

AI presented an obvious opportunity. Train a model using the existing process, let it evaluate the candidates, and compare its decisions with the historical human process.

The result looked remarkable. What had taken weeks could now be completed in hours. More importantly, the results appeared to match the existing process at approximately 96 percent.

From an engineering perspective, that sounds like success. The inputs went in. The expected outputs came out. The model reproduced the behavior it had been trained to reproduce. Verification passed.

And then there were thirty officers.

They had effectively disappeared from the selection.

Initially, one could imagine all kinds of explanations. Perhaps these were borderline candidates. Perhaps the human appeal process was allowing personal relationships or organizational politics to override the algorithm.

That was not what had happened.

The model had learned something nobody intended it to learn. The written evaluations used for the selection contained organizational language patterns. Officers from the intelligence community described performance differently from officers in infantry. Their reports followed different narrative conventions because the communities had developed different ways of writing about achievement and leadership.

The model had largely learned the patterns associated with one of those communities. It was not rejecting the other officers because they were unqualified. It was rejecting them because they spoke the wrong organizational dialect.

And here is the uncomfortable part:

The model was working.

It was doing what it had been trained to do. The failure was not necessarily in execution. The failure was in what execution had been asked to represent.

02 Building the System Right

This is where an old engineering distinction suddenly becomes much less academic.

Verification asks: did we build the system right? Does the implementation conform to the specification? Given these inputs, does it produce the outputs the design calls for? Does the component behave as expected? Do the interfaces satisfy their contracts? Do the tests pass?

This is familiar territory for software engineering. We have spent decades getting better at it. Unit testing verifies increasingly small pieces of behavior. Integration testing verifies interactions. Static analysis finds structural problems. Architecture reviews identify deviations from intended design. CI pipelines continuously determine whether changes violate known expectations.

Specification-driven development pushes this even further. If we can make intent sufficiently explicit, AI can generate implementations and we can evaluate those implementations against the specification. There is enormous value in that.

But there is also a dangerous assumption hidden inside it:

That the specification deserves to be satisfied.

Verification cannot answer that question. A perfectly verified implementation of the wrong specification is still the wrong system. In fact, it may be worse. It is the wrong system implemented exceptionally well.

03 Building the Right System

Validation asks a different question: did we build the right system? Does it serve the need for which it exists? Does it work in the real world, under real conditions?

This sounds similar to verification until something passes one and fails the other. The Army system makes the distinction visible.

Imagine that every automated test passes. Imagine that the model behaves consistently. Imagine that its results closely reproduce historical outcomes. Imagine that the architecture is sound, the pipeline reliable, and the system performs its task thousands of times faster than the previous process.

We could accumulate an impressive amount of evidence demonstrating that the system works according to its design. And still be excluding the wrong people.

That is not a software defect in the conventional sense. It is something more uncomfortable:

The implementation may be correct while our representation of reality is wrong.

This is why validation cannot simply be another testing stage at the end of development. Validation is reality's opportunity to disagree with the specification.

04 Making the Wrong Thing Faster

There was another observation from the conversation that stayed with me.

When organizations adopt AI, the obvious benchmark is usually what they already do. That makes sense. If a process currently requires eight weeks and AI completes it in two hours, we compare the two. If humans produce a document and AI produces the same document, we compare them. If developers implement a feature in two weeks and an agent implements it in twenty minutes, we compare the resulting software.

We need a baseline. The problem begins when the baseline quietly becomes our definition of correctness.

A historical process tells us how an organization solved a problem before AI. It does not prove that it was the best way to solve the problem. Organizations contain years of accumulated compromises. Processes exist because of old technical limitations, staffing constraints, organizational boundaries, regulatory interpretations, forgotten incidents, and sometimes simply because nobody has had enough reason to change them.

Humans learned to work around those imperfections. Then AI arrives. And we automate them.

The result can be spectacular efficiency applied to something that should have been reconsidered first. AI can make a good process faster. It can also make a mediocre process faster. It can make a biased process faster. It can make an unnecessary process faster. It can make the wrong decision faster.

Speed does not transform a bad process into a good one.

Sometimes it only removes the friction that previously limited how quickly the mistake could propagate.

05 The Things We Cannot Measure

The second story came from a completely different era of computing.

After the Challenger disaster, my boss worked as a software developer on a system related to detecting conditions associated with O-ring leaks. There was an immediate engineering problem. You could not simply attach a useful monitor to the O-ring and ask the software whether it was leaking. The thing they needed to know was not directly observable in the way they needed.

So the engineers asked a different question. If an O-ring were leaking, what else would happen? Fuel and gases would behave differently. Pressure relationships would change. Chemical depletion rates and mixtures elsewhere in the system would deviate from what should normally occur as velocity increased and fuel burned.

Those things could be measured. The software therefore did not need to observe the O-ring directly. It could observe the consequences of an O-ring behaving incorrectly.

That idea has stayed with me since the conversation ended. Because it describes something much larger than instrumentation on a spacecraft.

There will always be things in complex systems that we cannot measure directly.

The question is whether we can identify the evidence those things should leave behind.

06 Evidence Is Part of the Design

Software engineers often treat observability as something added after the important design decisions have already been made. Logs. Metrics. Traces. Dashboards. Useful operational machinery.

But the Challenger story suggests a more fundamental interpretation. Evidence itself can be designed. If we claim that a system possesses some property, we should ask: if this claim is true, what else should we be able to observe?

That question changes how we think about verification. It also changes how we think about validation. Verification requires evidence that the implementation corresponds to the specification. Validation requires evidence that the specification corresponds to reality.

Those are not necessarily the same evidence. And they should not be.

If we validate a new system exclusively using the same assumptions that produced its specification, we have created a closed epistemological loop. The system tells us it is correct because it behaves according to the definition of correctness we gave it.

That proves conformity. It does not prove truth.

07 The Old V Has Something to Teach Us

Systems engineering has represented this distinction for decades through the V-Model.

On the left side, we progressively decompose the problem: requirements, system analysis, software design, module design, eventually code. On the other side, we climb back toward the original intent: unit testing, integration testing, system testing, acceptance testing.

The lower levels ask increasingly technical questions. Does this module behave correctly? Do these components interact correctly? Does the integrated system behave according to its design?

As we climb toward the top, however, the question changes. We eventually return to where everything began.

Does this thing actually solve the problem?

That symmetry is more important in an AI-driven engineering environment than it first appears. AI is rapidly making the descent easier. Requirements can become specifications. Specifications can become designs. Designs can become code. Code can become tests. Agents can move through enormous amounts of that work faster than any engineering organization composed entirely of humans.

But accelerating the journey down the left side of the V does not automatically solve the journey back up. If anything, it makes it more important. Because the faster we can transform intent into implementation, the faster an incorrect assumption can become a functioning system.

08 I Saw a Smaller Version of This Yesterday

The conversation also made me think about something I had been working on the day before.

I was reviewing one of our AI-assisted code analysis skills. Its job was to inspect a codebase and produce a set of improvement priorities.

Technically, the work was good. The information was there. The analysis was detailed. The output was structured. And when I tried to use the resulting document, I realized I could barely make a decision from it.

The AI had succeeded at producing the artifact. It had failed at producing an artifact for a human.

So I changed the way the information was organized. Not because the underlying analysis was necessarily wrong, but because correctness was insufficient. The document needed to help someone classify evidence, understand priority, compare alternatives, and decide what should happen next.

That experience seems trivial compared with selecting Army officers or detecting failures in spacecraft. But structurally it is the same problem.

A correct output is not necessarily a useful output.

If a human remains responsible for judgment, then human comprehensibility is not cosmetic formatting. It is part of the system requirement.

09 The Purpose of an Artifact Is Not to Exist

This matters increasingly as AI begins producing engineering artifacts at scale. Requirements. Specifications. Architecture documents. Risk analyses. Test plans. Traceability matrices. Code reviews. Improvement candidates.

An AI system can generate all of them. Soon the difficult problem will not be generating them. The difficult problem will be deciding what deserves human attention.

A 900-line analysis that contains every relevant fact may be technically superior to a 50-line analysis. But if the person responsible for making the decision cannot identify the important evidence inside it, the larger document may be operationally worse.

This leads to a principle I think we will need to take much more seriously:

The purpose of an AI-generated artifact is not to exist. Its purpose is to enable the next decision.

That changes what quality means. Completeness matters. Correctness matters. Traceability matters. But so do comprehensibility, prioritization, uncertainty, and the ability to expose disagreement. AI should not merely produce more information for humans to review. It should help humans understand where their judgment matters.

10 Human in the Loop Is Not Enough

We often respond to AI uncertainty with a reassuring phrase: human in the loop.

But that description is incomplete. Where in the loop? Doing what? Reviewing everything? Approving everything? Repeating the analysis the machine just performed?

If AI produces ten times more work and humans must manually review ten times more output, we have not created an intelligent engineering system. We have created a faster producer feeding the same human bottleneck.

The more useful question is:

Where does human judgment create information that the machine cannot obtain simply by executing the specification?

That tends to happen around uncertainty. Exceptions. Contradictory evidence. Unexpected consequences. Weak assumptions. Novel conditions. And places where reality disagrees with what we believed reality would do.

The thirty officers matter precisely because they represent disagreement. The unusual sensor relationships around a leaking O-ring matter because they reveal something that cannot be observed directly. The difficult-to-read engineering report matters because a human cannot make the decision the artifact supposedly exists to support.

These are not inconveniences around the system. They are where learning happens.

11 From Pipeline to Learning Loop

For years, software development has often been represented as some variation of a pipeline:

Requirements → Design → Implementation → Testing → Release

AI makes that model increasingly inadequate. A more useful representation may look something like this:

Intent → Specification → AI Execution → Evidence → Verification → Validation → Human Judgment

But even that is incomplete. Because validation can tell us that our specification was wrong. Human judgment can expose an assumption we did not know we had. Operational evidence can reveal a condition nobody anticipated.

Those discoveries have to travel backward. So the process closes:

Intent → Specification → Execution → Evidence → Verification → Validation → Human Judgment → Learning → Refined Specification

And then it happens again. This is not merely human-in-the-loop. It is a continuous learning loop between specification and reality.

12 AI Changes Where Engineering Value Lives

For most of software history, implementation was expensive. Turning an idea into reliable software required enormous amounts of human effort. So engineering organizations optimized around implementation. Frameworks reduced boilerplate. Cloud platforms reduced infrastructure work. DevOps reduced deployment friction. CI/CD reduced integration cost.

AI is another step in that progression, but potentially a much larger one. If implementation becomes dramatically cheaper, the bottleneck moves. The scarce resource is no longer simply the ability to produce code. It becomes the ability to decide what should be built, express that intent precisely, identify meaningful evidence, recognize when reality contradicts our assumptions, and decide what to change when it does.

AI makes execution cheaper. It does not make judgment cheaper.

It may make judgment more valuable.

13 Reality Gets the Last Word

There is something appealing about specifications. They create certainty. We can inspect them. Version them. Test against them. Trace requirements through architecture into implementation.

For AI-driven software engineering, I believe specifications will become even more important because machines need explicit representations of intent if we expect them to act autonomously.

But specifications have a dangerous property. They can be wrong. And no amount of implementation quality can repair an incorrect understanding of the problem.

That is why verification and validation must remain separate. Verification asks whether reality inside the machine conforms to our specification. Validation asks whether our specification survives contact with reality outside the machine. Evidence connects the two. Human judgment decides what to do when they disagree.

This morning's conversation reminded me of something I had known for years but had allowed to become an engineering definition rather than an engineering principle.

We should absolutely make AI better at building systems right. We should make specifications precise, implementations traceable, tests executable, evidence observable, and verification increasingly automated.

But none of that removes the question that matters most.

Did we build the right system?

Because the most dangerous AI system may not be the one that fails. It may be the one that does exactly what we told it to do.