From AI Demo to Engineering System: Why Evals Matter

articleOctober 1, 2026By Ali Haghighi

Engineering Reliable AI Systems, part 2

One of the uncomfortable parts of building my first small AI engineering experiments was realizing how quickly I started trusting repetition.

A prompt worked. I changed something and ran it again. It still worked. After enough successful runs, the feature began to feel stable even though I had not defined what "stable" meant.

That is a familiar software-engineering mistake in a new form.

As soon as I changed a prompt, model, context or tool response, I needed a better answer than "the output still looks reasonable." I needed cases that recorded the behavior I actually cared about.

That was the point where evals stopped feeling like an AI-specific technique and started looking like ordinary regression engineering.

Start with cases you actually care about

I started with a deliberately small set of explicit cases. For the kind of engineering assistant I was experimenting with, a benchmark with hundreds of synthetic examples would have given me a larger number, but not necessarily more understanding.

A handful of cases I could explain was more useful.

For a code review feature, a case might contain a diff, some repository context and an expected property of the review. The expectation does not have to prescribe the exact wording.

For example:

TEXT
Case: cancellation token is ignored
Input: a method starts an external operation without forwarding cancellation
Expect: the review identifies the missing cancellation propagation

Another case can deliberately contain no meaningful issue:

TEXT
Case: harmless refactoring
Input: rename and formatting changes only
Expect: no invented defect

That second case is important. An AI system that always finds something to say can look productive while steadily creating noise.

A small contract is enough to make those cases executable:

C#
public sealed record EvalCase(
    string Id,
    string Input,
    EvalExpectation Expectation);

public sealed record EvalExpectation(
    string RequiredFinding,
    bool AllowAdditionalFindings = true);

public sealed record EvalResult(
    string CaseId,
    bool Passed,
    string? FailureReason);

The contract is intentionally boring. The interesting knowledge belongs in the cases and expectations, not in a large evaluation framework.

Evaluate behavior, not prose

Exact string matching is usually the wrong level of comparison for model output.

If I expect one particular paragraph, a harmless wording change becomes a failure. If I only check that the response is non-empty, almost anything passes.

The useful layer is somewhere in between: properties of the result.

Did it identify the relevant risk? Did it cite evidence that exists in the supplied context? Did it avoid claiming facts that were unavailable? Did it return nothing when there was nothing useful to report?

Some of these checks can be deterministic. Others may need a model-based evaluator. I keep deterministic checks wherever possible and make the less deterministic part explicit.

Keep the evaluator separate from the feature

One design mistake is letting the same production path decide whether its own answer is good.

The application produces an output. The evaluation harness should consume that output as data and apply its own expectations.

That separation gives me a useful workflow:

TEXT
[Test Case]
     |
     v
[System Under Evaluation]
     |
     v
[Observed Output]
     |
     v
[Evaluator]
     |
     v
[Pass / Fail + Evidence]

It also means I can run the same cases against a new prompt, model or implementation without rewriting the application.

A failed eval should be diagnosable

A red number is not enough.

When a case fails, I want to see the case identifier, the observed result, the expectation that was not met and enough context to reproduce the run.

That sounds ordinary because it is ordinary. Good AI engineering borrows heavily from software engineering: reproducibility, small changes, useful failure messages and versioned test inputs.

The interesting part is deciding what behavior deserves to be tested.

Evals are not proof

Passing an eval suite does not prove that an AI system is correct.

The cases reflect what we thought to test. Production inputs will be messier. Models can behave differently across versions, and some qualities are difficult to reduce to a binary assertion.

I treat evals as regression protection and accumulated knowledge.

When a real failure teaches me something, I can turn it into a case. That is the part I find most valuable: the suite becomes a record of assumptions the system has already violated once, rather than a benchmark assembled only to look comprehensive.

The practical threshold

I do not think every prototype needs an evaluation platform.

But once an AI capability is being changed repeatedly, used by other people, or trusted for a meaningful decision, manual spot checks stop scaling.

At that point I want a small, runnable eval suite in the repository before I want a more sophisticated prompt.

The first article in this series argued for keeping measurable decisions outside the model where possible. Evals extend the same idea to behavior that cannot be reduced to one deterministic calculation.

The goal is not to make an LLM deterministic.

The goal is to make changes to an AI system observable enough that we can tell when we made it worse.