Do you run evals on your AI features?

You tweak a system prompt to fix one complaint, or switch to a newer model because it's cheaper. The app still compiles, the unit tests still pass, and a week later users report that answers got worse. LLMs are non-deterministic, so a classic test that expects an exact string either fails randomly or checks too little to catch the regression.

Evals fix this. An eval runs your AI feature against a set of real examples and scores each output against criteria you define, so you can see whether a change made things better or worse before you ship it.

What an eval is made of

• A dataset - Inputs that represent real usage, each with the expected answer or the qualities a good answer must have. Start with 20-50 cases taken from real user requests and real failures, and include edge cases (typos, ambiguous questions, off-topic requests)
• Graders - The checks that score each output (see below)
• A threshold - The pass mark for the run, e.g. "at least 95% of cases pass" or "average relevance score of 4 out of 5 or higher". Thresholds, not exact matches, are what make LLM output testable

Pick the cheapest grader that works

1. Code-based checks - Is the output valid JSON? Does it match the schema? Did the agent call the right tool with the right arguments? Does it contain the order number? These are fast, cheap and repeatable, so use them wherever you can
2. LLM-as-judge - Another model scores the output against a rubric, for qualities like relevance, groundedness, completeness or tone. Use a different model from the one under test, and spot-check its scores against human judgement before you trust it
3. Human review - The gold standard, but slow. Use it to build the dataset, calibrate the judge, and read a sample of real transcripts regularly

When to run evals

• On every prompt or model change - Run the suite in your CI pipeline and fail the build when the score drops below the threshold
• Before upgrading a model - Run the same suite against the current and the new model and compare. This turns "the new model feels better" into numbers. See Do you pick the best Large Language Model for your project?
• In production - Score a sample of live traffic with the same graders, and alert when quality drifts. Turn every new failure into a dataset case, so the suite grows from real problems

❌ Figure: Bad example - Bad example - An exact-match assertion fails whenever the model rewords a correct answer, so the team learns to ignore it

✅ Figure: Good example - Good example - A dataset of real cases with code checks, an LLM judge and a pass threshold catches regressions without flaky failures

Tools

You don't need a platform to start. A spreadsheet of cases and a test that loops over them is already an eval. When you outgrow that, use a tool built for it:

• .NET - Microsoft.Extensions.AI.Evaluation runs evals inside MSTest, xUnit or NUnit, with built-in quality, safety and agent evaluators, response caching and HTML reports
• Azure - Microsoft Foundry evaluations cover pre-production evaluation, CI/CD quality gates and continuous evaluation of production traffic
• Any stack - promptfoo (YAML-configured CLI, also does red teaming), DeepEval (pytest-style), and hosted platforms such as LangSmith and Braintrust

Here is a .NET eval that uses an LLM judge to score coherence, adapted from the Microsoft quickstart:

IEvaluator coherenceEvaluator = new CoherenceEvaluator();
EvaluationResult result = await coherenceEvaluator.EvaluateAsync(messages, response, chatConfiguration);

NumericMetric coherence = result.Get<NumericMetric>(CoherenceEvaluator.CoherenceMetricName);
Assert.IsFalse(coherence.Interpretation!.Failed);
Assert.IsTrue(coherence.Interpretation.Rating is EvaluationRating.Good or EvaluationRating.Exceptional);

Running evals in the same test suite as the rest of your code is a good start, because the whole team sees them on every build.

🙂 Figure: OK example - OK example - EagleEye runs its OpenAI tests alongside its other integration tests. Scoring them against a dataset with thresholds turns them into evals