Before you swap models: a small eval set that catches regressions

October 9, 2026 · Updated November 9, 2026

When a provider retires a model, the code change is often one line. The risk isn't in the line. It's in what the new model does differently with your prompts. A summary gets longer, a classifier changes its label for edge cases, a JSON output gains an extra field, or a parameter you've always sent is now rejected.

With retirement notices now as short as about 60 days (Anthropic's 2026 announcements gave 60 to 62), you don't have time to discover those differences in production. You need a way to compare old and new models quickly, before the deadline. A small evaluation set is that way.

This post shows how to build one that's useful without being a research project.

What "small" means

Twenty to fifty cases per feature. Not hundreds. The goal isn't a benchmark score; it's catching the regressions that would embarrass you or break a downstream system. A short list you actually run beats a long one you don't.

Step 1: pick cases from real traffic

Start from what your feature really sees, not what you imagine it sees.

Typical cases (about half): ordinary inputs, the kind that make up most traffic.
Hard cases (about a quarter): long inputs, ambiguous requests, inputs in other languages, the cases that caused bugs before.
Contract cases (about a quarter): inputs where the output format matters to other code: JSON that a parser reads, a label that drives a workflow, a length limit a UI depends on.

Remove personal or customer data before saving cases. Rewrite them by hand if needed; the structure matters more than the exact text.

Step 2: write checks, not just expected answers

For most generative features, an exact expected output is too strict. Instead, write checks per case:

Format checks: valid JSON, required fields present, no extra prose around it.
Constraint checks: under N words, includes a citation, avoids a banned phrase.
Label checks: for classifiers, the expected label, or a set of acceptable labels.
Reference comparisons: for open-ended outputs, keep the current model's output as a reference and review differences by eye.

Automate the first three. The last one is a human review, and that's fine at 20 to 50 cases.

Step 3: run old and new side by side

Run the same cases through the current model and the candidate replacement, with the same prompts and settings. Save both outputs in a simple table: case, old output, new output, check results.

Look at differences, not averages. "95% pass on both" can hide the fact that the new model fails three cases the old one passed. Those three cases are your work list.

Step 4: watch for request-level breaks

Some failures happen before the model even responds. Newer models may reject parameters older ones accepted. For example, Claude 4.7 and later models reject temperature, top_p and top_k with HTTP 400, and version 1.0 of Anthropic's Python SDK removed them (see our note on sampling parameters). An evaluation run catches this at once: every case errors. That's a much better time to find out than after deploy.

Also check:

limits on input or output length;
system prompt handling;
tool or function-calling formats;
streaming behavior, if your code depends on it.

Step 5: fix prompts only where the evidence says so

When a case fails on the new model, resist rewriting the whole prompt. Change the smallest thing that fixes the failing case, then re-run everything. Prompt changes that fix one case often break another, and the eval set is how you notice.

Step 6: keep it in the repository and run it in CI

Store the cases and checks next to the feature's code. Add a script that runs them against whichever model is configured. Then:

run it on any pull request that changes the model ID or the prompt;
run it on a schedule, so you notice if a provider changes behavior behind an alias.

Running against live models costs money, so keep the set small and run it when it matters, not on every commit.

Step 7: record the decision

When you switch, write down in the pull request or ticket which cases changed, why you accepted the differences, and which provider notice triggered the change. The next retirement will come, and this record makes the next migration faster.

How this fits a retirement timeline

If you already have the set, a model retirement looks like this:

Week 1: read the notice, find each reference (see finding deprecated model IDs), pick the replacement.
Week 2: run the evaluation set, fix failures, re-run.
Weeks 3 to 6: roll out and clean up, including fallback paths.

If you don't have one, building it is week 1. That's still possible inside a 60-day window, but only just.

A minimal file layout

features/summarize/
  prompt.txt
  evals/
    cases.jsonl        # one case per line: input, checks
    run_evals.py       # runs cases against the configured model
    last_run.md        # table of old vs new outputs from the latest comparison

Common mistakes

Using only synthetic cases. They miss the weird inputs real users send.
Comparing to a gold answer written months ago. Compare against the current model's actual output instead.
Treating the score as the decision. The score tells you where to look; a person decides.
Forgetting other paths. Batch jobs and fallbacks may use different prompts. Give them a few cases too.

The takeaway

You can't control when a provider retires a model. You can control how long it takes you to trust the replacement. Twenty to fifty real cases with simple checks turn a stressful migration into a routine afternoon. Build the set before the next notice arrives. Pair it with model IDs kept in one place and a 60-day runbook, and retirements become maintenance rather than emergencies.