Skip to content
SunsetLint

The fallback that fails on shutdown day

October 9, 2026 · Updated November 9, 2026

Here's a failure that's easy to miss. Your team gets a retirement notice, migrates the main code path, runs the evaluations and ships. Usage of the old model drops to zero. Done.

Then, weeks after the shutdown date, the primary model has a short outage or rate-limits a burst of traffic. Your retry wrapper does what it was built to do: it falls back to a second model. That second model is the one you retired. The request fails, the fallback fails, and a resilience feature becomes the cause of the incident.

This post covers why fallbacks keep retired model IDs alive, where they hide, and how to find and test them.

Why fallbacks survive migrations

Fallback paths have three properties that make them easy to miss.

They rarely run. A fallback only executes when something else fails. In a normal week it might not run at all. That means it won't show up in a usage report covering the weeks when you checked.

They're written once and forgotten. Fallbacks are often added during an incident: "if the main model is overloaded, use this one". The fix ships, the incident closes, and nobody revisits the choice of fallback model.

They live in different places. The primary model is usually in config. The fallback is often hardcoded in a retry helper, a client wrapper, or a default argument, because it was added in a hurry.

Why usage exports don't catch them

Providers' usage data is valuable. Anthropic, for example, offers a usage export so you can see which models your organization has called. But it shows what ran in the period you export. A fallback that didn't fire during that period looks exactly like code that doesn't exist.

That's why a migration needs two views: what you called (usage data) and what's written (a code search). Fallbacks are the clearest case where only the second view finds the problem.

Where to look

Search for these patterns in each service that calls a model API.

Retry and fallback helpers

Look for functions with names like with_fallback, call_with_retry, fallback_model, backup_model, or lists of models tried in order:

MODELS_IN_ORDER = ["<primary model>", "<old model kept as a backup>"]

An ordered list is a common pattern, and the last entry is often the oldest.

Default arguments

def summarize(text, model="<model ID from last year>"):
    ...

If most callers pass model= explicitly, the default only applies in the one place that doesn't, often a script or a job.

Environment defaults

FALLBACK_MODEL = os.getenv("FALLBACK_MODEL", "<old model ID>")

The variable may be set in production but not in staging, a cron job or a new region.

Feature flags and config branches

A flag that switches between "new" and "legacy" model paths. If the legacy branch can still be enabled, it can still call the retired model.

Gateway or proxy configuration

If you route model calls through a gateway, check its fallback rules too. They live outside your application code entirely.

A search that helps

Start from the identifiers. If you're dealing with the Claude Sonnet 4.5 retirement on 30 November 2026, a tolerant pattern finds most spellings:

rg -n -i "sonnet[-_. ]?4[-_.]5"

Then search for fallback structure, regardless of model:

rg -n -i "fallback|backup_model|models_in_order|retry" --type py --type ts

Review each hit that names a model. Our guide to finding deprecated model IDs covers more file types, including Terraform and notebooks.

Test fallbacks on purpose

Finding the code is half the job. The other half is proving the fallback works with the model it names now.

  1. Force the fallback in a test. Mock the primary call to fail, then assert the fallback model is one you expect and is not on a retirement list.
  2. Run the fallback once against the live API in staging, using the real ID, after any model change.
  3. Alert on fallback use. Log a distinct event when a fallback fires, and watch the error rate on that event. A fallback that errors is worse than no fallback, because it hides the original failure.

A simple unit test:

def test_fallback_model_is_not_retiring():
    model = pick_fallback_model()
    assert model not in RETIRING_MODELS

Keep RETIRING_MODELS up to date from the providers' deprecation pages, and review it when a notice arrives.

Decide whether you need the fallback at all

Sometimes the honest answer is that a fallback to an older, weaker model isn't worth it. Consider:

  • Same model, different region or platform, if your provider or cloud supports it, instead of an older model.
  • A newer, cheaper model as the fallback, kept in the same config as the primary.
  • Failing clearly with a user-facing message and a retry later, for features where degraded output would be worse than none.

Whatever you choose, put the fallback model in the same config module as the primary, so they're reviewed together. See keeping model IDs in one place.

Add it to the migration checklist

The clean-up phase of any model migration should include an explicit line: fallbacks, retries and defaults checked and tested. It's in our 60-day runbook, and it's worth adding to yours.

The takeaway

Fallbacks exist for bad days, which is exactly why they're tested least. After each migration, search for the old ID in retry helpers, defaults and gateway rules, force the fallback in a test, and alert when it fires. It's an hour of work that prevents the most confusing kind of incident: the one caused by your safety net.

SunsetLint, our GitHub App in early access, is being built to flag retiring identifiers wherever they appear in a repository, including the rarely run paths. But the search and the tests above work without it.