Design a model upgrade migration runbook

Model Upgrades Are Deployments, Not Configuration Changes

Upgrading a model version without a structured rollout is equivalent to deploying untested code to 100% of production. Anthropic's model migration guidance and Stack Overflow's AI tooling writing both document the right pattern: full eval comparison, cohort rollout from internal to 100%, rollback criteria with automated triggers. User-facing behaviors regress in non-obvious ways across model versions — assume regression and rollback-ready.

How the Design a model upgrade migration runbook Prompt Works

The prompt runs a four-stage rollout (internal → 5% → 50% → 100%) with eval comparison, cost impact, and rollback criteria per stage. The "task we'd watch most carefully" output forces naming the single highest-risk behavior — the one most likely to regress without showing in aggregate metrics.

When to Use It

• A new model version has been released and the team wants to upgrade.
• A cost reduction opportunity requires migrating to a cheaper model.
• A previous migration caused production issues.
• A new AI PM is establishing upgrade discipline.
• A compliance review requires documented model migration procedures.

Common Pitfalls

• Full rollout without eval comparison. New model versions regress in non-obvious ways. Test first.
• No rollback criteria. Without explicit criteria, regressions get rationalized.
• Assuming better-means-better everywhere. A model that's better on average can be worse on load-bearing tasks for your product.

Sources

• Anthropic Research — Anthropic
• Stack Overflow Blog — Stack Overflow
• GitHub Developer Research — GitHub
• AI Adoption in Product Orgs — Reforge