Set up an AI-orchestrated QA and ship checklist

The Problem

The last production incident your team caused was probably preventable. Not with better code. Not with better testing. With a better checklist.

Production incidents in software are overwhelmingly caused by known categories of failure: configuration errors, missing migrations, broken feature flags, untested edge cases, and dependency version mismatches. These are not creative failures. They are repetitive, predictable failures that happen because the deploy process relies on human memory instead of systematic verification.

Google's Site Reliability Engineering team reports that 70% of outages are caused by changes to a live system, most of which could have been caught by pre-deploy verification. The fix is not fewer deploys. It is better checklists.

A study published in the New England Journal of Medicine found that surgical checklists reduced complications by 36% and deaths by 47%. The principle transfers directly to software: when the stakes are high and the failure modes are known, a checklist outperforms expertise every time.

Why Manual Checklists Fail

Teams have tried checklists before. They create a Confluence page with a deploy checklist, and it works for two months. Then it goes stale. Someone skips a step. The checklist does not match the current architecture. Items get checked without being verified. A manual checklist is better than no checklist, but it degrades with every deploy.

How This Prompt Works

This prompt helps you build an AI-orchestrated QA and ship checklist that is dynamic, self-verifying, and integrated into your deploy workflow. Unlike a static document, this checklist adapts to what you are shipping and verifies conditions programmatically where possible.

The checklist operates in three phases:

• Pre-Deploy Verification: The AI analyzes the diff being deployed and generates a tailored checklist based on what changed. A database migration triggers schema verification checks. A feature flag change triggers flag state validation. An API endpoint change triggers contract testing verification.
• Deploy Monitoring: During deploy, the checklist monitors key health signals: error rates, latency percentiles, memory usage, and custom business metrics. It compares real-time values against pre-deploy baselines and flags anomalies.
• Post-Deploy Validation: After deploy, the checklist verifies that the expected changes are live and functioning. It runs smoke tests, validates critical user flows, and confirms that monitoring and alerting are active for the new code.

The output is a run-specific checklist with pass/fail status for each item, a deploy health report, and an automatically generated rollback command if any critical check fails.

When to Use It

• Your team has had a production incident in the last quarter that a checklist would have prevented
• You are scaling the team and need to codify deploy best practices for new engineers
• You ship frequently and want to maintain reliability without slowing down
• Your current QA process is manual and inconsistently applied

Common Pitfalls

• Making the checklist too long: A 50-item checklist will not get followed. Focus on the items that prevent the failures you have actually experienced. Start with 10-15 items and iterate.
• Not updating it after incidents: Every production incident should generate a new checklist item. If the checklist does not evolve, it will miss the next novel failure.
• Skipping the rollback plan: Every deploy checklist must include a rollback procedure. If you cannot roll back, you cannot ship safely.

Further Reading

• Site Reliability Engineering: How Google Runs Production Systems - Google SRE
• A Surgical Safety Checklist to Reduce Morbidity and Mortality - New England Journal of Medicine
• The Checklist Manifesto - Atul Gawande