Design an AI feature evaluation rubric before shipping

Description

You're about to ship an AI feature and your only eval is "looks good to the team." This designs a proper eval rubric — task set, scoring criteria, golden answers, regression guardrails — so you can ship with confidence and catch drift later.

You are an AI eval designer building the pre-launch rubric for {{feature_name}} on {{product_name}}. User task: {{user_task}}.

## Step 1 — Task set
Build a test set of 30-100 real user inputs spanning:
- Happy path (clear input, obvious intent)
- Edge cases (ambiguous input, unusual phrasing)
- Adversarial (jailbreaks, prompt injections, out-of-scope)
- Representative error modes (typos, unsupported languages, missing context)

## Step 2 — Scoring criteria
Define 3-5 dimensions, each 1-5:
- Correctness (did it answer the right thing?)
- Completeness (did it answer fully?)
- Faithfulness (did it hallucinate?)
- Safety (did it refuse appropriately?)
- User-facing quality (would a user be satisfied?)

## Step 3 — Golden answers
For the 30 happy-path tasks:
- Write the ideal answer (or 2-3 acceptable variants)
- Annotate what makes it good/bad

## Step 4 — Regression guardrails
- Minimum score per dimension to ship
- Automated diff on every model/prompt change
- Human review cadence for random samples

## Step 5 — Output
1. Task set spec
2. Rubric with scoring examples
3. Pass/fail ship criteria
4. The one failure mode we'd tolerate (and why)
5. The monitoring plan for post-launch drift

Customize This Prompt