Run an AI feature eval with golden dataset

Description

Your AI feature needs an objective quality measurement — not vibes, not shipping-week demos. This runs an eval with a golden dataset, blind human scoring, and statistical significance checks so you can report "quality" with a number, not an impression.

You are running a formal eval for {{ai_feature}}. Goal: produce a defensible quality number.

## Step 1 — Dataset
- 100-500 real production inputs
- Balanced across use cases
- Ground-truth labels from domain experts (not PMs)
- Held-out test split (not used for prompt iteration)

## Step 2 — Scoring
### Automatic metrics
- Exact match (where applicable)
- ROUGE/BLEU for text generation
- Custom task-specific metrics

### LLM-judge
- Calibrate LLM judge on 50 labeled examples first
- Spot-check judge agreement with human labels (should be >85%)

### Human scoring
- Blind (judge doesn't know which model/prompt)
- 3 annotators per item, aggregate
- 5-point rubric with anchor examples

## Step 3 — Statistical rigor
- Confidence intervals (bootstrap)
- Sample size for 95% confidence
- Multiple comparison correction if testing >2 variants

## Step 4 — Reporting
- Overall quality score (0-100 or pass rate)
- Per-category scores
- Confidence intervals
- Variant comparison if applicable
- The one category where model is unreliable

## Output
1. Eval spec (dataset, metrics, judge setup)
2. Scoring rubric
3. Report template
4. The one eval decision (metric choice) that most affects the headline number

Customize This Prompt