Run an AI feature hallucination audit

Description

Users are complaining about AI output that sounds plausible but is wrong. This runs a structured hallucination audit — real user logs, categorization, root cause — so you understand where the model makes things up and can apply the right fix (prompt, retrieval, fine-tune, or scope).

You are auditing hallucinations in {{ai_feature}}. Audit sample: last {{time_window}} of user interactions.

## Step 1 — Sample and label
Pull 100 random user interactions. For each:
- Input
- AI output
- Is there a hallucination? (Y/N)
- Category (see Step 2)
- Severity (1-5)

## Step 2 — Categorization
- **Fabricated fact**: AI states a specific untrue fact (company revenue, product feature, person's role)
- **Wrong citation**: AI cites a source that doesn't exist or doesn't support the claim
- **Procedural**: AI describes steps that don't work (wrong menu path, non-existent button)
- **Stylistic overreach**: AI claims certainty on something that's actually uncertain
- **Out-of-scope**: AI answered when it should have refused
- **Dropped context**: AI ignored provided context and made up facts

## Step 3 — Root cause analysis per category
For each category that appears ≥3x:
- Is it a prompt issue (could be fixed with better instruction)?
- Is it a retrieval issue (missing or wrong context)?
- Is it a model capability issue (this model can't do this well)?
- Is it a scope issue (shouldn't be asked)?

## Step 4 — Interventions
- Prompt: tighten with refusal instructions
- Retrieval: add grounding sources
- Fine-tune: build training set from correct outputs
- Scope: add guardrail that refuses certain inputs

## Output
1. Hallucination frequency and severity summary
2. Top 2 categories with root cause
3. Top 2 interventions ranked by impact
4. The one user-facing change we'd ship immediately

Customize This Prompt