Use this to define when humans review AI output. Fill this out before building any UI. If you skip this, you'll end up with a "just ship it and see" approach to AI autonomy.
Upstream: autonomy levels and human review rules from the AI PRD define what goes here. Downstream: review point failures feed into the PRD risk table, launch gate, and weekly post-launch review.
HITL mode
Pick the review mode per AI action. Different actions in the same product can use different modes.
| Mode | What it means | Use when | Example |
|---|---|---|---|
| Human-in-the-loop | Human approves before the AI output or action takes effect | Customer-facing, high-impact, irreversible, or low-confidence work | Agent drafts an email; support rep reviews before sending |
| Human-on-the-loop | AI acts, humans monitor and can intervene or roll back | Action is reversible, monitored, and low-to-medium risk | AI categorizes tickets; ops reviews alerts and samples |
| Human-after-the-loop | Humans review samples, incidents, and aggregate trends after the fact | Risk is low, action is reversible, and online metrics are strong | Weekly review of accepted summaries and rejection reasons |
| No human loop | AI acts without human review | Only for low-risk, reversible, well-tested actions with observability | Internal formatting cleanup or low-stakes enrichment |
Mode map
| AI action | HITL mode | Why this mode | Upgrade or downgrade trigger |
|---|---|---|---|
Actions AI can take alone
Actions AI can take with rollback
| Action | Rollback mechanism | Undo window | Who monitors |
|---|---|---|---|
Actions AI can suggest only
| Action | Who reviews | Max review latency | What reviewer sees |
|---|---|---|---|
Actions AI must never take
Required review points
| Review point | Trigger | Reviewer | What they check | What happens if rejected |
|---|---|---|---|---|
Review UI requirements
- AI output displayed: full output, diff, summary?
- Source/evidence shown: does the reviewer see what the AI based its answer on?
- Confidence indicator: is confidence shown? how?
- Edit capability: can the reviewer modify the output before approving?
- Time to review target: how long should a single review take?
Escalation path
- Reviewer flags output as uncertain
- next step
- resolution
Audit trail
- AI output (full)
- Reviewer identity
- Review decision (approve/reject/edit)
- Edits made by reviewer
- Timestamp
- add product-specific fields
Feedback captured from review
| Feedback type | How captured | How used |
|---|---|---|
| Rejection reason | e.g., dropdown + free text | e.g., added to eval set |
| Edit diff | e.g., stored automatically | e.g., fine-tuning data |