Catch what AI gets wrong, before anyone else does
Each summary below has three planted mistakes. Read what the AI was given, click the sentences you think are wrong, then check.
Staff survey summary
SampleAn AI summarized a staff survey.What the AI was given (survey results)
- 214 responses from staff in four offices.
- Question 4, "I feel supported by my manager": 54% agree.
- Question 9, "I get useful feedback": 90% agree.
- Comment themes: clearer priorities came up most. Workload complaints rose from 31 to 47 comments. The new meeting schedule began in the same months.
- Communication rating (out of 5): remote staff 3.4, office staff 4.1.
The AI's summary: click each sentence you think is wrong
Nothing marked yet
Watch a check run
Review a draft before you send itA check runSample
Draft email: plan for the new client portal
1Can you approve the portal plan by Thursday, so we can book the developers?
2The pilot cut support calls by 18%, based on the June usage report.
3We will have the portal delivered in two weeks, ready for the board meeting.
4Cost stays at $42,000, as in the approved budget sheet.
Every number matches the sourceWaiting
Every claim names where it came fromWaiting
The ask is in the first paragraphWaiting
No promise the team cannot keepWaiting
Under one pageWaiting
Kinds of checks
- Compare to the sourceNumbers, names and quotes must match what you gave it.
- ChecklistYes or no items: is every part there, in the right shape?
- Scoring guideRate it against what good looks like, with the evidence.
- A person looksThe points where your judgment has to come in.
Prompt before and after
The prompt
Summarize this report.
Length of what came back600 words
White line: the 200-word limit set in the after prompt
Over the 200-word limit
What came backSample
The regional office was set up in 2019 to bring services closer to rural clients, and since then it has grown steadily...
ProblemOpens with background
Uptake rose by about a fifth over the year.
ProblemRounds 18.4% to "about a fifth"
Paragraph 5: the board must decide on the second office and the hiring freeze.
ProblemThe two decisions needed, buried in paragraph 5
We recommend expanding to three more regions next year.
ProblemA recommendation the report never made
More ways to practice
Trace the number
SampleClick a claim to find its source line.
Source: annual volunteer table
| Volunteer hours | 1,240 last year, 1,488 this year |
|---|---|
| Towns served | 12 |
| Active volunteers | 96 |
Diagnose and fix
SampleProject update, week 6
The new intake form is live in two of three offices. Average wait time dropped from 9 days to 6. The third office goes live after staff training on Oct 20. Training materials are ready.
The new intake form is live in two of three offices. Average wait time dropped from 9 days to 6. The third office goes live after staff training on Oct 20. Training materials are ready.
Given to the AI: Written for a manager. The AI was given the team tracking sheet: wait times of 9 days before and 6 days after the new form.
What went wrong?