Builder track: page 2 of 8

How Workflows Fail

Workflows rarely fail with an error message. They fail with a result that looks finished and is wrong. Each kind of failure starts in one place, and each has a check that can catch it there.

InputStep 1Step 2CheckYou lookResult
The input
Trouble can start before any step runs: a document too long for one step loses its middle.

What Every Failure Has in Common

  1. 1It looks finishedA made-up number reads as well as a real one, and a list with a gap still looks like a list. You cannot spot these by how the result looks.
  2. 2It starts in one placeIn the input, in a step, in a hand-off between steps, in a loop or at the end, where only a person can judge.
  3. 3A check can stand right therePut the check straight after the place the failure starts, so the next step never builds on it.
  4. 4Plant it to test the checkTo know a check works, plant the failure in a sample and see the check catch it. The next page shows how.

Eight Failures and the Check That Catches Each

In each picture the part that goes wrong is outlined in red, and the check that catches it is drawn below.

  1. Notes:12 sitesSummary:15 sitesCompare to the source

    A Made-Up Detail

    A step writes a number, name, date or quote that is not in the material, and it reads as well as the real ones.

    For example: The notes say the program ran at 12 sites; the summary says 15.

    The check that catches it: Compare to the source: every number, name and quote in the result must be found in the input, word for word, or the check fails and names the line.

    Lesson 4: ChecksCommon AI mistakes
  2. Notes:5 decisionsList:4 decisionsCount both ways

    Something Left Out

    A step drops an item, often the awkward one, and the list still looks complete because nothing marks the gap.

    For example: Five decisions were made in the meeting; the list has four, and the one dropped is the costly one.

    The check that catches it: A checklist item that counts both ways: one row for each decision in the notes, and no row that is not in the notes.

    The kinds of checksLesson 4: Checks
  3. Asked: forthe boardWrote: fornew staffScoring guide: the reader

    A Different Question Answered

    The step does a nearby job instead of the one asked: the wrong reader, the wrong length or the wrong purpose.

    For example: Asked for a one-page brief for the board, it writes a how-to guide for new staff.

    The check that catches it: A scoring guide row that names the reader and the job: written for the board, one page or less, ends with the decision the board is asked to make.

    Scoring guidesLesson 3: Writing the prompt
  4. Step 1:a paragraphStep 2 needsa tableShape check between steps

    A Broken Hand-Off

    A step hands over its work in a shape the next step does not expect, and the next step guesses instead of stopping.

    For example: Step 1 returns a paragraph; step 2 expects a table with an owner column, so it invents owners.

    The check that catches it: A shape check between the steps: the hand-off is a table with the agreed columns, and no cell is blank without saying none named.

    Lesson 5: Hand-offsHand-offs, the parts guide
  5. 60 pagesSummary:start, endEvery part covered

    The Middle Goes Missing

    A long input goes into one step whole, and the result covers the opening and the end but thins out or skips the middle.

    For example: A 60-page report goes in at once; the summary never mentions chapters three to six.

    The check that catches it: Split the input into parts and summarize each one, then check that every part appears in the merged summary.

    Lesson 7: Split, angles and best of severalSplit and merge, the parts guide
  6. ReviseFails again,try 7A limit of two tries

    A Loop That Will Not Stop

    A check keeps failing and the step keeps rewriting, each round changing parts that were fine, until the draft has drifted far from the source.

    For example: A check on tone keeps failing; by the seventh rewrite the draft has lost two of its facts.

    The check that catches it: A limit: two tries, then stop, keep the best try and flag it for a person, with what the check found each time.

    Lesson 6: Loops with a limitLoops and limits
  7. All checkspassTakes onesideA person looks

    A Call Only a Person Can Make

    Every automatic check passes, but the result is unfair, unwise or wrong for the people who will read it.

    For example: The summary of a disagreement between two teams gets every fact right and still reads as taking one side.

    The check that catches it: A person looks: the workflow stops and asks one plain question, such as whether the summary is fair to both teams.

    Lesson 8: Where a person must lookCheck it yourself
  8. PrompteditCase 3 nowfailsRe-run the test set

    A Fix That Breaks Another Case

    A change that fixes one case quietly breaks another that used to pass, and nobody finds out until it matters.

    For example: A prompt edit fixes long meeting notes, and short notes now come back with no actions at all.

    The check that catches it: Re-run the whole test set after every change, with the same checks, and compare the result with the last run record.

    Lesson 9: Testing a workflowTest your own workflow

Go Deeper

Common AI mistakesWhat to look for in any AI answer, and how to catch it.The kinds of checksCompare to the source, checklists, scoring guides and a person looking.Practice: spot the mistakeFind the planted mistakes in real-looking AI answers.