Scoring guides

A scoring guide says what good looks like, so you can rate AI work the same way every time. Each row is rated Strong, Partly or Weak, with the evidence you quote. Read them below, or download them to keep.

Scoring guide: a summary

Use it when: You asked an AI to summarize a report, a thread or a set of notes, and you want to know if you can rely on it.

  • Leads with the bottom line

    Can the reader tell the main point from the first sentence?
    Evidence: Quote the first sentence.
    StrongThe first sentence states the main point and any decision needed.
    PartlyThe main point is there, but it comes after background.
    WeakThe main point is missing, or buried near the end.

  • Matches the source

    Does every number, name and date match the material?
    Evidence: Quote each figure next to its source line.
    StrongEvery figure, name and date is as written in the source.
    PartlyFigures are right, but one is rounded or reworded.
    WeakA figure is wrong, blended from two lines, or not in the source.

  • Nothing added

    Does every sentence trace back to the material?
    Evidence: Quote the sentence and the line it came from, or say there is none.
    StrongEvery sentence traces to the source.
    PartlyOne sentence goes a little past the source, such as a mild conclusion.
    WeakIt adds claims, causes, quotes or recommendations the source never makes.

  • Right length and shape

    Does it fit the length and order that was asked for?
    Evidence: Count the words and list the parts that are there.
    StrongIt fits the length and the order the reader asked for.
    PartlyIt is a little long, or one asked-for part is thin.
    WeakIt is far over length, or an asked-for part is missing.

Scoring guide: a review of a draft

Use it when: You asked an AI to review a draft, a plan or a proposal, and you want to know if the review is worth acting on.

  • Specific

    Does each point say exactly where the problem is?
    Evidence: Quote two of its points.
    StrongEvery point quotes the line it is about and says why.
    PartlyMost points name a place, and a few are general.
    WeakPoints are general, such as "add more detail".

  • Finds real problems

    Does it find what a careful reader would find, and nothing invented?
    Evidence: Name the main problem you know of, and say whether the review found it.
    StrongIt finds the problems a careful reader would, and none that are not there.
    PartlyIt finds the main problem but misses one, or flags something that is fine.
    WeakIt misses the main problem, or invents problems.

  • Ranked

    Is the most important fix first?
    Evidence: List the first three fixes in the order given.
    StrongFixes are in order of what matters, the biggest first.
    PartlyThere is some order, but big and small fixes are mixed.
    WeakThere is no order, or style notes come before real problems.

  • Leaves the draft yours

    Does it point at fixes without rewriting?
    Evidence: Compare a rewritten line with your own.
    StrongIt points at fixes and leaves the writing to you.
    PartlyIt rewrites one or two lines as examples.
    WeakIt rewrites the draft, or changes what the draft means.

Scoring guide: a recommendation

Use it when: You asked an AI to compare options and recommend one, and you want to know if the recommendation holds up.

  • A clear choice

    Does it name one option, in the first line?
    Evidence: Quote the first line.
    StrongIt names one option to choose, in the first line.
    PartlyIt leans one way but hedges.
    WeakIt lists the options without choosing.

  • Reasons from the material

    Does each reason come from facts you gave it?
    Evidence: Quote each reason and the line it rests on.
    StrongEach reason comes from facts in the material, quoted.
    PartlyThe reasons are sound, but one rests on an assumption.
    WeakThe reasons are general, or made up.

  • Cost and risk shown

    Are the main cost and the main risk of the choice stated?
    Evidence: Quote the cost line and the risk line.
    StrongThe main cost and the main risk are both stated.
    PartlyOne of the two is missing.
    WeakOnly the benefits are shown.

  • What would change it

    Does it say what new fact would change the answer?
    Evidence: Quote the line, or say it is missing.
    StrongIt says what new fact would change the recommendation.
    PartlyIt hints at uncertainty without saying what matters.
    WeakIt presents the choice as certain.

How to use a scoring guide

  1. 1
    Find the evidence
    For each row, look in the work for the line the row is about, and quote it.
  2. 2
    Rate the row
    Pick Strong, Partly or Weak using the words in the guide. If you cannot quote evidence, it is Weak.
  3. 3
    Fix the weakest first
    Change the weakest row, or send it back to the AI with that row quoted. Then run the guide again.

More in Quality

Quality home