Turn "make it better" into a check

When you tweak a prompt you reuse, how do you know the new version is better? Usually you try it once, it looks fine, and you move on. Then three weeks later it does something odd on a case you didn't try. The fix is the same thing a careful business does with any process: write down what good means in a way you can check, and test against real examples every time you change something.

Step by step

1. Pick one prompt you reuse often and write down, in a sentence, what it's for.
2. Turn "better" into rules you can answer yes or no. Not "friendly" but "uses the customer's name once, no exclamation marks". Aim for four to six rules: length, what it must include, what it must never do, and one about tone that you can actually check.
3. Use pass or fail, not a 1 to 5 score. "Is it a 3 or a 4?" is a mood. "Did it include the order number?" is a fact.
4. Collect ten real examples from your own work. Include at least three awkward ones: the angry customer, the request outside your policy, the message that asks two things at once.
5. Run your current prompt on all ten and score each rule. That's your starting score.
6. Change one thing in the prompt, then run all ten again. Keep the change only if the score goes up and nothing that passed before now fails.
7. Save the prompt with a version number and date, next to the rules and the ten examples.
8. Re-run the ten when you change tools or the tool changes, and look at the rules again every few months.

I reuse a prompt for [task]. Right now I judge the results by feel: [what you want, in your own words, e.g. "replies should be better"].
Help me turn that into four to six rules, each one a yes-or-no question someone else could answer by looking at the output. Cover: length, what it must include, what it must never do, and tone (in a checkable way).
Ask me questions about my work first if you need to. Don't write the rules until you understand what good looks like for me.

Here are my rules:
[paste numbered rules]

Here are [number] outputs, numbered:
[paste outputs]

For each output and each rule, answer PASS or FAIL, with a few words of evidence quoted from the output. Then give the total passes per rule. Don't give an overall impression. Rules only.

Four rows from her version 1 scorecard. A simple grid like this, in a spreadsheet or on paper, is all you need.

Watch for hitting the rule and missing the point

Rules are a stand-in for what you actually want. A reply can be under 120 words, polite and correct, and still leave the customer feeling brushed off. When something passes every rule and still feels wrong, don't ignore the feeling: it's telling you a rule is missing. Add it, and add that example to your ten.

Common mistakes

• Testing on the easy examples only. The awkward ones are where prompts break.
• Changing three things at once, so you can't tell which one helped.
• Using a score out of five, which drifts with your mood.
• Throwing away the old version. Keep it, so you can go back.
• Never re-running the tests after the tool updates.