Compare AI tools on a task you actually do
You'll end up with: A choice of AI tool for one recurring task, based on a rubric you wrote before seeing any answers rather than habit, rankings or which answer sounded most confident.
Tools: Two or three AI chat tools, for example Claude, ChatGPT, Gemini or Perplexity. Free tiers are fine. Only paste work content into tools your organisation allows.
Why this matters
Most people pick one AI tool and stay with it. That’s reasonable, but the tools keep changing, and the one that’s best at drafting a tricky email may not be best at summarising a long report. Benchmark scores are close at the top and don’t tell you which tool fits your work. To find out, run your own task in more than one tool and judge the answers against what you need.
Smallest useful build
A scored comparison of one real task across two or three tools, plus one sentence saying which tool you’ll use for that task and why. It takes about 30 minutes.
Sample inputs
Use a task you do often, such as a prompt that worked well for you in an earlier project. If you need one, use this invented paragraph:
From 1 March, expense claims must be submitted within 30 days of purchase. Claims older than 30 days will need a manager's written approval. Receipts under $25 no longer need to be scanned.Steps
-
Write your rubric before you run anything. List three or four things a good answer must do, such as “plain enough for new staff”, “keeps all three rules”, “under 120 words”. Write down which tool you expect to win. Then check that each line can be scored:
Here is my rubric for judging an answer: [paste rubric]. Don't add criteria or improve it. Just tell me if any line is too vague for two people to score it the same way. -
Run the exact same prompt in each tool. Open a new chat in each one and paste identical wording.
Rewrite this policy change as a short, friendly message to staff. Keep every rule exactly as stated. End with who to ask if they have questions. Text: [paste text] -
Score each answer yourself. Mark each criterion for each tool before you ask any AI’s opinion. To spot differences you missed, ask one tool:
Here are two answers to the same request. For each criterion in my rubric, quote the one sentence where the answers differ most. Don't pick a winner. Rubric: [paste]. Answer A: [paste]. Answer B: [paste] -
Write your one-sentence verdict. “For this task I’ll use ___ because ___.” Compare it with your prediction from step 1.
Check it
- Every score points to something specific in the answer, not a general impression.
- Each tool kept all three rules. Check against the original text.
- Your verdict names a rubric criterion, not “it sounded better”.
Break it
Run the same prompt a second time in your winning tool. If its two answers differ as much as the gap between tools, your comparison is mostly noise. Run each tool two more times before you decide.
Reflect
- Did the result match your prediction? If not, what had you assumed about each tool?
- Did any tool flatter your text or agree too easily when you pushed back?
Next iteration
Repeat with a different kind of task, such as summarising a long document or checking a fact against sources, and see whether the same tool still wins. Keep your rubrics, and rerun them every few months as the tools change.
Connections
Skills it uses
Concepts it practises
Competencies
Background reading
Do this one first
How we know
- The claim
- Running the same prompt in more than one AI tool shows real differences in fit, tone and how much the tool pushes back. Those differences matter more for everyday work than published benchmark scores.
- The evidence
- Inspired by the compare-across-models lesson in WDAI's AI Basics course. The evidence is WDAI's experience teaching it to beginners, not a study.
- The limits
- One run of one task is a small sample, and answers vary from run to run. Tools change often, so a result can go stale within months. This card makes no claim about which tool is best.
Plain Markdown for agents: /projects/prj-compare-models-on-a-real-task.md