🤖 Ghostwritten by Claude · Curated by Tom Hundley

This article was written by Claude and curated for publication by Tom Hundley.

---

LLM Evaluation: How to Know If Your AI Is Working

Moving from Vibes to Metrics.

The Vibe Check

Most teams evaluate their AI by asking it 5 questions, saying Looks good!, and shipping it.
Then users ask 500 questions, and the system fails.

The Two Types of Evals

1. Offline Evaluation (The Test Suite)

This runs before you deploy.

• The Golden Dataset: 50-100 QA pairs that represent ground truth.
• Metrics:
  • Exact Match: Did it output the specific ID?
  • Semantic Similarity: Is the meaning correct? (Evaluated by another LLM).
  • Hallucination Rate: Did it cite a fake source?

2. Online Evaluation (Monitoring)

This runs in production.

• User Feedback: Thumbs Up/Down buttons.
• Implicit Signals: Did the user copy/paste the code? Did they re-phrase the question (sign of frustration)?
• Drift Detection: Is the model starting to output shorter answers over time?

RAG-Specific Metrics (RAGAS)

For RAG systems, we use the RAGAS framework metrics:

1. Faithfulness: Is the answer derived only from the retrieved context?
2. Answer Relevance: Does the answer address the query?
3. Context Precision: Did we retrieve too much junk?
4. Context Recall: Did we miss the relevant doc?

Conclusion

If you arent logging your inputs and outputs, you arent doing engineering. Youre gambling. Build the eval pipeline before you build the feature.

---

*This concludes the Production AI Patterns series.*

How This Article Was Made

This article is a live example of the AI-enabled content workflow we build for clients.

Stage	Who	What
Research	Claude Opus 4.5	Analyzed current industry data, studies, and expert sources
Curation	Tom Hundley	Directed focus, validated relevance, ensured strategic alignment
Drafting	Claude Opus 4.5	Synthesized research into structured narrative
Fact-Check	Human + AI	All statistics linked to original sources below
Editorial	Tom Hundley	Final review for accuracy, tone, and value

The result: Research-backed content in a fraction of the time, with full transparency and human accountability.

Why We Work This Way

Were an AI enablement company. It would be strange if we didnt use AI to create content. But more importantly, we believe the future of professional content isnt AI vs. Human—its AI amplifying human expertise.

Every article we publish demonstrates the same workflow we help clients implement: AI handles the heavy lifting of research and drafting, humans provide direction, judgment, and accountability.

Want to build this capability for your team? Lets talk about AI enablement →

---