When to Eval

Relax. We're not going to explain this chapter like a textbook — we'll just talk it through so it's easy to follow.

Let's think about it this way for a second

Worth eval-ing: handbook Q&A, stopping when nothing's found, not leaking secrets, keeping the quiz format consistent — recurring tasks like these. Not worth it: a one-off poem, a joke, an exploratory chat — a vibe check is plenty for those. A chef checks the daily menu, but doesn't build a spreadsheet to track the decorations for a single wedding. Don't rerun every single card every time you tweak a prompt, context, or RAG setup — just pull back the 3 cards tied to the behavior you actually changed.

Let's connect it to everyday life

Good times to eval — after tweaking a prompt, after adding a new file, after adding a 'stop if nothing's found' rule. Bad times — every time you polish a sentence, every time you add an emoji. In your first week, stick to no more than 5 cards. You don't need a big framework or a nightly pipeline yet. Don't throw three different dishes into one pot and expect them all to turn out perfectly.

Only test behavior that recurs
After a fix, only rerun the cards tied to that change
Start with 5 cards

Let's try it together

text

Eval now:
- after a prompt rewrite
- after adding a new note
- after a “don’t guess” rule

Skip:
- one-off jokes
- pretty wording tweaks

You should see

You'll be able to tell whether a task is worth eval-ing or not.

5-Minute Try-It

List 5 of your own AI tasks. Split them into 3 that deserve an eval and 2 where a vibe check is enough, and explain why.

A Quick Word of Caution

More evals doesn't mean more safety. Cards nobody reads are just noise.