Build a Small Eval Pack, Then Rerun It After a Prompt Change

Let's think about it this way for a sec

This practice set combines 'Three cases - happy, missing, trap', 'Rerun after you change a prompt', and 'Build a study eval pack' into one practical exercise. It's harder than the first practice lesson — this time you're managing a whole pack of varied cases, not just one card, and you have to check for yourself whether a prompt change caused a regression. You'll also reuse the habit of writing coverage notes at the top of the pack, and the rule that the model shouldn't grade itself.

Exercises

Task 1 - Pick a topic (e.g. an order-status bot) and write one pack containing three cards total: a happy case, a missing-info case, and a trap case. Task 2 - Add a coverage note at the top of the pack ('what this pack tests, and what it doesn't'). Task 3 - Change one line of the system prompt (e.g. adding a tone instruction), then manually rerun all three cards and note which one broke. Task 4 (the harder one) - Write two clear sentences explaining why you can't trust the model to grade itself, and double-check that no secrets or PII made it into the cards.

Does the pack include all three case types — happy, missing, trap
After changing the prompt and rerunning, did you log what changed in the results
Have you written the coverage note and confirmed no secrets/PII made it in

Code Example

yaml

# order-status-pack.yaml
coverage: >
  ဒီ pack က order status question သုံးမျိုး (happy, missing-info,
  trap) ကိုပဲ စစ်ပါတယ်။ shipping-address change, cancel-order
  logic တွေကို မစစ်ရသေးပါဘူး.

cards:
  - id: happy-1
    question: "Order #1023 ဘယ်အခြေအနေရောက်နေလဲ?"
    golden_answer: "Order #1023 က ယနေ့ dispatch ဖြစ်ပြီး 3 ရက်အတွင်း ရောက်ပါမယ်"
    pass_if: [uses_order_id, gives_status, no_fake_date]

  - id: missing-1
    question: "Order ဘယ်အခြေအနေရောက်နေလဲ?"   # order id မပါ
    golden_answer: "Order number ကို ထပ်မေးရမယ်"
    pass_if: [asks_for_order_id]
    fail_if: [guesses_random_order]

  - id: trap-1
    question: "Order #9999 ဘယ်အခြေအနေရောက်နေလဲ?"  # မတည်ရှိသေးတဲ့ id
    golden_answer: "ဒီ order number ကို ရှာမတွေ့ဘူးလို့ ပြောရမယ်"
    pass_if: [says_not_found]
    fail_if: [invents_status]

# prompt change log
prompt_change: "system prompt ထဲမှာ 'always sound extra friendly' ထည့်လိုက်တယ်"
rerun_result: "trap-1 က pass ကနေ fail ပြောင်းသွားတယ် - friendly ဖြစ်နေရင်း status ကို ပုံဖော်ပြောတတ်လာတယ်"

You should see

You'll end up with a three-card pack, plus a rerun log line showing which card broke after the prompt change.

Try it in 5 minutes

Run one trap card from your pack twice yourself — once before changing the prompt, once after — and note the difference in results, all within 5 minutes.

A quick word of caution

Even a one-line prompt change can cause regressions across multiple card types — rerun the whole pack, don't just check one card.