Here is how most LLM features get built. You write a prompt, run it against the handful of inputs sitting in your terminal, nudge the wording until those look good, and ship behind a flag. The regression shows up in production, usually inside a day, on an input that looks nothing like your handful.
A recent dev.to post named the trap perfectly. Every LLM feature the author shipped went through the same loop: tweak the prompt, try five examples, feel good, ship, then get a bug report the five examples never covered. The work looked like engineering because the prompts lived in git and there was a flag in the config. The missing piece was a scoreboard.
Write the eval before the prompt
Flip the order. Define what good means before you write a single line of prompt. A customer-support summary should stay faithful to the input and under 80 words, a SQL generator should produce a query that executes against the schema, and a multi-step agent should pick the correct tool on the first try. The metric has to be measurable. A gold summary scored with ROUGE-L above 0.4 is a target you can track; ‘helpful’ is a feeling.
The minimum viable eval for an agent is small on purpose. It is a JSONL file, two kinds of checks, a script, and a CI job. Each case has an input, the expected answer, and a note on where it came from. The script loops the cases, calls the agent, counts hits, and prints a percentage with the failures listed underneath. The CI job runs it.
Keep the dataset honest and representative. Twenty to fifty inputs that look like what your agent will really face, including the messy edge cases, is the highest-leverage thing you will build, and a small set of representative cases beats a big set of easy ones.
Two kinds of checks do most of the work. Deterministic checks handle the mechanical stuff and are cheap to write, and an LLM judge with a written rubric handles task completion and reasoning quality. The rubric is where teams get sloppy. Vague ‘rate this 1 to 10’ instructions produce noise, while explicit score thresholds such as 0.8 to 1.0 meaning fully resolved produce consistent, reproducible scores.
Grow it from real failures
Where do cases come from? Real inputs come first, with names and addresses replaced, and then every bug report goes in word for word.
This is the flywheel. Every production failure becomes a new row in the JSONL. The agent that bit you yesterday cannot bite you twice, because the fix now has to pass the exact case that caught it. For agents specifically, keep feeding the set. Start with 50 golden traces from production traffic and grow the dataset weekly through a structured review ritual.
Then gate it in CI. Set a regression threshold so that if the overall pass rate drops by more than 3 percent, the deploy is blocked. Now a prompt change is a pull request with a number attached. Green means ship. Red means you broke something you already paid to learn.
The cost of doing nothing
Here is the CODN math, and it is brutal once you run it. Skipping the eval set feels free. It saves you an afternoon. Then it charges interest forever.