Prompts degrade. A prompt that reliably produced good output last month may produce noticeably worse output after a model update, after your underlying data changed, or after a well-intentioned tweak that introduced an unintended side effect. Without a way to detect these regressions automatically, you discover them the way nobody wants to — in production, from a user, after the degraded output has already caused a problem.
Software engineers have long used automated test suites to catch regressions before they reach users. The same principle applies to AI prompts, and building a basic prompt test suite is more accessible than it sounds. This guide covers what a prompt test suite is, what it contains, and how to build one for the prompts that matter most in your workflow.
What a Prompt Regression Looks Like
Prompt regressions come in several forms, not all of them obvious. A model update might change the default verbosity of responses — outputs that were previously concise become longer and require more editing. A prompt that worked well with one version of your input data might start failing when the data structure changes slightly. A tweak to the system prompt intended to improve one aspect of the output inadvertently breaks formatting in another section. The underlying model changes in a way that affects how it handles a specific instruction you’ve been relying on.
In each case, the problem isn’t visible until you run the prompt on real inputs and compare the output to what you expected. If you’re running prompts in production without any automated quality checking, these regressions may go undetected for days or weeks — producing subtly worse outputs at scale before anyone notices and investigates.
The Core Concept: Golden Examples and Checks
A prompt test suite has two components: a set of test cases (input/expected output pairs, known as golden examples) and a set of checks (automated evaluations that verify the output meets defined criteria). Together they let you run your prompt against known inputs, compare the outputs to your quality criteria, and flag any case where the prompt is no longer producing acceptable results.
Golden examples are collected from your existing production outputs — cases where you know the output quality was good. For each case, you note what about the output made it good: the structure it used, the information it contained, the length it hit, the tone it maintained. These observations become your check criteria — the automated assertions that verify a new output meets the same standards as the known-good historical examples.
⚙️ Building Your First Prompt Test Suite
Writing Checks That Can Be Automated
The most reliable automated checks are deterministic — they either pass or fail based on objective criteria that don’t require human judgment. String presence checks (“does the output contain the phrase ‘in summary’?”), length checks (“is the output between 200 and 400 words?”), format checks (“does the output parse as valid JSON?”), and structure checks (“does the output contain exactly three numbered sections?”) are all fully automatable and catch a large proportion of real-world regressions.
Regular expressions extend deterministic checks to more complex patterns: checking that dates are in a specific format, that required fields are present in a structured output, that specific prohibited phrases don’t appear. These checks require some initial investment to write but run reliably at zero marginal cost per test execution, which makes them the right foundation for a test suite before adding more expensive human or LLM-based evaluation.
For quality dimensions that can’t be captured by deterministic checks — whether the tone is appropriate, whether the logical flow is sound, whether the output addresses the actual question — LLM-as-judge evaluation provides automated coverage at the cost of some evaluation variability. The pattern: after generating the test output, make a second API call with a prompt like “evaluate whether this output [specific quality criterion] on a scale of 1–5, respond with only the number.” Aggregate these scores across test cases and flag when the average drops significantly. Less reliable than deterministic checks but more scalable than human review for every regression check run.
🧪 Types of Checks to Include in a Prompt Test Suite
What to Test First
The prompts worth testing first are the ones where a regression would cause the most damage — prompts that run at high volume in production, prompts that produce outputs that go directly to customers or stakeholders without human review, and prompts where the output format is load-bearing for downstream systems that parse it. A prompt that generates a JSON payload consumed by another system needs a test that validates the JSON structure — if the prompt starts producing malformed JSON, the downstream system breaks in ways that may not be immediately obvious.
The prompts worth testing second are the ones where you have personally observed failures in the past. Every time a prompt produces an output that makes you think “that’s wrong, I should fix the prompt,” the fix should be accompanied by a test case that would have caught the original failure. Test suites built this way — growing from observed failures rather than theoretical coverage — tend to be more practically useful than suites built from first principles, because they test the failure modes that actually occur rather than the ones that seem plausible.
Tools for Running Prompt Tests
PromptFoo is an open-source CLI tool specifically designed for prompt testing and evaluation — it supports defining test cases in YAML, running them against multiple models or prompt versions simultaneously, and evaluating outputs with a range of check types including LLM-as-judge. For teams with developer resources who want a dedicated tool, PromptFoo provides the most purpose-built feature set for prompt regression testing.
Braintrust provides a full evaluation and observability platform — logging production outputs, running evaluations, comparing performance across prompt versions, and integrating with CI/CD pipelines. It’s the more complete solution for teams that want both regression testing and production monitoring, at the cost of more integration work than PromptFoo’s simpler CLI approach.
For teams without developer resources or who want to start simply, a spreadsheet of test cases with manual checks run periodically is better than no testing at all. The value of a test suite comes from its consistent application, not from its sophistication — a simple suite that runs every time you update a critical prompt catches more regressions than a sophisticated suite that rarely gets run because it’s inconvenient to execute.
Maintaining the Test Suite Over Time
A test suite that doesn’t grow alongside the prompts it tests becomes less useful over time. Every new failure mode observed in production should generate a new test case. Every new prompt added to a critical workflow should have at least three to five test cases covering its expected behaviour. And periodically — every quarter or after major model updates — the golden examples themselves should be reviewed: are they still representative of what you consider good output, or have your standards evolved in ways the test suite doesn’t yet reflect? That review, done deliberately rather than reactively, keeps the test suite honest about what it’s actually measuring and ensures it continues to reflect current quality standards rather than historical ones.