Benchmarks
Benchmarks are reusable test suites for an agent. Run the same representative tasks after changing instructions, tools, skills, models, or runtime settings to measure improvements and catch regressions.
- 01Open an agent, select Benchmarks, and create or import a benchmark.
- 02Add cases and checks, then review and start a run.
- 03Inspect results, pin a baseline, change the agent, and rerun to find regressions.
How benchmarks work
Each benchmark belongs to one agent and has an editable draft. Select Save as new version, or Save and run when the draft has changes, to create an immutable version. A reviewed and confirmed run plan executes that version, and later edits do not change earlier versions or runs.
| Concept | Description |
|---|---|
| Case | One representative task with an instruction and optional expected result, variables, files, and metadata. |
| Check | A rule that grades the final output, behavior, resulting files, commands, or overall quality. |
| Version | An immutable snapshot of the benchmark used to make runs reproducible. |
| Run | One execution of selected cases against the current agent and any optional model configurations. |
| Baseline | A pinned result used to measure later improvements and regressions. |
Benchmark tasks run through the agent's normal model and tools and consume credits like other agent work. They remain separate from the agent's regular task history.
Build a benchmark
Create or import
Start from an empty benchmark, or import JSON, YAML, CSV, a Brainbase bundle, a public Hugging Face dataset, or a completed task. When importing a completed task, review the captured input and explicitly approve its output before using it as the expected result.
Through the API, manifest, CSV, and bundle imports return the new benchmark and its first immutable revision. Hugging Face and completed-task imports use separate preview and materialize requests. For Hugging Face, materialization repeats the preview's expected commit, materialization digest, and content digest. For completed tasks, it repeats the expected source and content digests and sets confirm_output_is_correct to true. You can validate a manifest without saving it at POST /v2/agents/{agent_id}/benchmarks/validate. Draft updates use PATCH /v2/agents/{agent_id}/benchmarks/{benchmark_id} with the complete manifest and latest expected_version.
Export a benchmark as JSON or YAML with GET /v2/agents/{agent_id}/benchmarks/{benchmark_id}/export?format=json or format=yaml. Add revision_id to export a specific immutable version instead of the current draft.
Add cases
Cases should represent real work the agent needs to perform. A case can include an instruction, expected result, variables, starting files, metadata, and check overrides. Files are copied into the workspace before the agent starts.
Configure checks
| Check | Use it to |
|---|---|
| Output | Check the agent's final response for required content or structure. |
| Behavior | Check the actions and tool activity recorded during the task. |
| File | Inspect files produced or changed in the final workspace. |
| Command | Run a command or test package against the final workspace. |
| Model judge | Use a model to grade the completed task against written criteria. |
Turn on Decides pass when a failed check should fail the case. Select one numeric check as the Headline score when you want trends and comparisons to follow that score. Without one, Brainbase uses pass rate as the headline result.
Configure the environment
In Settings, you can add setup commands, time limits, required credential names, and files to retain after a run. Required credential names are checked before the run starts. Credential values stay in the agent's credential store, are not saved in the benchmark definition or run plan, and are provided to the agent through its normal runtime environment during execution.
Run a benchmark
- Select Run benchmark. If you have unpublished edits, the button is labeled Save and run and saves a new immutable version first.
- Choose the cases and optional model configurations to test.
- Set repetitions to measure consistency and concurrency to control how many cases run at once.
- Select Review run and inspect the version, number of case runs, models, credentials, time limits, and estimated cost when one is available.
- Acknowledge the review and select Confirm and start run.
While a run is active, the run page shows progress by configuration and case. You can cancel remaining work from the run page. Completed agent work is not silently rerun when a quality check fails.
Understand results
Results keep execution health separate from agent quality. This prevents a provider or evaluation error from being presented as though the agent simply answered incorrectly.
| Result | Meaning |
|---|---|
| Coverage | How much of the planned work completed with usable evidence. |
| Could not run | How many planned case runs failed before producing usable evidence. |
| Cancelled | How many planned case runs were stopped before completion. |
| Pass rate | The percentage of measured case runs that passed every required check. |
| Headline score | The normalized score from the check selected for trends and comparisons. |
| Stability | Whether repeated runs changed pass/fail outcome or primary score. |
| Case change | Whether a case improved, regressed, stayed the same, or was not scored. |
Open a case result to inspect the final output, check results, files, logs, and execution details. Pin an eligible configuration result as its baseline, add a Run name or change note, then rerun the benchmark after changing the agent.
Brainbase labels comparisons as Compared, Partial, Not comparable, or No baseline. Changing cases or checks can reduce comparison coverage, so the interface reports the comparable coverage and added or removed cases instead of presenting a misleading continuous trend.
API clients can list run history at GET /v2/agents/{agent_id}/benchmarks/runs, fetch a run's aggregate and case-level results from GET /v2/agents/{agent_id}/benchmarks/runs/{run_id}/results, update its name and change note with PATCH /v2/agents/{agent_id}/benchmarks/runs/{run_id} using label and change_note, and manage a configuration baseline with PUT or DELETE at /v2/agents/{agent_id}/benchmarks/{benchmark_id}/baselines/{target_key}. Setting one with PUT requires baseline_target_id. Use GET /v2/agents/{agent_id}/benchmarks/analytics/summaries?benchmark_id={benchmark_id} for library summaries and GET /v2/agents/{agent_id}/benchmarks/{benchmark_id}/analytics for one benchmark's trends.
Benchmarks and evaluations
| Use | Evaluations | Benchmarks |
|---|---|---|
| When they run | Automatically after normal agent turns. | When you launch a versioned test run. |
| Inputs | Whatever task the user submitted. | A fixed set of reusable cases. |
| Best for | Monitoring behavior across real usage. | Regression testing, model comparison, and controlled experiments. |
| Results | A check attached to an individual task. | Scores, evidence, trends, baselines, and case-level comparisons. |
Best practices
- Keep cases realistic; overly clean synthetic examples miss important failure modes.
- Add a case whenever a production task reveals behavior you need to preserve or improve.
- Use stable case and check names so results remain understandable over time.
- Separate required checks from informational diagnostics.
- Use repetitions for nondeterministic tasks instead of trusting a single run.
- Record what changed on the agent before rerunning, then compare against a pinned baseline.
- Review cases imported from real tasks for sensitive information before saving them.