Docs navigation
Monitoring Your Agent

Evaluations

Evaluations are automatic checks that run when an agent finishes a turn. Use them to grade qualities that should be true across many tasks, such as helpfulness, policy compliance, or whether the agent asked for missing information.

How to use evaluations
  1. 01Open the agent's Evaluations area.
  2. 02Add the behavior you want to check and the scoring criteria.
  3. 03Enable the evaluation and review results on completed tasks.
The manifest list is authoritative
agent push sends the whole evals list, so removing an eval from the file archives it in the cloud. Push names what it will archive and refuses if the cloud holds an eval you have not pulled. Read verdicts from the terminal with brainbase agent eval runs. Full field reference: agent manifest → evals.
Model evaluations consume credits
Each enabled model-based evaluation runs a judge after the agent finishes a turn and uses Brainbase credits like another model call. The judge request and result are not added as chat messages, but their usage is recorded in the credit ledger. Additional enabled evaluations and occasional transcript summarization can add more calls. This does not delay the agent's primary response, but with no credits left or a spending limit reached the judge call is rejected and the evaluation is recorded as errored with no verdict. If a model provider omits token usage, Brainbase records a conservative estimate instead of leaving the evaluation unmetered.

What evaluations are for

  • Quality gates: Check whether the answer is complete, grounded, or useful.
  • Policy checks: Flag missing disclaimers, unsafe actions, or skipped approvals.
  • Workflow checks: Confirm the agent followed required steps before finishing.

Evaluation examples

  • answer-is-helpful: The reply directly addresses the user's question.
  • asks-for-missing-info: The agent asks a follow-up question when required inputs are absent.
  • uses-approved-tone: The response follows the team's tone and formatting rules.

Best practices

  • Write criteria as observable outcomes, not vague preferences.
  • Keep each evaluation focused on one behavior.
  • Use stable slugs so results are easy to compare over time.
  • Pair evaluations with benchmarks when you need both automatic scoring and repeatable test cases.