Docs navigation
Monitoring Your Agent

Evaluations

Evaluations are automatic checks that run when an agent finishes a turn. Use them to grade qualities that should be true across many tasks, such as helpfulness, policy compliance, or whether the agent asked for missing information.

How to use evaluations
  1. 01Open the agent's Evaluations area.
  2. 02Add the behavior you want to check and the scoring criteria.
  3. 03Enable the evaluation and review results on completed tasks.
The manifest list is authoritative
agent push sends the whole evals list, so removing an eval from the file archives it in the cloud. Push names what it will archive and refuses if the cloud holds an eval you have not pulled. Read verdicts from the terminal with brainbase agent eval runs. Full field reference: agent manifest → evals.
Model evaluations consume credits
Each enabled model-based evaluation runs a judge after the agent finishes a turn and uses Brainbase credits like another model call. The judge request and result are not added as chat messages, but their usage is recorded in the credit ledger. Additional enabled evaluations and occasional transcript summarization can add more calls. This does not delay the agent's primary response, but with no credits left or a spending limit reached the judge call is rejected and the evaluation is recorded as errored with no verdict. If a model provider omits token usage, Brainbase records a conservative estimate instead of leaving the evaluation unmetered.

What evaluations are for

  • Quality gates: Check whether the answer is complete, grounded, or useful.
  • Policy checks: Flag missing disclaimers, unsafe actions, or skipped approvals.
  • Workflow checks: Confirm the agent followed required steps before finishing.

Evaluation examples

  • answer-is-helpful: The reply directly addresses the user's question.
  • asks-for-missing-info: The agent asks a follow-up question when required inputs are absent.
  • uses-approved-tone: The response follows the team's tone and formatting rules.

API write credentials

Creating an eval with POST /v2/agents/{agent_id}/evals, or changing or deleting one with PATCH or DELETE /v2/agents/{agent_id}/evals/{eval_id}, requires permission to manage the agent. Supabase login sessions, MCP OAuth JWTs and user-wide PATs retain their existing access checks. A caller without access receives 404. PAT writes retain the existing publish scope rules: the control-plane guard enforces them in enforce mode and logs a would-be denial in shadow mode.

When both INTEGRATION_CONTROL_ENABLED and cutover:strict_admission are enabled, these three write endpoints refuse sandbox and other resource-scoped PATs, plus bb_live_… keys, with 403 and detail.code: agent_credential_refused. This refusal applies even when AUTH_SCOPE_ENFORCEMENT_CONTROL_PLANE=shadow. Use a person's login session or user-wide PAT with permission to manage the agent.

Strict admission is off by default. With either switch off, the existing write behavior applies. If the master switch is on but the strict-admission setting cannot be read, resource-credential writes return 503 with detail.code: origin_unavailable and Retry-After: 5; retry after five seconds. Listing or reading evals and listing eval runs keep their existing read permissions; strict admission adds no new restriction to those reads.

Best practices

  • Write criteria as observable outcomes, not vague preferences.
  • Keep each evaluation focused on one behavior.
  • Use stable slugs so results are easy to compare over time.
  • Pair evaluations with benchmarks when you need both automatic scoring and repeatable test cases.