Human review
Human review records one person's judgment of a completed task. A review is a verdict for the whole task, backed by the transcript evidence you cite while you read it. It is written next to the transcript in the task detail, so judging a task and finding the exact step that went wrong happen in the same place.
Open a task in Tasks and select the Human review tab, after Transcript. The tab shows the evidence on the left and your review form on the right; on narrow screens the two stack and scroll. Switching to Transcript and back keeps what you have typed.
@brainbaselabs.com accounts. The behavior below applies only when Human review is enabled for your account. When it is off or unavailable, Tasks and the task detail look and work exactly as before. Saving a review also requires permission for the task's agent; the server enforces this even when the tab is visible.When Human review is off or unavailable, Settings → Roles hides review permissions from automatically generated descriptions and from notes about permissions kept on a role. Existing roles and grants, and any role names or descriptions you wrote, are unchanged.
What a review records
- Verdict: Looks right, Something's wrong, or Can't tell. The verdict is always for the whole task, never for a single turn or message.
- Note: Optional, except for Can't tell, which requires saying what is missing.
- Cited steps: Any transcript steps, from any turn, that you want to point at. A citation references the step by its transcript id, for example
T1·E03. - Coverage: The turns the verdict covers, recorded when you save. The footer shows it as
Saved · <verdict> · covers Turns 1–N.
The form footer always tells you where you stand: Nothing recorded yet, Not saved yet, Saved · <verdict> · covers Turns 1–N, or Unsaved changes.
The verdict vs turn evidence
A fixed strip at the top of the form reads Verdict for the whole task · Turns 1–N. Turns are evidence for the verdict; there is no per-turn verdict to give. On a multi-turn task the same one verdict covers every turn, including turns that went well and turns that did not.
- The evidence column shows Show evidence from: All turns / Turn 1 / Turn 2 …. All turns is the default, with a divider between turns. Your choice is remembered per task for your account.
- Each turn chip shows how long the turn took, how many of its steps you cited, how many reactions you left, and whether new evidence arrived since your saved review.
- Each step is a card with a Cite / Cited toggle. Citing a step adds it to your review; citing again removes it.
- Evidence is masked on the server before it reaches you. Anything you cannot see in full is labelled, for example a truncated result, so a gap never reads as a clean step.
Reactions are deliberately not part of this. A thumbs up or down is saved on one agent message, through its own Save button, and never decides the task. You can like a recovery message on a task whose overall verdict is Something's wrong; the two records stay separate, and a task with only reactions is not reviewed. See Reactions.
Reactions
A reaction is your thumbs up or thumbs down on one finished agent message, with an optional note. You leave it in Transcript, next to the message. It is about that message only: it never sets, changes or counts as the task's verdict, and it is saved separately from your review, with its own Save reaction button.
- Which messages: Only finished agent messages get thumbs. A message that is still being written, user messages, tool steps, the agent's thinking, and messages from sub-agents have none. A running task can be reacted to as soon as a message is finished; only the verdict waits for the run to end.
- Save a reaction: Click 👍 or 👎. A small editor opens with an optional note, What's good about it? or What's wrong with this message? Nothing is written until you select Save reaction. Cancel closes the editor and leaves any saved reaction as it was.
- Save and open Human review: Saves the reaction, then switches to the Human review tab. It is offered only once the task has finished running.
- Change or remove it: Click the other thumb to reopen the editor with your saved note, then save again. Click your saved thumb again to remove the reaction.
- One per message: You have at most one reaction on a message. If it was changed somewhere else while you were editing, the save is refused and the message shows the saved version.
Reactions are personal: the console only ever shows you your own. In the Human review tab, each turn chip counts your reactions in that turn, and each reacted step shows your thumb. Below the note, the form lists Your reactions to single messages (don't change the verdict), in transcript order, and is shown even while the verdict is locked. When a reaction points the other way from your verdict, the form says so, for example You liked T2·E04. The task can still be wrong overall. This is a reminder, not a warning; you can save the review as it is.
Saving or removing a reaction needs the same Review permission as saving a review, and the server checks it. A note left blank is saved as no note, and a note you write is masked before it is stored. In the Tasks list, a task that has reactions but no saved review shows Reactions only.
Where it first went wrong
When the verdict is Something's wrong, the form asks Where did it first go wrong? The picker lists every agent and tool step, grouped by turn; user messages are excluded. Picking a step cites it automatically, and removing that citation clears the pick. The choice is saved only as part of the review; the server rejects a first-wrong step that is not cited or that belongs to a different verdict.
Labels and Other
Labels are short tags your team defines to describe what happened in a task. They are optional for every verdict: the Labels field appears once you choose a verdict, and a review with no labels saves normally. Brainbase ships no fixed list of labels. Each agent has its own, and a new agent starts with none, so the field reads No labels for this agent yet until someone creates one. You can always describe the problem in the note instead.
- Pick labels: Type to search the agent's labels and pick as many as fit. Each picked label shows as a chip; hover a chip to see what the label means.
- Create a label: When nothing matches what you typed, choose Create "<name>", give it a one-line meaning, and select Create and add. The label is added to your review and becomes available to everyone reviewing this agent. Names are unique per agent regardless of capitalization, so creating a name that already exists adds the existing label rather than making a second one.
- Other…: When no label fits, choose Other… and describe what happened. The description is required: a review with Other and no description is not saved. Removing the Other chip removes the description too.
Manage labels, next to the field, opens the agent's full list. Each label shows its meaning and how many saved reviews currently use it. From there you can add a label, rename it or change its meaning, and retire or restore it.
- Renaming updates every review that already uses the label, including saved reviews in History.
- Retiring removes the label from the picker, but reviews that already have it keep it, shown as
<name> (retired). A retired label cannot be added to a review, and creating a new label with its name is refused until it is restored. - Restoring returns a retired label to the picker. If another label has taken its name in the meantime, restore is refused until one of them is renamed.
Adding a label needs the same Review permission as saving a review. Renaming, retiring and restoring need the separate Manage labels permission on your role; without it, Manage labels still shows the list but not those actions. Both are checked by the server. Labels belong to one agent only: a label from another agent cannot be saved on this agent's reviews. In History, each saved review shows its labels as chips, and Other with its description.
Drafts and saving
- What you type is kept while you move around: back to the Tasks list, into Transcript, or back again. A failed save keeps it too.
- Drafts are not written to disk. Closing or refreshing the browser with unsaved work asks you to confirm first.
- Drafts belong to one account and one agent at a time. Switching account or agent drops the previous drafts.
- Save commits on the first click after typing; you do not need to click away from the note first.
Before a save is accepted, the review has to be valid: a verdict must be chosen, Can't tell needs its note, a step cannot be cited twice, and a first-wrong step only fits Something's wrong. Two situations are refused until you act: if the task's evidence changed while you were reviewing, the save asks you to check the evidence and save again; and if your review was saved somewhere else after you started editing, the save conflicts and Discard your changes loads the newer review. A retried save is never recorded twice, and every failed save keeps your draft exactly as you left it.
Running tasks and new turns
While a task is still running, the verdict is locked: it unlocks when the run finishes and will cover every turn up to then. If more turns arrive later, you are asked to re-check. Citing steps stays available while the verdict is locked, so you can collect evidence while the run is still going.
When a finished task gets new turns after your saved review, the tab label switches to re-check. Re-saving updates your verdict's coverage to include the new turns. You do not have to change anything first: the save button reads Confirm <verdict> for Turns 1–N and saves the same verdict, note and citations over every current turn.
Blind review
While you review a task, its automatic evaluation results are hidden from you, so the verdict you save is your own judgment rather than a copy of the eval's. Hiding starts when you open the Human review tab on a task and lasts while you stay on that task, including when you switch to Transcript. It ends when you save a review or leave the task. It applies only to you: teammates who are not reviewing the task see its results as usual.
- Tasks list: The task's Evals cell reads Hidden, with no results, counts or tooltip. An eval detail that was open for the task closes.
- Task detail: The evals card reads
Evals hidden while you review this task, so your verdict is independent. - API, CLI and MCP: Reading the task's eval runs returns one hidden marker for the task in place of its results, with every verdict field empty. Older clients that do not know the marker still receive no verdict. See brainbase agent eval runs.
If Brainbase cannot confirm whether the results should be hidden, for example during an outage, they stay hidden.
To see the results before you save, select Show anyway on the evals card. Brainbase records that you looked before it shows anything; if that record cannot be written, the results stay hidden and the card reads Couldn't record that you looked at the evals, so they stay hidden. Try again. Show anyway reveals that task's eval results only, never other reviewers' reviews or check flags. Reading a task's eval results some other way before your first save, for example through the API after you left the task, is recorded the same way.
Check questions
After you save a Looks right or Something's wrong verdict, Brainbase compares it with the task's pass/fail evals on the turns your review covers. When an eval disagrees, for example it failed a task you said looks right, the form asks whether the check is wrong: Your verdict differs from the eval. Only the newest pass/fail result on each turn counts. Rating and category evals, results that are still running or errored, and Can't tell reviews never raise a question.
- Yes, flag the check flags the eval for its owner, who sees it in the eval's results on the agent's Evaluations page. See Review flags.
- No, they measure different things records your answer and flags nothing.
- Change next to an answer lets you answer again. Changing Yes to No removes the flag. Each answer applies only to that check.
- When several evals disagree, pick the check you mean first; the buttons stay disabled until you do.
An eval that finishes after you save can still raise a question. The question compares your verdict with the result the eval had at that moment; a later rerun of the eval does not change it. Saving the review again replaces any unanswered questions with those for the new save, and answers you already gave are kept.
Check questions · N, in the Tasks list and above History, counts the questions waiting on your saved reviews. It opens a panel titled with the task's name and the saved revision it is about, with Previous task and Next task to move between tasks. Closing the panel is not an answer; the questions stay waiting. If you saved the review again since a question was raised, the panel says the questions no longer apply.
History
The History sub-tab lists saved reviews only; an unsaved draft never appears in it. Other reviewers' reviews stay hidden until you save your own, shown only as how many are hidden. Once visible, an entry shows the reviewer's verdict, note, cited steps and coverage. The task's automatic evaluation result may appear alongside for reference, labelled not a human review.
Not included yet
A saved review currently records the verdict, note, cited steps, first-wrong step, labels and coverage. Approval of a written correction, importing a review into a benchmark, final decisions when reviewers disagree, reviewing many tasks as a set, and notifications are not part of this yet.