Use case
Bake off harnesses on one task
Give the same broken function to several coding-agent harnesses, each in its own isolated sandbox, and ask each to fix it. When a run finishes, download that harness's solution.py and grade it against a hidden test suite the agent never saw. Rank by score, break ties by wall-time, and tear the sandboxes down. Because the agents only see the spec and never the tests, the score is hard to game.
You'll need an API key
Create one at app.brainbaselabs.com and export it as
BRAINBASE_API_KEY. The examples are Python; install the one dependency with pip install requests. See Authentication for the header format.What you'll build
- One task, a buggy binary search, sent to four harnesses at once.
- A local grader that runs each returned solution against twelve hidden cases in a subprocess.
- A ranked table, and a teardown pass so idle sandboxes stop costing credits.
Set up the shared bits once.
import concurrent.futures as cf
import os, subprocess, sys, tempfile, time
from pathlib import Path
import requests
BASE = "https://api.brainbaselabs.com"
HEADERS = {"Authorization": f"Bearer {os.environ['BRAINBASE_API_KEY']}"}
TERMINAL = {"success", "fail", "need_more_info", "idle"}
# harness -> model (None uses the harness default). Not every harness runs on
# every provider; an unavailable one shows up as an error row, not a crash.
MATRIX = {
"claude_code": "claude-sonnet-5",
"codex": None,
"opencode": "zai-org/GLM-5.2",
"qwen": "claude-sonnet-5",
}
1. The task and the hidden tests
The task ships the buggy function in the prompt and asks for a fixed solution.py with the same name and signature. The hidden cases stay in your script. The buggy version behaves like bisect_right: it never returns -1 and lands one past the target, so it scores 0/12 against the cases below.
BUGGY = '''def search(arr, target):
"""Return the index of target in the sorted list arr, or -1 if absent."""
lo, hi = 0, len(arr)
while lo < hi:
mid = (lo + hi) // 2
if arr[mid] <= target:
lo = mid + 1
else:
hi = mid
return lo
'''
TASK = (
"A file solution.py contains this buggy binary search:\n\n"
f"{BUGGY}\n"
"It is wrong. Rewrite solution.py so that search(arr, target) (keep that "
"exact name and signature) returns the index of target in the ascending, "
"possibly-duplicate sorted list arr, or -1 if target is not there. "
"Requirements:\n"
" - return -1 when target is not present (empty list, or below or above "
"every element);\n"
" - when target appears more than once, return the LEFTMOST index;\n"
" - keep it O(log n).\n"
"Write the fixed function to solution.py in the workspace root. No extra "
"prints, and do not rename the function."
)
# Hidden tests: (args, expected). The agent never sees these.
CASES = [
(([1, 3, 5, 7], 5), 2), # found, unique
(([1, 3, 5], 4), -1), # not found, in range
(([2, 4, 6], 1), -1), # not found, below all
(([2, 4, 6], 9), -1), # not found, above all
(([], 3), -1), # empty list
(([1, 2, 2, 2, 3], 2), 1), # duplicates -> leftmost
(([5], 5), 0), # single, found
(([5], 4), -1), # single, not found
(([1, 2, 3], 1), 0), # first element
(([1, 2, 3], 3), 2), # last element
(([2, 2, 2], 2), 0), # all duplicates
(([-5, -3, 0, 4], -3), 1), # negatives
]
The grader writes the returned source to a temp dir and runs it against the cases in a subprocess with a 15-second timeout. That bounds how long a slow solution can stall the grader; it is not a sandbox — read the warning below before you run it. The run_one step below calls grade, so include this helper (in the expandable block) in your script.
grade(solution_src) -> (passed, total, note), required by run_one
def grade(solution_src):
"""Run a solution.py source against CASES; return (passed, total, note)."""
total = len(CASES)
with tempfile.TemporaryDirectory() as d:
(Path(d) / "solution.py").write_text(solution_src)
grader = (
"import importlib.util as u\n"
"s = u.spec_from_file_location('solution', 'solution.py')\n"
"m = u.module_from_spec(s); s.loader.exec_module(m)\n"
f"CASES = {CASES!r}\n"
"p = 0\n"
"for args, exp in CASES:\n"
" try:\n"
" ok = m.search(*args) == exp\n"
" except Exception:\n"
" ok = False\n"
" p += 1 if ok else 0\n"
"print(f'SCORE {p}/{len(CASES)}')\n"
)
(Path(d) / "_grade.py").write_text(grader)
try:
r = subprocess.run(
[sys.executable, "-I", "_grade.py"], cwd=d,
capture_output=True, text=True, timeout=15,
)
except subprocess.TimeoutExpired:
return 0, total, "timeout"
for line in reversed(r.stdout.splitlines()):
if line.startswith("SCORE"):
try:
return int(line.split()[1].split("/")[0]), total, "ok"
except (IndexError, ValueError):
break
return 0, total, "error"
This runs model-written code with your OS permissions
grade executes whatever each harness returned as solution.py, as your user, on the machine running this script. Nothing here contains it: the code can read and write your files, use the network, and start processes that outlive the 15-second timeout. -I only stops it importing modules from the working directory and your user site-packages. Run this script on a disposable machine — a throwaway VM, a container with no mounts, or a Brainbase sandbox of your own — never on a workstation with credentials on it. Grading cannot move into the harness's sandbox: the API has no endpoint that runs a command there, and the hidden tests have to stay hidden from the agent.2. Run every harness in its own sandbox
Each harness gets one POST /v2/threads with the same task. Wait for the turn to settle, then download solution.py. File operations live under /v2/tasks/{thread_id}, and the path is relative to the workspace root, so pass solution.py, not an absolute path.
def wait(thread_id, poll_s=6, timeout_s=900):
deadline = time.time() + timeout_s
while time.time() < deadline:
thread = requests.get(f"{BASE}/v2/threads/{thread_id}", headers=HEADERS).json()
if thread["status"] in TERMINAL:
return thread
time.sleep(poll_s)
raise TimeoutError("thread did not settle")
def run_one(harness, model):
"""Fix the bug on one harness, then grade its solution.py."""
t0 = time.time()
row = {"harness": harness, "model": model or "(default)", "status": "error",
"passed": 0, "total": len(CASES), "note": "-", "secs": 0.0, "machine_id": None}
try:
agent = {"harness": harness}
if model:
agent["model"] = model
created = requests.post(
f"{BASE}/v2/threads", headers=HEADERS,
json={"agent": agent, "input": TASK, "title": f"bakeoff: {harness}"},
)
created.raise_for_status()
tid = created.json()["thread_id"]
thread = wait(tid)
row["status"] = thread.get("status", "?")
row["machine_id"] = thread.get("machine_id")
dl = requests.get(f"{BASE}/v2/tasks/{tid}/files/download",
headers=HEADERS, params={"path": "solution.py"})
if dl.status_code >= 400:
row["note"] = "no-file"
return row
row["passed"], row["total"], row["note"] = grade(dl.content.decode("utf-8", "replace"))
except Exception as exc:
row["note"] = f"error: {str(exc)[:40]}"
finally:
row["secs"] = round(time.time() - t0, 1)
return row
rows = []
with cf.ThreadPoolExecutor(max_workers=len(MATRIX)) as pool:
futures = [pool.submit(run_one, h, m) for h, m in MATRIX.items()]
for fut in cf.as_completed(futures):
row = fut.result()
rows.append(row)
print(f" finished {row['harness']:12} {row['status']:8} "
f"{row['passed']}/{row['total']} {row['secs']}s")
Files are under /v2/tasks, not /v2/threads
The same id names both a thread and its task, but file operations (
files/download, files/stat, files/tree) live under /v2/tasks/{id}/files/*. Reaching for them under /v2/threads/{id} returns 404.3. Rank and tear down
Sort by score first, then by wall-time, print the board, and delete every machine so idle sandboxes stop burning credits.
rows.sort(key=lambda r: (-r["passed"], r["secs"]))
print(f"\n{'harness':12} {'model':22} {'status':9} {'score':7} {'secs':>6}")
print("-" * 60)
for r in rows:
print(f"{r['harness']:12} {r['model']:22} {r['status']:9} "
f"{str(r['passed']) + '/' + str(r['total']):7} {r['secs']:>6}")
for r in rows:
if r.get("machine_id"):
requests.delete(f"{BASE}/v2/machines/{r['machine_id']}", headers=HEADERS)
That's the whole loop: one task run on several harnesses, scored the same way. Swap in your own broken function and cases to benchmark them on the work you care about.