Fix a failing test suite
Hand a coding agent a small Python project whose pytest suite is red on purpose. It runs the tests, tracks down the bug, fixes it, and re-runs until the suite is green. Then a second message adds a feature on the same warm sandbox. Every step is one API call, and nothing is provisioned by hand.
BRAINBASE_API_KEY. The examples below are Python; install the one dependency with pip install requests. Every call is a plain HTTP request, so any language works. See Authentication for the header format.What you'll build
- A sandbox seeded with a buggy
roman.pyand atest_roman.pythat fails on the subtractive cases (IV, IX, and the year in MCMXCIV). - A first turn where the agent turns the suite green, editing only implementation code.
- A second turn on the same sandbox that adds an inverse
int_to_romanfunction with its own tests.
Set up the shared client bits once. Every snippet below reuses BASE and HEADERS.
import json, os, time
import requests
BASE = "https://api.brainbaselabs.com"
HEADERS = {"Authorization": f"Bearer {os.environ['BRAINBASE_API_KEY']}"}
TERMINAL = {"success", "fail", "need_more_info", "idle"}
1. Seed the project and start the agent
The entrypoint field is bash that runs inside the sandbox before the agent wakes up, with cwd=/workspace. Use it to plant the project. The seed writes a roman_to_int that only ever adds symbol values, so subtractive pairs read too high and three of the five tests break, then it installs pytest.
set -e
cat > /workspace/roman.py <<'PY'
def roman_to_int(s):
values = {"I": 1, "V": 5, "X": 10, "L": 50, "C": 100, "D": 500, "M": 1000}
total = 0
for ch in s:
total += values[ch]
return total
PY
cat > /workspace/test_roman.py <<'PY'
from roman import roman_to_int
def test_plain_additive():
assert roman_to_int("III") == 3
def test_subtractive_four():
assert roman_to_int("IV") == 4
def test_subtractive_nine():
assert roman_to_int("IX") == 9
def test_mixed_symbols():
assert roman_to_int("LVIII") == 58
def test_full_year():
assert roman_to_int("MCMXCIV") == 1994
PY
python3 -m pip install -q pytest
Save the block above as seed.sh next to your script. Then describe the agent inline and pass the first message. One POST /v2/threads creates the agent, boots the sandbox, runs the seed, and starts the first turn. The response is a handle to a running thread.
SEED = open("seed.sh").read() # the bash from above
resp = requests.post(
f"{BASE}/v2/threads",
headers=HEADERS,
json={
"agent": {
"harness": "claude_code",
"model": "claude-sonnet-5",
"instructions": (
"You are an autonomous engineer working in a throwaway Linux "
"sandbox. A small Python project is already here and its tests "
"are red. Turn them green by changing implementation code only; "
"leave every test file exactly as it is. Check yourself with "
"pytest and keep iterating until nothing fails."
),
"entrypoint": SEED,
},
"input": (
"The Python project under /workspace has failing tests. Run pytest "
"to see what breaks, then fix the defect in roman.py until the whole "
"suite is green. The tests are off-limits."
),
},
)
resp.raise_for_status()
thread_id = resp.json()["thread_id"]
print("thread:", thread_id)
claude_code here. Change harness to codex, cursor, opencode, qwen, or any other supported value and nothing else changes. See Any harness for the full list.2. Watch the agent work
The event stream is a live server-sent events feed of everything the agent does: its text, each tool call, and a final idle event with the turn's outcome. backfill=100 replays recent events on connect so you don't miss the start of the turn. Read until the idle event, then stop.
def stream_turn(thread_id, backfill=100):
"""Print the agent's activity until the turn goes idle; return the outcome."""
with requests.get(
f"{BASE}/v2/threads/{thread_id}/events/stream",
headers=HEADERS, params={"backfill": backfill}, stream=True, timeout=None,
) as r:
r.raise_for_status()
for line in r.iter_lines(decode_unicode=True):
if not line or not line.startswith("data:"):
continue
event = json.loads(line[len("data:"):].strip())
kind, data = event.get("type"), event.get("data") or {}
if kind == "tool_call.start":
print(" ->", data.get("name", "tool"))
elif kind == "assistant.message":
text = "".join(
p.get("content", "") for p in data.get("content", [])
if isinstance(p, dict) and p.get("type") == "text"
)
if text.strip():
print("\nAgent:", text.strip()[:400])
elif kind == "idle":
print(f"[turn {data.get('status')}] {(data.get('summary') or '')[:200]}")
return data
stream_turn(thread_id)
Prefer to poll? GET /v2/threads/{thread_id} returns the thread's status, which settles to success, fail, need_more_info, or idle when the turn ends.
3. Add a feature on the same sandbox
A thread is a conversation, not a one-shot job. Append a message with run: true and the agent picks the thread back up on the same sandbox, with full context. Here it builds the inverse function and tests it.
def get_messages(thread_id):
r = requests.get(f"{BASE}/v2/threads/{thread_id}/messages",
headers=HEADERS, params={"limit": 200})
r.raise_for_status()
return r.json()["items"]
def run_and_wait(thread_id, content, poll_s=5, timeout_s=600):
"""Append a message, start its turn, and wait for THAT turn to finish."""
before = len(get_messages(thread_id))
deadline = time.time() + timeout_s
# The turn may still be settling when it goes idle, so the POST can
# briefly return 409. Retry it until the message is accepted.
while True:
resp = requests.post(
f"{BASE}/v2/threads/{thread_id}/messages", headers=HEADERS,
json={"messages": [{"role": "user", "content": content}], "run": True},
)
if resp.status_code != 409:
resp.raise_for_status()
break
if time.time() > deadline:
raise TimeoutError("follow-up POST kept returning 409")
time.sleep(1.5)
while time.time() < deadline:
thread = requests.get(f"{BASE}/v2/threads/{thread_id}", headers=HEADERS).json()
replied = len(get_messages(thread_id)) > before + 1
if replied and thread["status"] in TERMINAL:
return thread
time.sleep(poll_s)
raise TimeoutError("follow-up turn did not settle")
run_and_wait(thread_id, (
"Now add the inverse to roman.py: int_to_roman(n) turns an integer between "
"1 and 3999 into its Roman-numeral form. Cover it with your own tests and "
"run the suite again."
))
run: true; without it the message is appended and nothing runs. Two timing wrinkles follow. The POST itself can briefly return 409 while the previous turn is still settling, so run_and_wait retries it. And once the message is accepted, the thread can still report the previous turn's terminal status for a moment, so a naive "wait until not running" returns at once and skips the new turn. run_and_wait handles both: it retries the POST, then waits until the transcript grows (the agent replied) and the status is terminal.4. Clean up
The sandbox stays warm for more turns and stops on its own when idle. To stop it right away, delete its machine. GET /v2/threads/{thread_id} carries the machine_id you need.
thread = requests.get(f"{BASE}/v2/threads/{thread_id}", headers=HEADERS).json()
machine_id = thread.get("machine_id")
if machine_id:
requests.delete(f"{BASE}/v2/machines/{machine_id}", headers=HEADERS).raise_for_status()
else:
# Provisioning never reached a sandbox, so there is nothing to stop;
# skipping keeps a failed run from raising a second error here.
print("no machine to delete")
DELETE /v2/machines/{id} expects the Brainbase machine_id. The thread also carries a provider-side sandbox_id; depending on the sandbox provider the two can look identical (both are often UUIDs), so tell them apart by field name, not by shape. Always tear down with machine_id.