Nebius Global AI Hackathon · Coding and Agentic Engineering
git bisect run in one reused environment convicted the wrong commit 10 times out of 10. Faultline gives every commit it tests its own fresh Nebius container that installs that commit's own dependencies — and named the real culprit in 3 rounds, 19 jobs, for $0.0948.
Each burst is one probe: a fresh container that fetched the repo at that commit, installed that commit's dependencies and ran one test. The quiet before each burst is real — 4 min 21 s of VM start-up (p50) for 5.3 s of fetching, installing and testing — replayed at ×100 from the recorded timestamps. Round times overlap by about 4 s where two ticks raced at the end of round 1.
Instrument readings
With the environment built at the known-good ref, five trials named the pin bump, e46e5342. Built at the failing ref, five named a changelog edit, acebfbfa. Neither touched the bug.
10 / 10
times git bisect run in one reused environment named the wrong commit
evidence/bench/hero-local.json
255 → 1
commits narrowed to one, in 3 rounds
data/recorded-hunt.json
19
Nebius Serverless AI Jobs across 4 regions, 8 at a time
data/recorded-hunt.json · jobId × 19
20 / 20
concurrent-write races on the real bucket, exactly one winner each
evidence/g5-conditional-writes.json
513 unit and integration tests at 100 % coverage of the engine · 7 probe contract tests · 50 end-to-end checks — npm test needs no credentials.
Trace 01 · the same 255 commits, bisected twice
On the left, the standard tool in one environment — excerpted from the benchmark's recorded output (trial 1), with Faultline's verdicts on the same two commits below it. On the right, the live app's report for the same history — round 2 tested the pin bump in its own container, and it passed.
# one venv, built once at v0.1.0, reused for every step $ git bisect start 43f3528c 0260b672 # main, v0.1.0 $ git bisect run python -m pytest -q -p no:cacheprovider tests/test_normalize.py::test_naive_local_midnight_is_interpreted_in_zone … FAILED tests/test_normalize.py::test_naive_local_midnight_is_interpreted_in_zone e46e534219b7d051e003508c88b5bfe621536349 is the first bad commit build(deps): bump pytz to 2023.3 — Mexico abolished DST, update golden values bisect found first bad commit # 8 steps · 1.45 s · the same wrong answer in 5 of 5 trials # faultline · hunt 827dfe7e · a fresh container per probe ✓ 150 e46e5342 pass the pin bump, in its own container — innocent ✗ 173 4e133d6c fail normalize: attach the zone directly instead of localize()
Benchmark output, evidence/bench/hero-local.json — the env built at the failing ref names acebfbfa instead, 5 of 5.
Faultlinelocalize() became replace(tzinfo=…).
Why it breaks — one Token Factory call to NVIDIA Nemotron 3 Super, labelled generated in the app. It gets the mechanism and 06:37 vs 06:00 right, but calls UTC−6 “DST-adjusted” — Mexico City has had no DST since 2022. The culprit came from the test, not this paragraph.
git bisect run, one env built at v0.1.0reused for every stepgit bisect run, one env built at mainreused for every stepgit bisect run, a fresh venv at every stepthe harness you would have to writeRead the time column honestly: Faultline is the slowest way to the right answer here. The fresh-venv harness is right too — Faultline is that harness, already written, run 8 at a time in clean Nebius containers, with both ends proven and refusals when the evidence contradicts itself.
Under the needle
Six decisions that keep a wrong commit from being named — each one covered by tests.
Round 1 runs your known-good ref and your failing ref in clean containers beside the first suspects. If the failing ref passes there, the break is your environment, not a commit — and Faultline says so instead of naming one.
round 1 · v0.1.0 + mainAny public repo's conftest.py is arbitrary code. A probe gets one presigned URL for its own verdict and a tick URL it can only ping with an empty POST — no key, no bucket mount.
presigned PUTEvery dispatch is claimed before it happens; the hunt record is only replaced if nobody changed it first. On the real bucket: 20 races of 2–8 writers, one winner each time.
claim PUT hunts/<id>/claims/<probe>.json If-None-Match: "*" 412 another tick already won hunt PUT hunts/<id>.json If-Match: <etag>
A probe that dies without reporting is retried, then the hunt pauses and says why. A throttled create (429) waits out its backoff. Neither becomes a verdict — a probe that never got to run the test is a skip, never a fail.
429 → backoffEach probe pings the hunt when its verdict lands, so the next round goes out by itself. Close the tab.
POST …/tickAfter the tests name the culprit, one Nebius Token Factory call asks NVIDIA Nemotron 3 Super why it breaks, with the diff and the failing output. The paragraph is labelled generated, and the hunt record keeps its Token Factory request ID.
nvidia/nemotron-3-super-120b-a12bThe live instrument
The app's home replays this recording with its real job IDs; “Run a live hunt” dispatches real Nebius jobs. These are screenshots of the deployed app, not mock-ups.
Honest by design · calibration notes
Drawn to scale. Almost all of a probe is waiting for its machine — the reason Faultline is slower than a local fresh-venv bisect, and why we want to try Sandboxes next (unmeasured: its beta access is pending).
“Read the wall-clock column honestly: Faultline is slower on both.”
— README · Engineering Rigor
“That fixture was built to drift, so the reused-environment failures are by construction; what it shows is the mechanism.”
— README · Engineering Rigor
“The root-cause paragraph is generated and can be wrong; the culprit is not.”
— README · Honest limits
Questions a judge would ask
git bisect?No. Fresh-venv git bisect run finished the drift fixture in 57.4 s (p50, N = 5) and attrs in 43.8 s (p50, N = 3); Faultline took 957 s and 919 s. A probe's VM took 4 min 21 s to start (p50) for about 5 s of fetching, installing and testing, and Nebius throttles job creation. What it saves is knowing about the drift, writing the fresh-environment harness, and running it on your own machine.
README · Engineering Rigor · DEMO.md §5
git bisect fail?Yes — faultbench-tzdata-pipeline moves its pytz pin at commit 150 on purpose, so the reused-environment failures are by construction. On a real regression — attrs 22.1.0 → 22.2.0, 135 commits, where a reused environment gets it right too — every method, Faultline included, named 89a890a3, the commit the fix (#1085) cites.
README · Engineering Rigor · evidence/bench/attrs-*.json
The replay on this page and on the app's home is hunt 827dfe7e, recorded 2026-10-13, labelled as a recording, with its real job IDs. “Run a live hunt” in the app dispatches real Nebius jobs — one hunt at a time, with a daily cap so traffic cannot drain the credit.
DEMO.md §6 · README · Honest limits
One Nebius Token Factory call per hunt (a second only if the first is abandoned mid-call), after the tests have named the culprit, with the diff and the failing output. It was partly wrong on both benchmark hunts; the culprit was not, because it came from running the test. The hunt record keeps the request ID.
DEMO.md §5 · README · NVIDIA Nemotron on Token Factory
Its conftest.py runs inside a probe, which holds one presigned URL for its own verdict object and a tick URL it can only ping with an empty POST — no key and no bucket mount. It can lie about its own commit's verdict; it cannot write another hunt's records — at worst it can ping a tick, or, like any visitor, start a hunt that counts against the visible daily cap.
README · Architecture · tests/hunt.integration.test.ts
Sandboxes would fit better, but its beta access is still pending: GET /sandboxes/v1/whoami reports every permission false. The engine talks to a four-verb ProbeBackend (dispatch, status, find, cancel); a Sandboxes backend would implement the same four. We have not been able to try one.
evidence/sandboxes-whoami.json · src/lib/backend/types.ts