Nebius Global AI Hackathon · Coding and Agentic Engineering

Faultline

The bisect that refuses
to blame the innocent.

Edy Cu · solo build

The fault · commit 173 of 255
4e133d6c
normalize: attach the zone directly instead of localize()
Seismogram of hunt 827dfe7e Three rows, one per round, across commits 0 to 255. Round 1 probed 8 commits from 0 to 255; round 2 probed 8 commits from 150 to 178; round 3 probed 171, 172 and 173. Commit 173, 4e133d6c, failed and is the fault. The dependency pin moves at commit 150. commit 0 32 64 96 128 150 173 192 224 255 150 · pytz pin moves ROUND 18 jobs · 318.5 s ROUND 28 jobs · 374.3 s ROUND 33 jobs · 268.6 s 173 · 4e133d6c · THE FAULT
T+15:57 · settled
10/10

times git bisect run in one reused environment named the wrong commit.

e46e5342 the pin bump · env built at v0.1.0 · 5 of 5
acebfbfa a changelog edit · env built at main · 5 of 5
Faultline · v0.4.0Trace 02 of 10

Why it happens

The pin moved at 150. The bug is at 173.

One set of pins for every commit: under drift, it blames a commit that never touched the bug.

installed with pytz 2022.12023.303264961281601922241 · acebfbfablamed · env built at main · 5/5150 · e46e5342blamed · env built at v0.1.0 · 5/5173 · 4e133d6cthe fault · named by executed tests
Faultline · v0.4.0Trace 03 of 10

Every probe gets its own clean room.

Both ends first

8 at a time

Tests decide

eu-west1
0 · pass
36 · pass
me-west1
219 · fail
255 · fail
uk-south1
73 · pass
109 · pass
uk-south2
146 · pass
182 · fail
Faultline · v0.4.0Trace 04 of 10

Hunt 827dfe7e · recorded 2026-10-13

255 → 1 in three rounds.

Seismogram of hunt 827dfe7e Three rows, one per round, across commits 0 to 255. Round 1 probed 8 commits from 0 to 255; round 2 probed 8 commits from 150 to 178; round 3 probed 171, 172 and 173. Commit 173, 4e133d6c, failed and is the fault. The dependency pin moves at commit 150. commit 0 32 64 96 128 150 173 192 224 255 150 · pytz pin moves ROUND 18 jobs · 318.5 s ROUND 28 jobs · 374.3 s ROUND 33 jobs · 268.6 s 173 · 4e133d6c · THE FAULT
36 leftround 1 · 8 jobs · 318.5 s
4 leftround 2 · 8 jobs · 374.3 s · 150 cleared
4e133d6cround 3 · 3 jobs · 268.6 s
T+15:57 · settled
Faultline · v0.4.0Trace 05 of 10

Right answer, honest clock.

Bisect
Named
Trials
Time
reused env · v0.1.0
e46e5342
1.43 s
reused env · main
acebfbfa
1.49 s
fresh venv per step
4e133d6c
57.4 s
Faultline1 run · 19 jobs · $0.0948
4e133d6c
957 s
right commitwrong commit
Faultline · v0.4.0Trace 06 of 10

A real regression

attrs 22.1.0 → 22.2.0: everyone agrees.

135 commits; a reused env is right here too. Every method named 89a890a3 — the commit fix #1085 cites.

919 s

Faultline

43.8 s

fresh-venv bisect · p50

Faultline · v0.4.0Trace 07 of 10

Slower, and we say so.

4m 21s

VM start-up · p50

5.3 s

fetch, install, test · p50

one probe, to scale

It saves the drift hunt and the harness — not the clock.

Faultline · v0.4.0Trace 08 of 10

Built on Nebius + NVIDIA.

Serverless AI Jobs

Every probe

  • 19 jobs · 4 regions
Object Storage

Every verdict

  • presigned PUT per probe
  • 20 / 20 write races
Token Factory · NVIDIA

Every why

  • nemotron-3-super-120b-a12b
  • once per hunt, labelled

513 + 7 + 50 tests

Faultline · v0.4.0Trace 09 of 10

Run a live hunt.

Paste a public repo, a failing test and a last-good ref. Close the tab; it finishes anyway.

faultline.edycu.dev…/about — the full storygithub.com/edycutjong/faultline

The culprit is proven. The paragraph is labelled.

scan → the live app

Faultline · v0.4.0Trace 10 of 10

Turn your phone sideways for the deck — or read the story at /about.

← → slides · Esc overview · P notes · R replay · ⌘P PDF