The first alarm is not the fault

Building a test that a cheap rule cannot pass, and what two AI models did with it.

Your car’s dashboard lights up at once: engine, temperature, oil pressure, battery. Four warnings, all of them real. But they are not four problems. They are one problem and three consequences. A mechanic who fixes the light that came on first replaces the wrong part and sends you back out with the actual fault still there.

Noticing that something is wrong is easy. Working out which wrong thing caused the others decides whether the repair works. A drone in flight has the same problem, with a few hundred lights, no dashboard, and seconds to act.

I build a monitor that watches a drone while it flies and tells the operator what is failing. Sensors disagree, motors run out of headroom and stop being able to correct, the navigation filter that combines the sensors loses confidence, vibration climbs. Most of it is downstream of one root cause.

The easy half already works. Run sentinel analyze on a flight log and it reports settings that contradict the hardware. On one 2024 log it flagged a low-battery warning set to 10.5 volts on a 22.2 volt pack — a threshold left over from a smaller battery. The alarm could never have fired, so that aircraft flew with no working low-battery protection.

That log is private. Treat the battery case as an illustration, not a claim you can check. Every other figure on this page regenerates from committed files.

Ranking is the hard half, and it is what the rest of this measures.

To test ranking I needed a fault where the obvious answer is the wrong answer. So I broke a drone’s compass on purpose in simulation — a magnetometer, the sensor that reads the Earth’s magnetic field, biased by a fixed amount so it pointed slightly wrong.

The gap is not luck and not tuning. The compass detector must see the anomaly persist for a full second before calling it, a deliberate guard against false alarms written in the source as MIN_ANOMALY_S = 1.0, in compass.py:45. The navigation filter has no such guard. So the careful design choice is exactly what puts the true cause last in the queue — every flight, for as long as that line exists.

There is a free rule that needs no AI:

The first alarm after something changed is the cause.

It costs nothing and on most faults it is simply right. When a motor fails, the motor alarm fires first. When GPS drops out, the GPS alarm fires first. Any expensive system has to beat this rule to justify itself, and usually it cannot. On compass_offset it cannot be tuned out of being wrong — its rule and the persistence gate point in opposite directions by construction.

Each flight is recorded once into a single fingerprinted file, so it cannot change unnoticed, and every judge runs against that file rather than against the aircraft. Scoring is strict: naming a symptom scores zero rather than partial credit, and every answer must cite evidence that exists in the flight, so a correct answer pointing at something that never happened also scores zero.

B2 (repeated sampling, cost-matched to B3) is deliberately absent from the table below. Its 0.89 comes from a different set — the 9- and 22-bundle sweeps — not from this 5-run protocol. Putting it in a neighbouring column would compare two different experiments.

Gemini’s agent had a legible failure. When its tools were allowed to report when each alarm fired, it treated that ordering as causation:

“The EKF inconsistency was detected first… this likely caused the compass inconsistency.”

Exactly backwards — EKF is the navigation filter, the symptom. Removing the timing from its tools, changing nothing else, cut the mistake from eight judgements out of nine to one.

GPT never named a symptom as the root cause in any configuration tested, and is marginally better with timestamps. So the tempting headline — “AI agents get worse when you give them more tools” — is not supported. One model confused ordering with causation. A stronger model read the same ordering and drew the correct conclusion.

A caveat I would rather state than have you find. This rests on three flights of one fault. An inverted ranking on that little data could be partly noise. What I can defend is the mechanism, because the misreading is visible in the model's own words and it disappeared when the ordering data was removed. What I cannot yet defend is the size of the gap.

Accuracy tables invite a question they cannot answer: which model should I actually use? The number that answers it is not tokens per call, it is tokens per correct answer — what you spend to buy one right result. Nine models, one frozen arm, identical prompts and tools, so any difference in spend belongs to the model rather than the task.

The cheapest model to call is not the cheapest model to be right with. gemini-2.5-flash is the cheapest per judgement at 4,652 tokens and finishes sixth once accuracy is priced in. gemini-3.7-flash costs more per call and wins outright. A model chosen on the per-call column alone would have been the wrong choice.

claude-sonnet-5 is the instructive row: 0.91 accuracy and seventh on cost. It is accurate and verbose, and per-token pricing charges for verbosity whether or not the answer is right. No accuracy table shows that.

Every row above is compass_offset only. Seven of the nine have never been measured on a second fault, and on the second fault that does exist, both of the two that were re-measured fall to 0.00. This is a ranking on one mechanism, not a league table.

This is the question everyone asks about AI in a safety context, so I measured it rather than reasoning about it. Every number a judge writes in its explanation is checked against what the flight actually recorded. Invent a reading and it is caught; quote a real one, even rounded, and it passes.

The case that made me build the check was Gemini writing “exceeded its threshold at 9.062 seconds, causing the compass inconsistency at 10.016 seconds.” Both of those times are real. It read the data correctly and drew the wrong conclusion from it.

That is the more uncomfortable finding. These models are not inventing evidence. They are misreading accurate evidence with confidence, and no amount of fact-checking the numbers can catch that. The only thing that catches it is a test where the correct answer and the plausible answer are different.

Each was caught by a check rather than an argument: an outside model reviewing the work against a written rubric, running things five times instead of once, testing a second model, and building comparisons that could have come out either way.

A monitor that quietly reports the wrong root cause is worse than one that crashes, because you act on it. The same applies to my own conclusions, which is why those checks run whether or not I expect them to find anything.

The detection half needs no simulator, no AI account and no hardware — it reads the .BIN files already on your SD card. The detectors ship in a sibling repo that is not on PyPI, so it is two clones rather than one:

git clone https://github.com/RahulRajelli/sentinel-realtime
git clone https://github.com/RahulRajelli/ardupilot-log-analyzer

pip install -e sentinel-realtime
pip install -e ardupilot-log-analyzer

cd sentinel-realtime
sentinel doctor                    # names anything missing, and the exact fix
sentinel analyze YOURFLIGHT.BIN    # findings, each printed with its evidence
sentinel analyze YOURFLIGHT.BIN --html report.html

ArduPilot only, and that is deliberate. Live telemetry uses the ardupilotmega dialect; log analysis reads ArduPilot dataflash. PX4 uses a different dialect, log format and parameter names — it would run and find nothing, which is worse than refusing. Betaflight and INAV are not supported either.

For a live link rather than a log: sentinel watch --conn COM5,57600, or udp:0.0.0.0:14550 for WiFi or a simulator. The HTML report is self-contained — no CDN, no fonts, no scripts — so it opens on a hangar machine with no network.

Survived: the fault works. The free rule scores 0.00 on it, on every model and every configuration tested. That is what this phase set out to build.

Not established: whether anything here about AI judgement holds beyond one fault and three flights. Two frontier models already disagree about it.

More models would not fix that. A second fault of the same kind would — and building one most likely requires a new detector rather than a new test, because only two detectors in the current set carry the built-in delay that makes a cause arrive after its own symptom. That work is not done.

python -m pytest -q                       # 215 passed, no simulator, no API key
python scripts/manifest.py --verify       # 28/28 bundles intact

# the free rule at 0.00, and the tool agent's score
python scripts/e4_report.py --bundles bundles \
  --verdicts results/crossmodel/gpt2_run1.json --only compass_offset

Scoring was broken for part of 2026-08-15 and is fixed. The schema-2 migration recomputed every bundle_id, which the 60 verdict files store as a foreign key, so every one of them stopped resolving. A verdict may now cite any id its flight legitimately answered to; the substitution is recorded on the row rather than applied silently. No verdict file was rewritten and bundle_id was not weakened.