The discriminating pair

The one injected fault in this archive on which a deterministic root-cause baseline is wrong by construction — and everything that is therefore still unproven.

Capture and judgement are separated: fly once, freeze a RunBundle, judge offline against the frozen file. bundle_id excludes wall-clock timings so it fingerprints the flight rather than the host; replay sets no ground-truth label; transport failures degrade to the deterministic baseline and are attributed to HARNESS, never to the model.

Scoring is strict and binary per bundle. Naming a symptom scores zero rather than partial credit. Every answer must cite evidence present in the flight, so a correct root cause anchored to an event that never happened also scores zero. The correct answer is withheld by construction and a test asserts it never appears in any judge-visible surface.

Statistics are per bundle (ci_bundle), never per judgement. 9 bundles × 3 prompt variants is 27 judgements but only 9 independent flights; treating them as 27 inflates n threefold. That exact error was made here and the significance claim built on it was retracted.

“The EKF inconsistency was detected first… this likely caused the compass inconsistency.”

gemini reads advisory ordering as causal ordering — exactly the trap the fault is built from. gpt-5.6-sol never names a symptom as root in any configuration tested, and is marginally better with timestamps present. So “give a diagnostic model evidence, not the alarm log” does not generalise.

Open decision. SPECS is currently the timestamp-free surface, selected on gemini-only evidence. Measured since: it rescues gemini (0.11 → 0.67) and costs gpt-5.6-sol (1.00 → 0.96). Revert it, keep it as the safer floor, or select per model — but do not flip it again without measuring both models.

check_rationale_grounding validates every measurement quoted in a verdict's rationale prose against everything the flight recorded. Across 595 verdicts spanning four judges and two model families: 2 ungrounded quotes, 0.34%.

The case that motivated the check — gemini writing “at 9.062s” — turns out to cite a real advisory time. The models are not inventing numbers; they draw wrong conclusions from real ones. No grounding check catches that. Only a scenario where the correct and the plausible answer diverge does.

Grounding reports rather than gates by default (strict_rationale=True switches it). Gating would silently change what every earlier number means.

Six advisory types is the vocabulary ceiling. A fault outside it produces silence, and that miss rate is unmeasurable with injected faults — injection can only exercise vocabulary the detectors already have.

null, vibration, gps_loss and wind are the base four and pass 12/12; null and wind are the hallucination controls and score 1.00 for all judges, separating nothing. stiff_airframe is retired and hot_gains is blocked. compass_offset is the only discriminating fault in the archive.

Rule, not preference: never lower a detector threshold to make a scenario pass. Three faults are recorded as unreachable rather than tuned into existence.

health.py is reported and never corrective — a monitor that quietly repairs itself is one whose degradation you learn about after the flight.

Unmeasured capabilities, not results: prior_incidents and signal_trajectory. No scenario repeats a fault on one airframe, and trajectory has never been offered to a live model. signal_trajectory is bounded by construction (1,056 samples render to 863 characters; 999 buckets clamp to 60), and detector_evidence is capped — unbounded it returned 191,465 characters in a single call.

SCHEMA_VERSION = 2 ships with a migrator, so an old capture upgrades rather than being orphaned. Three identities are legitimate here — (v1, timing included), (v1, timing excluded), (v2, timing excluded). Migration proves authenticity against all three and nothing else, so it cannot launder a tampered file. manifest.py adds full-length SHA-256 over file bytes, catching edits that bundle_id is designed to ignore.

It is not a signature. The optional HMAC is a shared secret, so anyone who can verify can also forge. This archive verifies as unsigned — tamper-evident against accidental drift, not against a motivated editor. Claiming otherwise would be the security theatre this project exists to avoid.

What that rule still failed to prevent. On 2026-08-15 the v1→v2 migration recomputed every bundle_id — and the 60 verdict files store it as a foreign key, so all of them stopped resolving and no figure regenerated. Bumping the version and adding a migrator covers the bundles and nothing that points at them. Fixed by letting a verdict cite any id its flight legitimately answered to, recorded on the row rather than applied silently; no verdict file was rewritten and bundle_id was not weakened. The rule is now three clauses, not two.

More models will not fix the base problem. A second discriminating fault would, and it most likely requires a new detector carrying its own persistence gate rather than a new scenario file — only compass.py and oscillation.py have time gates today, and one of those is unreachable. Three candidate scenarios are already ruled out with measurements.

git clone https://github.com/RahulRajelli/sentinel-realtime
git clone https://github.com/RahulRajelli/ardupilot-log-analyzer   # the detectors, not on PyPI
pip install -e sentinel-realtime && pip install -e ardupilot-log-analyzer

# offline, no API key
python -m pytest -q                       # 215 passed
python scripts/manifest.py --verify       # 28/28, signature: unsigned
sentinel analyze YOURFLIGHT.BIN           # your own ArduPilot dataflash log

# re-score the published figures from the committed verdicts
python scripts/e4_report.py --bundles bundles \
  --verdicts results/crossmodel/gpt2_run1.json --only compass_offset

# judge a new scenario - two models, five runs each, never one of either
export OPENAI_BASE_URL="https://api.llmapi.ai/v1"; export OPENAI_API_KEY=...
python scripts/e4_judge.py --bundles bundles --only YOUR_SCENARIO \
  --judges B0,B1,B3 --provider openai --model gpt-5.6-sol --out verdicts.json

B0 must score 0.00 on any scenario you claim is ambiguous, on every model and every run. It is deterministic code with no model in it. If B0 is not 0.00, the scenario is not ambiguous and nothing else in the run means anything.