Two checks that hold can clear a record. Add a third check with the same error bound, and OpenAuditor can return uncertain. Change that third answer to a model-qualified failure, and the gate returns false. We reproduced this asymmetry in the engine, then tested what its “99.5%” setting actually establishes.
The same bounds produce different verdicts
Our experiment calls the verdict gate in engine revision 6205721 directly. Each synthetic model answer carries an assumed error bound of 0.002. The gate receives an error budget of 0.005, which corresponds to the API’s default certainty_target: "0.995". No model was called, and these assumed bounds are not measured model results.
The recorded cases isolate that revision's verdict gate. They do not run model qualification or evaluate subsequent changes to threshold fitting and evaluation data. Those changes need their own measurements; the current evaluation reference describes the release's evidence. The historical cases and constructed populations below remain fixed.
| Synthetic assertion outcomes | Gate result | Actual rule applied |
|---|---|---|
| Two holds, each with bound 0.002 | true | The bounds total 0.004, within budget. |
| Three holds, each with bound 0.002 | uncertain | The bounds total 0.006; reason: combined_bound. |
| Two holds and one fail, each with bound 0.002 | false | One failure’s individual bound is within budget. |
| One hold plus insufficient evidence | uncertain | An applicable assertion remains unresolved. |
| One model-qualified fail plus insufficient evidence | false | The failure is sufficient for the gate’s exception verdict. |
The rule requires every applicable requirement to clear, but accepts one qualifying counterexample to flag an exception. The synthetic model failures above carry an assumed error bound; they are not infallible findings. In a purchasing review, a missing required PO can decide the verdict without settling whether the goods description is sufficiently specific. The false verdict’s undecided list is empty in this branch, while the individual assertion still records its unresolved outcome.
A target, a gate and a measurement
The requested target controls the gate’s budget. A question’s qualification bound comes from errors measured within that question’s accepted holds or fails region on held-out data. The gate combines those bounds according to its rules. Reported certainty is a separate lookup based on the observed error bound for a verdict group, such as false_with_model.
The response’s probability_true is different again. Our three-hold case supplies a raw p_yes of 0.9999 for every answer. Their product is reported as 0.9997, with an uncalibrated basis, yet the verdict remains uncertain because 0.006 exceeds the gate budget. A high displayed probability does not override the rule.
Without a published certainty table, a decided result reports basis: "unmeasured". A target is therefore neither a measurement nor a promise about accuracy on a customer’s records.
A union bound needs the right population
Adding error bounds resembles the standard union bound: the probability that any error occurs is at most the sum of the component error probabilities. That inequality does not require independent errors. But its terms must refer to the same probability space and conditioning. MIT’s probability notes state the union bound and the conditional-probability rules.
The distinction matters here. Error among all records where question A says “holds” need not equal error among records where both A and B say “holds.” Combining decisions changes the selected population. The gate’s arithmetic alone does not establish that the qualification bounds remain valid within that final population.
A constructed population exposes the selection problem
Consider this deliberately constructed distribution, with each record equally likely. It is a counterexample to a general inference, not a model benchmark or a measurement of OpenAuditor’s production errors.
| Record group | Count | Question A | Question B | Underlying truth |
|---|---|---|---|---|
| The shared hold | 1 | Holds | Holds | A should fail; B should hold. |
| Only A decides | 9,999 | Holds | Abstains | Both requirements hold. |
| Only B decides | 9,999 | Abstains | Holds | Both requirements hold. |
A’s hold region has one error in 10,000 records: 0.01%. B’s has none in 10,000. Both exact regional rates are below the assumed 0.2% bounds. Nevertheless, the intersection contains only the erroneous record. The gate accepts that one record because 0.002 + 0.002 fits its budget; the conditional error among its true verdicts is 100%. The other 19,998 records remain uncertain.
The failure branch has a related issue. Our second construction contains 998 genuine exceptions where all three questions fail, plus six valid records: two incorrectly flagged by A, two by B and two by C. Every question’s fail region has two errors in 1,000 records, exactly 0.2%. The combined false-verdict population contains all 1,004 records and six errors: 0.598%, above the 0.5% gate budget.
Neither example says these distributions occur in production. They show why individual regional bounds alone cannot establish an end-to-end error guarantee after selection. They also explain why measuring the final verdict groups is necessary.
The published certainty can differ from the target
The pinned repository’s AI-output table contains a false_deterministic group with zero errors in 450 observations. Its stored upper error bound is 0.00664; the certainty lookup returns 0.9934. The gate can still return false under the default 0.995 target, because this lookup does not veto the verdict.
Those are existing published evaluation counts, not 450 new requests from our experiment. The table uses the engine’s one-sided 95% binomial confidence-bound calculation. Zero observed errors does not mean zero underlying error, and this stored result does not measure production reliability.
The lookup is keyed by pack version, model artifact and dtype, with a no-model fallback. It does not independently remeasure a caller’s changed population, skipped checks or policy parameters. Read the evaluation scope alongside the response fields.
Scope changes are not free accuracy gains
The gate removes not_applicable and caller-skipped assertions from its calculation. Removing the third hold makes our 0.006 total become 0.004, but it narrows what the verdict covers. If nothing remains applicable, the result is uncertain.
Missing evidence, abstention, unasked questions, unavailable models, context-length failures and unqualified answers remain applicable and block a true verdict. A qualifying failure can still decide the item false. The risk-indicator guard can also prevent true; it does not override a qualifying failure. The harness covers these branches and flags one internal edge case: a manually constructed hold without a bound contributes zero in the pinned gate. Normal qualification generation supplies that bound; the experiment does not demonstrate a public request bypass.
Inspect the experiment
Download the recorded JSON results and standalone results checker. We ran 29 local gate cases against private engine revision 6205721, without model calls. The download records those outcomes; it does not include the engine.
The checker validates the recorded outcomes against the experiment’s case manifest, independently recomputes both constructed populations’ error rates from their record patterns, and checks the zero-error confidence-bound calculation. It verifies the supplied records and arithmetic, not a rerun of the private engine. Save both downloads in the same directory and run this command with Python 3.10 or later; no additional packages or service access are needed.
python3 check_results.py gate_asymmetry.jsonThe practical question is the denominator: errors among which accepted true decisions, which flagged exceptions, and how much unresolved work? Selective-classification research studies this risk–coverage trade-off; this investigation applies that established distinction to this engine. Use the batch review workflow to retain reference decisions and measure each final group on the population you intend to use.