Evaluations
See how often each use case reaches a verdict, which faults it catches, and how much work remains for a reviewer.
What is published
The published runs evaluate synthetic organizations, policy books, contract portfolios, document sets and agent sessions. Structured facts and injected faults come from the corpus generators, written by the same team as the engine. The free text that judgment questions read (expense descriptions, exception rationales, invoice lines, claim narratives, contract scopes, extracted fields, statements and agent instructions) comes from labelled phrase banks. Each bank row was written by a model writer and labelled again by a separate model labeller that saw neither the writer’s label nor any field revealing it; rows where the two disagreed, or where the labeller was unsure, were dropped. No person has labelled these rows yet. These results measure agreement with that constructed truth. They do not establish accuracy on a customer’s records or professional audit reliability.
Summary recorded 2026-10-07T06:10:30+00:00. Synthetic corpora, holdout seeds only. Structured facts and faults come from generators written by the same team; the text judgment questions read comes from phrase banks written by model writers and labelled a second time, blind, by model labellers. These numbers measure agreement with that constructed truth, not accuracy on any real population.
Read the published JSON for the machine-readable figures. Each use-case table retains its seeds, dataset hash, source revision and model identity so results can be compared on the same basis.
How to read the numbers
| Metric | Meaning | How to use it |
|---|---|---|
| Decided coverage | The share of items receiving a true or false verdict; called certified_coverage in the JSON. | Plan how much of the supplied population gets a decided result. This is not an accuracy percentage or an external certification. |
| Uncertain per 1,000 | The number of unresolved items per 1,000 evaluated items. | Estimate review volume for this corpus mix. Your own data mix can produce a different workload. |
| Coverage gain (pp) | Model-assisted coverage minus rules-only coverage, in percentage points at the same certainty target; called alpha_vs_deterministic_only in the JSON. | Look for added coverage only where a model-assisted run is published with comparable conditions. |
| Errors / n | Incorrect verdicts divided by decided verdicts in one outcome group. | Inspect true and false verdicts separately; also distinguish rule decisions from model-dependent decisions. |
| Upper error bound | The one-sided 95% Clopper–Pearson upper bound computed from that group’s errors and sample size. | Zero observed errors still has a nonzero upper bound. The bound describes the measured sample and its assumptions. |
| Faults: false verdicts | Share of injected faults receiving a false verdict. | Read this by failure family to find gaps hidden by an overall total. |
| Faults: false or uncertain | Share of injected faults receiving false or uncertain. | This includes abstentions; it does not mean every fault was conclusively identified. |
| Look-alikes | Valid records that resemble faults, with their false and uncertain result rates. | Check whether superficially suspicious records create false alarms or extra review. |
A dash means no measured stratum or comparison is available. Do not read it as zero error. The requested certainty target controls the decision gate; it is separate from the measured certainty reported for a verdict group.
How the evaluation is constructed
- Build a complete case. Each generator creates related records and context, including ordinary records designed to satisfy the checks.
- Inject named faults and look-alikes. Fault labels identify the assertions deliberately broken. Look-alikes resemble those faults while remaining valid by construction.
- Keep the splits separate. Development, validation, calibration and holdout use different seed sets. The published tables use holdout seeds; new IDs or seeds do not make the underlying scenario design independent.
- Score verdicts and reasons. Compare the engine’s output with the generator labels, retain unresolved outcomes and report the decided groups separately instead of hiding them inside an overall accuracy figure.
The source revision recorded for each run determines which data and procedures produced it. Holdout text comes from writers who saw no training, validation or calibration text, and a text found in two splits is kept only in the split that measures. The model trained on the training split, its checkpoint was chosen on validation, and each question’s thresholds were chosen on calibration before the holdout certified them. An earlier model was measured on the first third of this holdout; that result led to more varied training text for the published model, and the other two thirds of the holdout were written afterwards. A rules-only run supplies no model qualification or model-assisted comparison evidence.
Phrase banks
Rows per question and split, after the second labelling and after removing texts shared across splits.
| Question | Training | Validation | Calibration | Holdout |
|---|---|---|---|---|
business_expense | 1,861 | 596 | 1,587 | 3,966 |
exception_justified | 2,664 | 595 | 1,586 | 3,378 |
expense_category | 2,683 | 597 | 1,598 | 3,994 |
expense_reimbursable | 2,668 | 599 | 1,596 | 3,376 |
invoice_specific | 1,853 | 594 | 1,575 | 3,396 |
items_consistent | 2,686 | 600 | 1,600 | 3,989 |
line_in_scope | 2,689 | 600 | 1,599 | 3,398 |
narrative_matches_peril | 2,673 | 596 | 1,591 | 3,980 |
right_field | 1,601 | 529 | 1,361 | 2,471 |
serves_instruction | 2,467 | 529 | 1,445 | 3,249 |
statement_supported | 1,864 | 597 | 1,597 | 3,391 |
Judgment questions
A question decides an assertion only inside a region it qualified for: answers above its passing threshold, or below its failing threshold. Calibration proposes five thresholds per side, covering 90%, 75%, 60%, 45% and 30% of the calibration rows with that side’s label. The holdout tests every one with a one-sided 99% Clopper–Pearson upper bound, so the five bounds hold together at 95%, and the widest region whose bound is at most the 0.5% error budget qualifies. Every other answer is recorded with its probability and decides nothing. A side that did not qualify shows the candidate that came closest.
This rule replaced a single calibration-chosen threshold after the first result for business_expense: that threshold decided 6,519 holdout answers with 28 errors, a bound just over budget. The five candidates are fixed by calibration alone and include stricter thresholds.
| Question | Holdout texts | Passing side | Failing side | Status |
|---|---|---|---|---|
serves_instruction@1 | 3,197 | 5 errors in 414; bound 3.14%, over budget | 0 errors in 417; bound 1.10%, over budget | not qualified |
user_originated@1 | 0 | – | – | not qualified |
right_field@1 | 3,206 | 3 errors in 1,712; bound 0.59%, over budget | withdrawn: 5 errors in 64 in full cases | not qualified |
statement_supported@1 | 4,834 | 0 errors in 1,300; bound 0.35% | 1 error in 1,612; bound 0.41% | qualified: fails and holds |
expense_reimbursable@1 | 3,331 | 14 errors in 608; bound 4.15%, over budget | 2 errors in 1,063; bound 0.79%, over budget | not qualified |
line_in_scope@1 | 3,376 | 2 errors in 1,004; bound 0.83%, over budget | 0 errors in 479; bound 0.96%, over budget | not qualified |
business_expense@1 | 8,650 | 5 errors in 5,643; bound 0.23% | 1 error in 1,067; bound 0.62%, over budget | qualified: holds |
exception_justified@1 | 3,289 | 0 errors in 950; bound 0.48% | 1 error in 1,056; bound 0.63%, over budget | qualified: holds |
expense_category@2 | 6,936 | 3 errors in 4,124; bound 0.24% | 3 errors in 1,013; bound 0.99%, over budget | qualified: holds |
invoice_specific@1 | 2,859 | 0 errors in 1,389; bound 0.33% | 0 errors in 982; bound 0.47% | qualified: fails and holds |
exception_justified@2 | 3,289 | 0 errors in 941; bound 0.49% | 1 error in 1,051; bound 0.63%, over budget | qualified: holds |
items_consistent@1 | 7,895 | 2 errors in 1,255; bound 0.67%, over budget | 2 errors in 726; bound 1.15%, over budget | not qualified |
narrative_matches_peril@1 | 7,895 | 12 errors in 3,413; bound 0.67%, over budget | 2 errors in 999; bound 0.84%, over budget | not qualified |
Withdrawn after qualification
right_field@1, fails side: Certified on one context per bank row and one per table template, this side made 5 errors in 64 decisions in the full holdout cases: 2 on generated table rows and 3 on bank rows whose measured context differed from the one asked. Withdrawn until qualification measures several contexts per text.
Results by use case
Agent actions agent-actions@2
Read the supported checks alongside these results: the mix of item types affects both coverage and review load.
| Configuration | Items | Decided coverage | Coverage gain (pp) | Uncertain per 1,000 | True, rules: errors / n | True, with model | False, rules | False, with model |
|---|---|---|---|---|---|---|---|---|
| Rules only | 4496 | 10.7% | 0.0 pp | 892.6 | – | – | 0 / 483, ≤ 0.62% | – |
With smalldecide 00607d88f6e1:float32-cpu | 4496 | 10.7% | 0.0 pp | 892.6 | – | – | 0 / 483, ≤ 0.62% | – |
Faults
| Fault | Items | False verdicts | False or uncertain |
|---|---|---|---|
| agent_approval | 42 | 100.0% | 100.0% |
| burst | 51 | 100.0% | 100.0% |
| expired_grant | 42 | 100.0% | 100.0% |
| forbidden_resource | 41 | 100.0% | 100.0% |
| foreign_grant | 40 | 100.0% | 100.0% |
| late_approval | 39 | 100.0% | 100.0% |
| missing_approval | 38 | 100.0% | 100.0% |
| off_task | 75 | 0.0% | 100.0% |
| out_of_scope | 41 | 100.0% | 100.0% |
| over_limit | 40 | 100.0% | 100.0% |
| premature_grant | 36 | 100.0% | 100.0% |
| prompt_injection | 82 | 0.0% | 100.0% |
| revoked_grant | 40 | 100.0% | 100.0% |
| self_approval | 33 | 100.0% | 100.0% |
Look-alikes
| Look-alike | Items | False verdicts | Uncertain |
|---|---|---|---|
| approved_consequential | 64 | 0.0% | 100.0% |
| fast_session_within_rate | 30 | 0.0% | 100.0% |
| large_within_limit | 49 | 0.0% | 100.0% |
| numbers_from_tool_output | 71 | 0.0% | 100.0% |
Holdout seeds 101, 102, 103, 104, 105, 106, 107, 108, 109, 110; dataset SHA-256 2d9deda7ef5ddb21….
Run identity and dataset
| Configuration | Recorded at | Source commit | Model artifact SHA-256 | Dtype |
|---|---|---|---|---|
| deterministic_only | 2026-10-05T07:37:13+00:00 | eb3fa43b231591b57bb51c19dc12c6621166a0de | none | none |
| smalldecide:00607d88f6e1:float32-cpu | 2026-10-05T10:56:00+00:00 | ef160343de4711e2616e17e7edc5fb595a8c626a | 00607d88f6e152b471e557f6a814471ab5a6faf900cae427d8b3d50d4614be17 | float32-cpu |
Dataset SHA-256: 2d9deda7ef5ddb211957b760572b2cf29afca48b5385c51e131fc5c21a9198d4.
Agent actions agent-actions@1 (superseded)
Superseded, and still served to requests that name agent-actions@1. These results are not results for the current version. The use-case page lists what changed between versions.
Read the supported checks alongside these results: the mix of item types affects both coverage and review load.
| Configuration | Items | Decided coverage | Coverage gain (pp) | Uncertain per 1,000 | True, rules: errors / n | True, with model | False, rules | False, with model |
|---|---|---|---|---|---|---|---|---|
| Rules only | 4496 | 10.7% | 0.0 pp | 892.6 | – | – | 0 / 483, ≤ 0.62% | – |
With smalldecide 00607d88f6e1:float32-cpu | 4496 | 10.7% | 0.0 pp | 892.6 | – | – | 0 / 483, ≤ 0.62% | – |
Faults
| Fault | Items | False verdicts | False or uncertain |
|---|---|---|---|
| agent_approval | 42 | 100.0% | 100.0% |
| burst | 51 | 100.0% | 100.0% |
| expired_grant | 42 | 100.0% | 100.0% |
| forbidden_resource | 41 | 100.0% | 100.0% |
| foreign_grant | 40 | 100.0% | 100.0% |
| late_approval | 39 | 100.0% | 100.0% |
| missing_approval | 38 | 100.0% | 100.0% |
| off_task | 75 | 0.0% | 100.0% |
| out_of_scope | 41 | 100.0% | 100.0% |
| over_limit | 40 | 100.0% | 100.0% |
| premature_grant | 36 | 100.0% | 100.0% |
| prompt_injection | 82 | 0.0% | 100.0% |
| revoked_grant | 40 | 100.0% | 100.0% |
| self_approval | 33 | 100.0% | 100.0% |
Look-alikes
| Look-alike | Items | False verdicts | Uncertain |
|---|---|---|---|
| approved_consequential | 64 | 0.0% | 100.0% |
| fast_session_within_rate | 30 | 0.0% | 100.0% |
| large_within_limit | 49 | 0.0% | 100.0% |
| numbers_from_tool_output | 71 | 0.0% | 100.0% |
Holdout seeds 101, 102, 103, 104, 105, 106, 107, 108, 109, 110; dataset SHA-256 2d9deda7ef5ddb21….
Run identity and dataset
| Configuration | Recorded at | Source commit | Model artifact SHA-256 | Dtype |
|---|---|---|---|---|
| deterministic_only | 2026-10-02T16:08:38+00:00 | 9bc3424a301c36ce532c2422dee9968dacc8fa71 | none | none |
| smalldecide:00607d88f6e1:float32-cpu | 2026-10-02T20:59:15+00:00 | 9bc3424a301c36ce532c2422dee9968dacc8fa71 | 00607d88f6e152b471e557f6a814471ab5a6faf900cae427d8b3d50d4614be17 | float32-cpu |
Dataset SHA-256: 2d9deda7ef5ddb211957b760572b2cf29afca48b5385c51e131fc5c21a9198d4.
AI output verification ai-output-verification@1
Read the supported checks alongside these results: the mix of item types affects both coverage and review load.
| Configuration | Items | Decided coverage | Coverage gain (pp) | Uncertain per 1,000 | True, rules: errors / n | True, with model | False, rules | False, with model |
|---|---|---|---|---|---|---|---|---|
| Rules only | 5462 | 8.0% | 0.0 pp | 920.5 | – | – | 0 / 434, ≤ 0.69% | – |
With smalldecide 00607d88f6e1:float32-cpu | 5462 | 21.1% | 13.2 pp | 788.7 | – | 0 / 641, ≤ 0.47% | 0 / 434, ≤ 0.69% | 0 / 79, ≤ 3.72% |
Faults
| Fault | Items | False verdicts | False or uncertain |
|---|---|---|---|
| amount_from_wrong_field | 21 | 100.0% | 100.0% |
| broken_locator | 91 | 100.0% | 100.0% |
| contradiction | 80 | 0.0% | 100.0% |
| hallucinated_value | 147 | 100.0% | 100.0% |
| invented_number | 55 | 100.0% | 100.0% |
| misquote | 34 | 100.0% | 100.0% |
| unsupported_claim | 95 | 0.0% | 100.0% |
| wrong_field | 72 | 0.0% | 100.0% |
| wrong_number | 88 | 100.0% | 100.0% |
| wrong_type | 8 | 100.0% | 100.0% |
Look-alikes
| Look-alike | Items | False verdicts | Uncertain |
|---|---|---|---|
| curly_quotes | 36 | 0.0% | 100.0% |
| elided_quote | 21 | 0.0% | 100.0% |
| multibyte_span | 50 | 0.0% | 100.0% |
| paraphrased_statement | 75 | 0.0% | 100.0% |
| reformatted_number | 17 | 0.0% | 100.0% |
| reformatted_value | 70 | 0.0% | 100.0% |
Holdout seeds 101, 102, 103, 104, 105, 106, 107, 108, 109, 110; dataset SHA-256 5b4d8c20209f197c….
Run identity and dataset
| Configuration | Recorded at | Source commit | Model artifact SHA-256 | Dtype |
|---|---|---|---|---|
| deterministic_only | 2026-10-02T16:08:37+00:00 | 9bc3424a301c36ce532c2422dee9968dacc8fa71 | none | none |
| smalldecide:00607d88f6e1:float32-cpu | 2026-10-03T04:17:45+00:00 | 3f342f426d49e735d132c30c3ea2646e07f57f9f | 00607d88f6e152b471e557f6a814471ab5a6faf900cae427d8b3d50d4614be17 | float32-cpu |
Dataset SHA-256: 5b4d8c20209f197cba778eb52a7a0e9ef1eaf248559a3eea8641f91f0a2cdca1.
Contract compliance contract-compliance@1
Read the supported checks alongside these results: the mix of item types affects both coverage and review load.
| Configuration | Items | Decided coverage | Coverage gain (pp) | Uncertain per 1,000 | True, rules: errors / n | True, with model | False, rules | False, with model |
|---|---|---|---|---|---|---|---|---|
| Rules only | 4458 | 27.1% | 0.0 pp | 729.5 | 0 / 814, ≤ 0.37% | – | 0 / 392, ≤ 0.76% | – |
With smalldecide 00607d88f6e1:float32-cpu | 4458 | 27.1% | 0.0 pp | 729.5 | 0 / 814, ≤ 0.37% | – | 0 / 392, ≤ 0.76% | – |
Faults
| Fault | Items | False verdicts | False or uncertain |
|---|---|---|---|
| amount_miscomputed | 41 | 100.0% | 100.0% |
| billed_outside_term | 38 | 100.0% | 100.0% |
| cap_exceeded | 30 | 100.0% | 100.0% |
| currency_mismatch | 44 | 100.0% | 100.0% |
| duplicate_invoice | 36 | 100.0% | 100.0% |
| non_reimbursable_expense | 41 | 0.0% | 100.0% |
| out_of_scope_work | 39 | 0.0% | 100.0% |
| rate_above_card | 41 | 100.0% | 100.0% |
| rate_increase_early | 43 | 100.0% | 100.0% |
| sla_credit_missing | 46 | 100.0% | 100.0% |
| total_mismatch | 37 | 100.0% | 100.0% |
| unlisted_role | 36 | 100.0% | 100.0% |
Look-alikes
| Look-alike | Items | False verdicts | Uncertain |
|---|---|---|---|
| borderline_in_scope | 30 | 0.0% | 100.0% |
| borderline_reimbursable | 20 | 0.0% | 100.0% |
| cap_reached_exactly | 10 | 0.0% | 0.0% |
| final_invoice_after_term | 20 | 0.0% | 0.0% |
| goodwill_credit | 20 | 0.0% | 0.0% |
| half_cent_rounding | 23 | 0.0% | 100.0% |
| rate_increase_effective | 57 | 0.0% | 100.0% |
| recurring_fixed_fee | 12 | 0.0% | 0.0% |
| sla_at_threshold | 20 | 0.0% | 0.0% |
| sla_breach_credited | 20 | 0.0% | 0.0% |
| supplementary_invoice | 20 | 0.0% | 0.0% |
| weekend_cutover | 20 | 0.0% | 100.0% |
Holdout seeds 101, 102, 103, 104, 105, 106, 107, 108, 109, 110; dataset SHA-256 5192466d83436be8….
Run identity and dataset
| Configuration | Recorded at | Source commit | Model artifact SHA-256 | Dtype |
|---|---|---|---|---|
| deterministic_only | 2026-10-02T16:08:36+00:00 | 9bc3424a301c36ce532c2422dee9968dacc8fa71 | none | none |
| smalldecide:00607d88f6e1:float32-cpu | 2026-10-02T18:51:58+00:00 | 9bc3424a301c36ce532c2422dee9968dacc8fa71 | 00607d88f6e152b471e557f6a814471ab5a6faf900cae427d8b3d50d4614be17 | float32-cpu |
Dataset SHA-256: 5192466d83436be8a81ec0efdfe6dee5440eea7f30cc03a0fdcc831fe66f01d5.
Financial audit financial-audit@4
financial-audit@4 is measured on its own holdout dataset (0043b0a536a0bc1c…), not the one financial-audit@1, financial-audit@2 and financial-audit@3 were measured on, so its numbers are not directly comparable with theirs.
Read the supported checks alongside these results: the mix of item types affects both coverage and review load.
| Configuration | Items | Decided coverage | Coverage gain (pp) | Uncertain per 1,000 | True, rules: errors / n | True, with model | False, rules | False, with model |
|---|---|---|---|---|---|---|---|---|
| Rules only | 8100 | 51.4% | 0.0 pp | 486.4 | 0 / 1932, ≤ 0.15% | – | 0 / 2228, ≤ 0.13% | – |
With smalldecide 00607d88f6e1:float32-cpu | 8100 | 87.2% | 35.9 pp | 127.7 | 0 / 3203, ≤ 0.09% | 0 / 1578, ≤ 0.19% | 0 / 2253, ≤ 0.13% | 0 / 32, ≤ 8.94% |
Faults
| Fault | Items | False verdicts | False or uncertain |
|---|---|---|---|
| approval_after_payment | 30 | 100.0% | 100.0% |
| approval_by_claimant_user | 30 | 100.0% | 100.0% |
| approved_faulty_claim | 336 | 100.0% | 100.0% |
| approved_faulty_invoice | 215 | 100.0% | 100.0% |
| approver_without_authority | 30 | 100.0% | 100.0% |
| arithmetic_error | 56 | 100.0% | 100.0% |
| category_code_mismatch | 55 | 100.0% | 100.0% |
| claim_after_termination | 52 | 100.0% | 100.0% |
| claim_before_hire | 55 | 100.0% | 100.0% |
| claim_outside_period | 52 | 100.0% | 100.0% |
| creator_also_pays | 50 | 100.0% | 100.0% |
| creator_without_role | 30 | 100.0% | 100.0% |
| duplicate_claim | 103 | 100.0% | 100.0% |
| duplicate_invoice_amount_date | 108 | 100.0% | 100.0% |
| duplicate_invoice_number | 96 | 100.0% | 100.0% |
| duplicate_invoice_typo | 112 | 100.0% | 100.0% |
| duplicate_payment | 60 | 100.0% | 100.0% |
| inactive_vendor_invoice | 53 | 100.0% | 100.0% |
| incomplete_vendor | 30 | 100.0% | 100.0% |
| invoice_approver_without_authority | 30 | 100.0% | 100.0% |
| invoice_entered_and_approved | 30 | 100.0% | 100.0% |
| miscategorized_claim | 89 | 0.0% | 100.0% |
| missing_po | 74 | 100.0% | 100.0% |
| missing_receipt | 82 | 100.0% | 100.0% |
| near_duplicate_claim | 110 | 100.0% | 100.0% |
| over_balance_payment | 30 | 100.0% | 100.0% |
| over_limit | 85 | 100.0% | 100.0% |
| payment_before_invoice | 30 | 100.0% | 100.0% |
| payment_before_verification | 60 | 100.0% | 100.0% |
| payment_to_employee_account | 30 | 100.0% | 100.0% |
| payment_to_other_account | 30 | 100.0% | 100.0% |
| personal_expense | 91 | 0.0% | 100.0% |
| release_by_invoice_approver | 30 | 100.0% | 100.0% |
| release_unverified_bank_change | 30 | 100.0% | 100.0% |
| releaser_without_authority | 30 | 100.0% | 100.0% |
| self_approval | 30 | 100.0% | 100.0% |
| self_verified_bank_change | 30 | 100.0% | 100.0% |
| shared_receipt | 108 | 100.0% | 100.0% |
| split_claim | 123 | 100.0% | 100.0% |
| subordinate_approval | 30 | 100.0% | 100.0% |
| unjustified_exception | 40 | 0.0% | 100.0% |
| unmatched_payment | 30 | 100.0% | 100.0% |
| unverified_bank_change | 30 | 100.0% | 100.0% |
| vague_invoice | 69 | 0.0% | 100.0% |
| vendor_employee_address | 30 | 100.0% | 100.0% |
| vendor_employee_bank | 134 | 100.0% | 100.0% |
Look-alikes
| Look-alike | Items | False verdicts | Uncertain |
|---|---|---|---|
| justified_exception | 40 | 0.0% | 100.0% |
| new_vendor_large_payment | 10 | 0.0% | 0.0% |
| recurring_invoice | 55 | 0.0% | 100.0% |
| repeat_purchase | 60 | 0.0% | 100.0% |
| round_amount | 55 | 0.0% | 100.0% |
| same_merchant_under_limit | 60 | 0.0% | 100.0% |
| team_dinner | 60 | 0.0% | 100.0% |
| weekend_travel | 50 | 0.0% | 100.0% |
Holdout seeds 101, 102, 103, 104, 105, 106, 107, 108, 109, 110; dataset SHA-256 0043b0a536a0bc1c….
Run identity and dataset
| Configuration | Recorded at | Source commit | Model artifact SHA-256 | Dtype |
|---|---|---|---|---|
| deterministic_only | 2026-10-07T05:00:29+00:00 | a35faac58138ed25537ebf9d6ee8f7eb215926d3 | none | none |
| smalldecide:00607d88f6e1:float32-cpu | 2026-10-07T06:08:26+00:00 | a35faac58138ed25537ebf9d6ee8f7eb215926d3 | 00607d88f6e152b471e557f6a814471ab5a6faf900cae427d8b3d50d4614be17 | float32-cpu |
Dataset SHA-256: 0043b0a536a0bc1c880de974087a20182e1c9620ca62fc19f000677b42bdb507.
Financial audit financial-audit@3 (superseded)
Superseded, and still served to requests that name financial-audit@3. These results are not results for the current version. The use-case page lists what changed between versions.
Read the supported checks alongside these results: the mix of item types affects both coverage and review load.
| Configuration | Items | Decided coverage | Coverage gain (pp) | Uncertain per 1,000 | True, rules: errors / n | True, with model | False, rules | False, with model |
|---|---|---|---|---|---|---|---|---|
| Rules only | 8100 | 51.4% | 0.0 pp | 486.4 | 0 / 1932, ≤ 0.15% | – | 0 / 2228, ≤ 0.13% | – |
With smalldecide 00607d88f6e1:float32-cpu | 8100 | 87.3% | 35.9 pp | 127.0 | 0 / 3203, ≤ 0.09% | 0 / 1583, ≤ 0.19% | 0 / 2253, ≤ 0.13% | 0 / 32, ≤ 8.94% |
Faults
| Fault | Items | False verdicts | False or uncertain |
|---|---|---|---|
| approval_after_payment | 30 | 100.0% | 100.0% |
| approval_by_claimant_user | 30 | 100.0% | 100.0% |
| approved_faulty_claim | 336 | 100.0% | 100.0% |
| approved_faulty_invoice | 215 | 100.0% | 100.0% |
| approver_without_authority | 30 | 100.0% | 100.0% |
| arithmetic_error | 56 | 100.0% | 100.0% |
| category_code_mismatch | 55 | 100.0% | 100.0% |
| claim_after_termination | 52 | 100.0% | 100.0% |
| claim_before_hire | 55 | 100.0% | 100.0% |
| claim_outside_period | 52 | 100.0% | 100.0% |
| creator_also_pays | 50 | 100.0% | 100.0% |
| creator_without_role | 30 | 100.0% | 100.0% |
| duplicate_claim | 103 | 100.0% | 100.0% |
| duplicate_invoice_amount_date | 108 | 100.0% | 100.0% |
| duplicate_invoice_number | 96 | 100.0% | 100.0% |
| duplicate_invoice_typo | 112 | 100.0% | 100.0% |
| duplicate_payment | 60 | 100.0% | 100.0% |
| inactive_vendor_invoice | 53 | 100.0% | 100.0% |
| incomplete_vendor | 30 | 100.0% | 100.0% |
| invoice_approver_without_authority | 30 | 100.0% | 100.0% |
| invoice_entered_and_approved | 30 | 100.0% | 100.0% |
| miscategorized_claim | 89 | 0.0% | 100.0% |
| missing_po | 74 | 100.0% | 100.0% |
| missing_receipt | 82 | 100.0% | 100.0% |
| near_duplicate_claim | 110 | 100.0% | 100.0% |
| over_balance_payment | 30 | 100.0% | 100.0% |
| over_limit | 85 | 100.0% | 100.0% |
| payment_before_invoice | 30 | 100.0% | 100.0% |
| payment_before_verification | 60 | 100.0% | 100.0% |
| payment_to_employee_account | 30 | 100.0% | 100.0% |
| payment_to_other_account | 30 | 100.0% | 100.0% |
| personal_expense | 91 | 0.0% | 100.0% |
| release_by_invoice_approver | 30 | 100.0% | 100.0% |
| release_unverified_bank_change | 30 | 100.0% | 100.0% |
| releaser_without_authority | 30 | 100.0% | 100.0% |
| self_approval | 30 | 100.0% | 100.0% |
| self_verified_bank_change | 30 | 100.0% | 100.0% |
| shared_receipt | 108 | 100.0% | 100.0% |
| split_claim | 123 | 100.0% | 100.0% |
| subordinate_approval | 30 | 100.0% | 100.0% |
| unjustified_exception | 40 | 0.0% | 100.0% |
| unmatched_payment | 30 | 100.0% | 100.0% |
| unverified_bank_change | 30 | 100.0% | 100.0% |
| vague_invoice | 69 | 0.0% | 100.0% |
| vendor_employee_address | 30 | 100.0% | 100.0% |
| vendor_employee_bank | 134 | 100.0% | 100.0% |
Look-alikes
| Look-alike | Items | False verdicts | Uncertain |
|---|---|---|---|
| justified_exception | 40 | 0.0% | 100.0% |
| new_vendor_large_payment | 10 | 0.0% | 0.0% |
| recurring_invoice | 55 | 0.0% | 100.0% |
| repeat_purchase | 60 | 0.0% | 100.0% |
| round_amount | 55 | 0.0% | 100.0% |
| same_merchant_under_limit | 60 | 0.0% | 100.0% |
| team_dinner | 60 | 0.0% | 100.0% |
| weekend_travel | 50 | 0.0% | 100.0% |
Holdout seeds 101, 102, 103, 104, 105, 106, 107, 108, 109, 110; dataset SHA-256 4e9ecbb5f3f57984….
Run identity and dataset
| Configuration | Recorded at | Source commit | Model artifact SHA-256 | Dtype |
|---|---|---|---|---|
| deterministic_only | 2026-10-05T07:14:20+00:00 | c30094b628dbb3b792b7dd78e8ebef2cd6b072ea | none | none |
| smalldecide:00607d88f6e1:float32-cpu | 2026-10-05T10:11:13+00:00 | ef160343de4711e2616e17e7edc5fb595a8c626a | 00607d88f6e152b471e557f6a814471ab5a6faf900cae427d8b3d50d4614be17 | float32-cpu |
Dataset SHA-256: 4e9ecbb5f3f57984f485ba1ae3362afc72bd62222592bb62c461f63dbfbb38e9.
Financial audit financial-audit@2 (superseded)
Superseded, and still served to requests that name financial-audit@2. These results are not results for the current version. The use-case page lists what changed between versions.
Read the supported checks alongside these results: the mix of item types affects both coverage and review load.
| Configuration | Items | Decided coverage | Coverage gain (pp) | Uncertain per 1,000 | True, rules: errors / n | True, with model | False, rules | False, with model |
|---|---|---|---|---|---|---|---|---|
| Rules only | 8100 | 52.9% | 0.0 pp | 471.0 | 0 / 2057, ≤ 0.15% | – | 0 / 2228, ≤ 0.13% | – |
With smalldecide 00607d88f6e1:float32-cpu | 8100 | 89.3% | 36.4 pp | 106.9 | 0 / 3366, ≤ 0.09% | 0 / 1583, ≤ 0.19% | 0 / 2253, ≤ 0.13% | 0 / 32, ≤ 8.94% |
Faults
| Fault | Items | False verdicts | False or uncertain |
|---|---|---|---|
| approval_after_payment | 30 | 100.0% | 100.0% |
| approval_by_claimant_user | 30 | 100.0% | 100.0% |
| approved_faulty_claim | 336 | 100.0% | 100.0% |
| approved_faulty_invoice | 215 | 100.0% | 100.0% |
| approver_without_authority | 30 | 100.0% | 100.0% |
| arithmetic_error | 56 | 100.0% | 100.0% |
| category_code_mismatch | 55 | 100.0% | 100.0% |
| claim_after_termination | 52 | 100.0% | 100.0% |
| claim_before_hire | 55 | 100.0% | 100.0% |
| claim_outside_period | 52 | 100.0% | 100.0% |
| creator_also_pays | 50 | 100.0% | 100.0% |
| creator_without_role | 30 | 100.0% | 100.0% |
| duplicate_claim | 103 | 100.0% | 100.0% |
| duplicate_invoice_amount_date | 108 | 100.0% | 100.0% |
| duplicate_invoice_number | 96 | 100.0% | 100.0% |
| duplicate_invoice_typo | 112 | 100.0% | 100.0% |
| duplicate_payment | 60 | 100.0% | 100.0% |
| inactive_vendor_invoice | 53 | 100.0% | 100.0% |
| incomplete_vendor | 30 | 100.0% | 100.0% |
| invoice_approver_without_authority | 30 | 100.0% | 100.0% |
| invoice_entered_and_approved | 30 | 100.0% | 100.0% |
| miscategorized_claim | 89 | 0.0% | 100.0% |
| missing_po | 74 | 100.0% | 100.0% |
| missing_receipt | 82 | 100.0% | 100.0% |
| near_duplicate_claim | 110 | 100.0% | 100.0% |
| over_balance_payment | 30 | 100.0% | 100.0% |
| over_limit | 85 | 100.0% | 100.0% |
| payment_before_invoice | 30 | 100.0% | 100.0% |
| payment_before_verification | 60 | 100.0% | 100.0% |
| payment_to_employee_account | 30 | 100.0% | 100.0% |
| payment_to_other_account | 30 | 100.0% | 100.0% |
| personal_expense | 91 | 0.0% | 100.0% |
| release_by_invoice_approver | 30 | 100.0% | 100.0% |
| release_unverified_bank_change | 30 | 100.0% | 100.0% |
| releaser_without_authority | 30 | 100.0% | 100.0% |
| self_approval | 30 | 100.0% | 100.0% |
| self_verified_bank_change | 30 | 100.0% | 100.0% |
| shared_receipt | 108 | 100.0% | 100.0% |
| split_claim | 123 | 100.0% | 100.0% |
| subordinate_approval | 30 | 100.0% | 100.0% |
| unjustified_exception | 40 | 0.0% | 100.0% |
| unmatched_payment | 30 | 100.0% | 100.0% |
| unverified_bank_change | 30 | 100.0% | 100.0% |
| vague_invoice | 69 | 0.0% | 100.0% |
| vendor_employee_address | 30 | 100.0% | 100.0% |
| vendor_employee_bank | 134 | 100.0% | 100.0% |
Look-alikes
| Look-alike | Items | False verdicts | Uncertain |
|---|---|---|---|
| justified_exception | 40 | 0.0% | 100.0% |
| new_vendor_large_payment | 10 | 0.0% | 0.0% |
| recurring_invoice | 55 | 0.0% | 100.0% |
| repeat_purchase | 60 | 0.0% | 100.0% |
| round_amount | 55 | 0.0% | 100.0% |
| same_merchant_under_limit | 60 | 0.0% | 100.0% |
| team_dinner | 60 | 0.0% | 100.0% |
| weekend_travel | 50 | 0.0% | 100.0% |
Holdout seeds 101, 102, 103, 104, 105, 106, 107, 108, 109, 110; dataset SHA-256 4e9ecbb5f3f57984….
Run identity and dataset
| Configuration | Recorded at | Source commit | Model artifact SHA-256 | Dtype |
|---|---|---|---|---|
| deterministic_only | 2026-10-04T17:05:46+00:00 | 074468fafc584805dd7ec4da045751ae4d6ecea4 | none | none |
| smalldecide:00607d88f6e1:float32-cpu | 2026-10-04T19:27:50+00:00 | 4c826cec83c4ce6fdfa368aa641499eca0b296ce | 00607d88f6e152b471e557f6a814471ab5a6faf900cae427d8b3d50d4614be17 | float32-cpu |
Dataset SHA-256: 4e9ecbb5f3f57984f485ba1ae3362afc72bd62222592bb62c461f63dbfbb38e9.
Financial audit financial-audit@1 (superseded)
Superseded, and still served to requests that name financial-audit@1. These results are not results for the current version. The use-case page lists what changed between versions.
Read the supported checks alongside these results: the mix of item types affects both coverage and review load.
| Configuration | Items | Decided coverage | Coverage gain (pp) | Uncertain per 1,000 | True, rules: errors / n | True, with model | False, rules | False, with model |
|---|---|---|---|---|---|---|---|---|
| Rules only | 8100 | 52.9% | 0.0 pp | 471.0 | 0 / 2057, ≤ 0.15% | – | 0 / 2228, ≤ 0.13% | – |
With smalldecide 00607d88f6e1:float32-cpu | 8100 | 89.3% | 36.4 pp | 106.9 | 0 / 3366, ≤ 0.09% | 0 / 1583, ≤ 0.19% | 0 / 2253, ≤ 0.13% | 0 / 32, ≤ 8.94% |
Faults
| Fault | Items | False verdicts | False or uncertain |
|---|---|---|---|
| approval_after_payment | 30 | 100.0% | 100.0% |
| approval_by_claimant_user | 30 | 100.0% | 100.0% |
| approved_faulty_claim | 336 | 100.0% | 100.0% |
| approved_faulty_invoice | 215 | 100.0% | 100.0% |
| approver_without_authority | 30 | 100.0% | 100.0% |
| arithmetic_error | 56 | 100.0% | 100.0% |
| category_code_mismatch | 55 | 100.0% | 100.0% |
| claim_after_termination | 52 | 100.0% | 100.0% |
| claim_before_hire | 55 | 100.0% | 100.0% |
| claim_outside_period | 52 | 100.0% | 100.0% |
| creator_also_pays | 50 | 100.0% | 100.0% |
| creator_without_role | 30 | 100.0% | 100.0% |
| duplicate_claim | 103 | 100.0% | 100.0% |
| duplicate_invoice_amount_date | 108 | 100.0% | 100.0% |
| duplicate_invoice_number | 96 | 100.0% | 100.0% |
| duplicate_invoice_typo | 112 | 100.0% | 100.0% |
| duplicate_payment | 60 | 100.0% | 100.0% |
| inactive_vendor_invoice | 53 | 100.0% | 100.0% |
| incomplete_vendor | 30 | 100.0% | 100.0% |
| invoice_approver_without_authority | 30 | 100.0% | 100.0% |
| invoice_entered_and_approved | 30 | 100.0% | 100.0% |
| miscategorized_claim | 89 | 0.0% | 100.0% |
| missing_po | 74 | 100.0% | 100.0% |
| missing_receipt | 82 | 100.0% | 100.0% |
| near_duplicate_claim | 110 | 100.0% | 100.0% |
| over_balance_payment | 30 | 100.0% | 100.0% |
| over_limit | 85 | 100.0% | 100.0% |
| payment_before_invoice | 30 | 100.0% | 100.0% |
| payment_before_verification | 60 | 100.0% | 100.0% |
| payment_to_employee_account | 30 | 100.0% | 100.0% |
| payment_to_other_account | 30 | 100.0% | 100.0% |
| personal_expense | 91 | 0.0% | 100.0% |
| release_by_invoice_approver | 30 | 100.0% | 100.0% |
| release_unverified_bank_change | 30 | 100.0% | 100.0% |
| releaser_without_authority | 30 | 100.0% | 100.0% |
| self_approval | 30 | 100.0% | 100.0% |
| self_verified_bank_change | 30 | 100.0% | 100.0% |
| shared_receipt | 108 | 100.0% | 100.0% |
| split_claim | 123 | 100.0% | 100.0% |
| subordinate_approval | 30 | 100.0% | 100.0% |
| unjustified_exception | 40 | 0.0% | 100.0% |
| unmatched_payment | 30 | 100.0% | 100.0% |
| unverified_bank_change | 30 | 100.0% | 100.0% |
| vague_invoice | 69 | 0.0% | 100.0% |
| vendor_employee_address | 30 | 100.0% | 100.0% |
| vendor_employee_bank | 134 | 100.0% | 100.0% |
Look-alikes
| Look-alike | Items | False verdicts | Uncertain |
|---|---|---|---|
| justified_exception | 40 | 0.0% | 100.0% |
| new_vendor_large_payment | 10 | 0.0% | 0.0% |
| recurring_invoice | 55 | 0.0% | 100.0% |
| repeat_purchase | 60 | 0.0% | 100.0% |
| round_amount | 55 | 0.0% | 100.0% |
| same_merchant_under_limit | 60 | 0.0% | 100.0% |
| team_dinner | 60 | 0.0% | 100.0% |
| weekend_travel | 50 | 0.0% | 100.0% |
Holdout seeds 101, 102, 103, 104, 105, 106, 107, 108, 109, 110; dataset SHA-256 4e9ecbb5f3f57984….
Run identity and dataset
| Configuration | Recorded at | Source commit | Model artifact SHA-256 | Dtype |
|---|---|---|---|---|
| deterministic_only | 2026-10-02T16:08:34+00:00 | 9bc3424a301c36ce532c2422dee9968dacc8fa71 | none | none |
| smalldecide:00607d88f6e1:float32-cpu | 2026-10-02T16:11:37+00:00 | 9bc3424a301c36ce532c2422dee9968dacc8fa71 | 00607d88f6e152b471e557f6a814471ab5a6faf900cae427d8b3d50d4614be17 | float32-cpu |
Dataset SHA-256: 4e9ecbb5f3f57984f485ba1ae3362afc72bd62222592bb62c461f63dbfbb38e9.
Insurance claims insurance-claims@2
Read the supported checks alongside these results: the mix of item types affects both coverage and review load.
| Configuration | Items | Decided coverage | Coverage gain (pp) | Uncertain per 1,000 | True, rules: errors / n | True, with model | False, rules | False, with model |
|---|---|---|---|---|---|---|---|---|
| Rules only | 4853 | 46.8% | 0.0 pp | 531.6 | 0 / 1763, ≤ 0.17% | – | 0 / 510, ≤ 0.59% | – |
With smalldecide 00607d88f6e1:float32-cpu | 4853 | 46.8% | 0.0 pp | 531.6 | 0 / 1763, ≤ 0.17% | – | 0 / 510, ≤ 0.59% | – |
Faults
| Fault | Items | False verdicts | False or uncertain |
|---|---|---|---|
| diverted_account | 35 | 100.0% | 100.0% |
| duplicate_claim | 80 | 100.0% | 100.0% |
| items_inconsistent | 44 | 0.0% | 100.0% |
| items_total_mismatch | 39 | 100.0% | 100.0% |
| loss_outside_period | 41 | 100.0% | 100.0% |
| narrative_mismatch | 41 | 0.0% | 100.0% |
| over_limit | 40 | 100.0% | 100.0% |
| overpayment | 34 | 100.0% | 100.0% |
| peril_not_covered | 46 | 100.0% | 100.0% |
| policy_lapsed | 40 | 100.0% | 100.0% |
| provider_link | 33 | 100.0% | 100.0% |
| reported_late | 41 | 100.0% | 100.0% |
| self_approved | 43 | 100.0% | 100.0% |
| wrong_payee | 38 | 100.0% | 100.0% |
Look-alikes
| Look-alike | Items | False verdicts | Uncertain |
|---|---|---|---|
| account_formatting | 16 | 0.0% | 0.0% |
| amount_at_limit | 18 | 0.0% | 100.0% |
| approver_other_adjuster | 20 | 0.0% | 0.0% |
| cancelled_after_loss | 18 | 0.0% | 100.0% |
| deductible_absorbs_excess | 18 | 0.0% | 100.0% |
| loss_on_policy_boundary | 17 | 0.0% | 100.0% |
| paid_above_claimed | 19 | 0.0% | 0.0% |
| paid_to_provider | 20 | 0.0% | 0.0% |
| provider_same_street | 21 | 0.0% | 100.0% |
| repeat_claim_later | 18 | 0.0% | 100.0% |
| repeat_claim_other_peril | 18 | 0.0% | 100.0% |
| reported_on_last_day | 16 | 0.0% | 100.0% |
| round_amount_itemized | 18 | 0.0% | 100.0% |
Holdout seeds 101, 102, 103, 104, 105, 106, 107, 108, 109, 110; dataset SHA-256 92adb019c9a07cb6….
Run identity and dataset
| Configuration | Recorded at | Source commit | Model artifact SHA-256 | Dtype |
|---|---|---|---|---|
| deterministic_only | 2026-10-07T04:08:41+00:00 | c7df92be8350d7e6afb7e5b6be60917e7e872b10 | none | none |
| smalldecide:00607d88f6e1:float32-cpu | 2026-10-07T04:12:53+00:00 | fda990f7052cbaf109483fea678cde178c92900f | 00607d88f6e152b471e557f6a814471ab5a6faf900cae427d8b3d50d4614be17 | float32-cpu |
Dataset SHA-256: 92adb019c9a07cb60a1b2b6f5d520020fea866ed529b3d8d75aed92334c277f9.
Insurance claims insurance-claims@1 (superseded)
Superseded, and still served to requests that name insurance-claims@1. These results are not results for the current version. The use-case page lists what changed between versions.
Read the supported checks alongside these results: the mix of item types affects both coverage and review load.
| Configuration | Items | Decided coverage | Coverage gain (pp) | Uncertain per 1,000 | True, rules: errors / n | True, with model | False, rules | False, with model |
|---|---|---|---|---|---|---|---|---|
| Rules only | 4853 | 46.8% | 0.0 pp | 531.6 | 0 / 1763, ≤ 0.17% | – | 0 / 510, ≤ 0.59% | – |
With smalldecide 00607d88f6e1:float32-cpu | 4853 | 46.8% | 0.0 pp | 531.6 | 0 / 1763, ≤ 0.17% | – | 0 / 510, ≤ 0.59% | – |
Faults
| Fault | Items | False verdicts | False or uncertain |
|---|---|---|---|
| diverted_account | 35 | 100.0% | 100.0% |
| duplicate_claim | 80 | 100.0% | 100.0% |
| items_inconsistent | 44 | 0.0% | 100.0% |
| items_total_mismatch | 39 | 100.0% | 100.0% |
| loss_outside_period | 41 | 100.0% | 100.0% |
| narrative_mismatch | 41 | 0.0% | 100.0% |
| over_limit | 40 | 100.0% | 100.0% |
| overpayment | 34 | 100.0% | 100.0% |
| peril_not_covered | 46 | 100.0% | 100.0% |
| policy_lapsed | 40 | 100.0% | 100.0% |
| provider_link | 33 | 100.0% | 100.0% |
| reported_late | 41 | 100.0% | 100.0% |
| self_approved | 43 | 100.0% | 100.0% |
| wrong_payee | 38 | 100.0% | 100.0% |
Look-alikes
| Look-alike | Items | False verdicts | Uncertain |
|---|---|---|---|
| account_formatting | 16 | 0.0% | 0.0% |
| amount_at_limit | 18 | 0.0% | 100.0% |
| approver_other_adjuster | 20 | 0.0% | 0.0% |
| cancelled_after_loss | 18 | 0.0% | 100.0% |
| deductible_absorbs_excess | 18 | 0.0% | 100.0% |
| loss_on_policy_boundary | 17 | 0.0% | 100.0% |
| paid_above_claimed | 19 | 0.0% | 0.0% |
| paid_to_provider | 20 | 0.0% | 0.0% |
| provider_same_street | 21 | 0.0% | 100.0% |
| repeat_claim_later | 18 | 0.0% | 100.0% |
| repeat_claim_other_peril | 18 | 0.0% | 100.0% |
| reported_on_last_day | 16 | 0.0% | 100.0% |
| round_amount_itemized | 18 | 0.0% | 100.0% |
Holdout seeds 101, 102, 103, 104, 105, 106, 107, 108, 109, 110; dataset SHA-256 92adb019c9a07cb6….
Run identity and dataset
| Configuration | Recorded at | Source commit | Model artifact SHA-256 | Dtype |
|---|---|---|---|---|
| deterministic_only | 2026-10-02T16:08:35+00:00 | 9bc3424a301c36ce532c2422dee9968dacc8fa71 | none | none |
| smalldecide:00607d88f6e1:float32-cpu | 2026-10-02T17:51:36+00:00 | 9bc3424a301c36ce532c2422dee9968dacc8fa71 | 00607d88f6e152b471e557f6a814471ab5a6faf900cae427d8b3d50d4614be17 | float32-cpu |
Dataset SHA-256: 92adb019c9a07cb60a1b2b6f5d520020fea866ed529b3d8d75aed92334c277f9.
Inspect the results or reproduce with source access
The published result JSON is public. The source repository is private; the commands below require repository access and a checkout at the recorded source revision. Run the rules-only holdout evaluation below. Replace the use-case ID, or name a version such as financial-audit@1, to reproduce another table. Preserve the output, source revision and dataset hash with your comparison.
python -m openauditor.evals.run --use-case financial-audit --split holdout \
--backend none --output run.jsonFor a model-assisted experiment, supply the exact model artifact and its qualification records. Loading a model does not qualify its answers.
python -m openauditor.evals.run --use-case financial-audit --split holdout \
--backend smalldecide --model /path/to/model --device cpu \
--qualifications /path/to/qualifications --output model-run.jsonBefore applying the results to your workflow
Build a reviewed sample that reflects your records, rules and failure costs. Keep reviewers independent of the system output where practical. Measure false passes, false alarms, evidence quality and unresolved volume separately. A corpus with many easy arithmetic faults can overstate the coverage of a workflow dominated by ambiguous descriptions or missing documents.
Keep evaluation and operational evidence distinct: a passing unit test, a fast model response and a successful API request each answer different questions from whether a business finding is correct. Our certainty investigation examines why the population of final decisions matters. Read how verdicts are reached or start a small integration.