AI output verification

Check an extracted value or generated sentence against its supplied sources before it enters a report, business record or customer-facing answer.

For teams shipping document extraction, retrieval-augmented answers and generated summaries.

Where it fits in your workflow

Use this after document extraction, retrieval or generation. Submit one extracted field or one generated sentence as the item, and send the source text with it. The resulting verdict is attached to that unit of output, which makes it possible to send a specific value or sentence for correction without treating a whole document as equally reliable.

1. Keep the source with the output. Provide the text of the relevant documents under stable IDs. For an extracted field, include its declared type and, where available, a text quote or UTF-8 byte span locating the value.
2. Separate presence from meaning. The direct checks establish whether values, quotations and numbers occur in the supplied text, whether types parse and whether declared totals reconcile. A separate judgment question checks whether a value belongs to the requested field or a sentence is supported by its sources.
3. Correct the smallest unresolved unit. Route an unsupported sentence or ambiguous field back to the extraction or review step. Retain the original source, locator, returned evidence and configuration with the output you eventually accept.

What to send

• Extracted field. Document ID, field name, value as a string and value_type: text, number, money, date or percent. A field description helps state what the extraction was intended to capture.
• Generated statement. One sentence and the IDs of the documents it cites. Split a longer answer into meaningful statements when you need a verdict for each one.
• Source documents. Document records containing the exact text to check. The service uses this supplied text; a document ID or URL alone does not cause it to retrieve a source.
• Locators and totals. Optional byte_span or text_quote evidence for a field, related extracted fields, and declared sum relations in context.references.relations. Byte spans count UTF-8 bytes, not character positions.

Use POST /api/v1/verify for one item or POST /api/v1/audits for a population with shared context. See the request structure.

The total is present, typed and reconciled

The recorded extraction reports a total_due of $505.53 from an invoice. Its source contains “Total due: $505.53” at the supplied byte span. Related fields hold a $467.00 subtotal and $38.53 tax, and the request declares that the total equals their sum.

The source-location, money-type and total-reconciliation checks all hold. The right-field question remains unqualified, so the recorded result is uncertain. The reviewer can inspect the nearby “Total due” label to confirm the field assignment. Finding a value in a document and adding its components correctly do not, by themselves, establish that it is the field requested.

Recorded verdict: uncertain. The table below comes from the saved response.

Recorded from the production service on 2026-10-03T04:20:15Z. This page displays a saved example, not a new verification request. Sensitive identifiers are masked in public examples.

{
  "use_case": "ai-output-verification",
  "item": {
    "type": "extracted_field",
    "id": "F-3",
    "data": {
      "document_id": "DOC-1",
      "field": "total_due",
      "field_description": "the total amount the customer must pay",
      "value": "505.53",
      "value_type": "money",
      "locator": {"kind": "byte_span", "start": 279, "end": 285, "text": "505.53"}
    }
  },
  "context": {
    "records": {
      "document": [
        {
          "id": "DOC-1",
          "text": "Northwind Traders\n200 Harbor Way, Portland\n\nInvoice number: INV-20417\nInvoice date: August 22, 2026\nBill to: Harbor Point Supply\n\n1  Steel brackets, 40 mm   12 x 18.50   222.00\n2  Hinge set               4 x 61.25    245.00\n\nSubtotal: 467.00\nSales tax (8.25%): 38.53\nTotal due: $505.53\n\nPayment is due within 30 days of the invoice date.\n",
          "media_type": "text/plain"
        }
      ],
      "extracted_field": [
        {
          "id": "F-1",
          "document_id": "DOC-1",
          "field": "subtotal",
          "field_description": "the sum of the line amounts before tax",
          "value": "467.00",
          "value_type": "money"
        },
        {
          "id": "F-2",
          "document_id": "DOC-1",
          "field": "tax_amount",
          "field_description": "the amount of sales tax charged",
          "value": "38.53",
          "value_type": "money"
        }
      ]
    },
    "references": {
      "relations": [
        {
          "document_id": "DOC-1",
          "field": "total_due",
          "equals": "sum",
          "of": ["subtotal", "tax_amount"]
        }
      ]
    }
  }
}

{
  "id": "ver_26fc39ae940a6751",
  "object": "verification",
  "created_at": "2026-10-03T04:20:15Z",
  "use_case": "ai-output-verification@1",
  "item": {"type": "extracted_field", "id": "F-3"},
  "verdict": "uncertain",
  "probability_true": "1.0000",
  "probability_basis": "calibrated",
  "certainty": null,
  "undecided": [
    {
      "assertion": "right_field",
      "reason": "unqualified",
      "message": "This question is not yet qualified for the running model; its answer is recorded but decides nothing."
    }
  ],
  "assertions": [
    {
      "id": "value_located",
      "test": "procedure:value_located@1",
      "outcome": "holds",
      "error_bound": "0",
      "facts": {
        "how": "exact",
        "matches": 1,
        "start": 279,
        "end": 285,
        "locator": "byte_span",
        "cited_start": 279,
        "cited_end": 285
      },
      "evidence": ["/item/data/value", "/context/records/document/0/text", "/item/data/locator"]
    },
    {
      "id": "value_typed",
      "test": "procedure:value_typed@1",
      "outcome": "holds",
      "error_bound": "0",
      "facts": {"value_type": "money", "parsed": "505.53"},
      "evidence": ["/item/data/value", "/item/data/value_type"]
    },
    {
      "id": "totals_consistent",
      "test": "procedure:totals_consistent@1",
      "outcome": "holds",
      "error_bound": "0",
      "facts": {
        "relations": [{"relation": 0, "stated": "505.53", "computed": "505.53", "holds": true}]
      },
      "evidence": [
        "/item/data/value",
        "/context/references/relations/0",
        "/context/records/extracted_field/0/value",
        "/context/records/extracted_field/1/value"
      ]
    },
    {
      "id": "right_field",
      "test": "question:right_field@1",
      "outcome": "unqualified",
      "message": "This question is not yet qualified for the running model; its answer is recorded but decides nothing.",
      "answer": {"p_yes": "0.995507"},
      "facts_given": {
        "located": true,
        "match_count": 1,
        "excerpt": {"document_id": "DOC-1", "start": 84, "end": 338}
      },
      "evidence": ["/item/data/field", "/item/data/value", "/context/records/document/0/text"]
    }
  ],
  "indicators": [],
  "risk": {
    "points": "0.00",
    "probability": "0.0200",
    "basis": "prior",
    "base_rate": "0.02",
    "reasons": []
  },
  "scope": {
    "stratum": null,
    "evidence": "supplied",
    "skipped": [],
    "certainty_target": "0.995",
    "deterministic_only_verdict": "uncertain"
  },
  "configuration": "cfg_74933712990f",
  "input_sha256": "42c397a9f788d229a1acc0660b2c758ed212b72999e3578de313df900309e729",
  "usage": {"questions_asked": 1, "questions_cached": 0, "model_ms": 2202}
}

Supported records and checks

These definitions are generated from the registered use-case pack. A procedure computes a result from the supplied data; a question can decide a check only when it is qualified for the model and meets its thresholds. The descriptions name the checks; they do not authenticate the underlying business events. Supplied fields, defaults and comparison populations determine what each procedure can establish. Risk weights help prioritize review and are not probabilities.

• extracted_field: One field an AI extracted from a document: its value as a string, its declared type, and optionally where in the document it was read.
• statement: One sentence an AI generated from the documents it cites.

extracted_field

One field an AI extracted from a document: its value as a string, its declared type, and optionally where in the document it was read. Required data fields: document_id, field, value, value_type.

statement

One sentence an AI generated from the documents it cites. Required data fields: statement, document_ids.

Judgment questions and qualification

The table describes published evaluation evidence. For the qualification attached to the running service, inspect this use case’s live definition. A model answer can be recorded without deciding a check. How qualification works.

Scope and limits

• The pack checks supplied text, not the original scan or image. If upstream extraction omitted or misread content, a matching text value does not establish that the original document was read correctly.
• Text, number and date comparisons have explicit normalization rules. Ambiguous numeric dates and recognized near matches can remain unresolved instead of being forced into a pass or failure.
• A number or quotation can be present in a source while the sentence around it is unsupported. Source presence and semantic support remain separate checks.

A true verdict means all applicable, unskipped checks hold within the decision gates; false means at least one qualifying failure decided the item. Other assertions can remain unresolved even with a false verdict. An uncertain verdict means the evidence and decision gates do not support a decided result. Review the assertion outcomes and evidence before taking a business action. Read the decision method.

Published evaluation results

These measurements use synthetic populations with generator-stated truth. Coverage is the share receiving true or false verdicts, not measured accuracy on your records. Read the metrics and their limits.

Results for ai-output-verification@1.

Faults

Look-alikes

Holdout seeds 101, 102, 103, 104, 105, 106, 107, 108, 109, 110; dataset SHA-256 5b4d8c20209f197c….

Schemas and integration

Every item, record and reference schema is at GET /api/v1/use-cases/ai-output-verification.

Supported context record types: document, extracted_field, statement.

Supported reference sets: relations.

Start with a reviewable sample

Choose a few high-impact output fields or sentence types. Supply stable source text and locators, then compare results against independently reviewed examples before deciding which outputs your workflow can accept automatically.

Try the console · API quickstart · Full API reference

From the blog

• How to verify AI-extracted data against source documents
• 99.5% of which decisions?