API reference

One call, three question types, one response shape. The Python API and the HTTP API carry identical JSON.

Concepts

A review is one call with one state and one or more questions. The state is any JSON value describing the situation. Each question is typed and names its own answer space in advance. The model scores every candidate answer independently and returns a probability distribution per question. Nothing is generated: there are no output tokens, and the answer can only be one of the candidates you supplied.

Question types

Choice: pick one of N

Choice("Which team should own this?", {"billing": "Payments and refunds", "technical": "Bugs and outages"})
{"type": "choice", "instructions": "Which team should own this?", "criteria": {"billing": "...", "technical": "..."}}

Score: place on an ordered rubric

Score("How urgent is this?", ["routine", "time-sensitive", "emergency"])
{"type": "score", "instructions": "How urgent is this?", "criteria": ["routine", "time-sensitive", "emergency"]}

Noul: yes or no, with a probability

Noul("Is a refund requested?")
Noul("Should the agent proceed?", {"true": "Proceed with the write", "false": "Stop and ask a human"})
{"type": "noul", "instructions": "Is a refund requested?"}

The call

OpenAuditor.from_pretrained(path, device="cuda") -> OpenAuditor
OpenAuditor.review(state, questions, *, model="openauditor") -> ReviewResponse

Questions in one call are independent: reordering them or adding an unrelated question does not change another question's probabilities. This is checked in the evaluation suite.

Response

{
  "model": "openauditor",
  "answers": {
    "team":        {"type": "choice", "choice": "technical",
                    "probabilities": {"billing": 0.04, "technical": 0.93, "sales": 0.03},
                    "confidence": 0.78},
    "urgent":      {"type": "noul", "noul": 0.91},
    "frustration": {"type": "score", "score": 1.62,
                    "legend": {"0": "calm", "1": "frustrated", "2": "very frustrated"},
                    "probabilities": {"0": 0.05, "1": 0.28, "2": 0.67},
                    "confidence": 0.31}
  },
  "usage": {"input_tokens": 41, "output_tokens": 0, "candidate_token_evaluations": 716}
}

HTTP

The local server exposes the same contract. Body and response are the JSON shapes above.

curl -s localhost:8080/v1/review -H 'content-type: application/json' -d '{
  "state": {"order_id": "A-1042", "amount": 1250, "customer_tier": "gold", "note": "duplicate charge, please refund"},
  "questions": {
    "action":  {"type": "choice", "instructions": "What should happen next?", "criteria": {"refund": "Issue a refund", "escalate": "Escalate to finance", "close": "Close with no action"}},
    "policy":  {"type": "noul",   "instructions": "Is this within the automatic refund policy?"},
    "risk":    {"type": "score",  "instructions": "Fraud risk", "criteria": ["low", "medium", "high"]}
  }}'

Any HTTP client works. In JavaScript: fetch("/v1/review", {method: "POST", headers: {"content-type": "application/json"}, body: JSON.stringify(request)}).

Confidence and calibration

• Noul is calibrated. The training loss is a proper score (cross-entropy plus Brier), and a temperature per question type is fitted on a held-out calibration slice that never overlaps test data. On the distribution you trained on, a Noul of 0.83 is meant to be right about 83% of the time. The evidence report publishes reliability bins.
• Choice and Score probabilities are calibrated the same way, but confidence is an entropy summary, not an accuracy estimate. Use the probabilities for thresholds; use confidence for sorting or display.
• Calibration does not transfer. A new domain needs its own validation data to refit temperatures and choose thresholds. Do not act on a threshold chosen on someone else's data.

Limits

Errors

The Python API raises ValueError; the HTTP server returns 422 with {"error": "..."}. Common causes:

• Choice with fewer than two criteria, or a criteria value that is not an object.
• Score with fewer than two levels, or a criteria value that is not a list.
• Noul criteria with keys other than true and false.
• Missing instructions, empty questions, or an unknown type.
• A candidate prompt over the token limit.