API reference

Judge

POST /v1/agents/judge. Check recorded turns against criteria you choose now (research preview).

Checks what Sai did in one or more turns against criteria you choose now, without running anything again. Use it to check old runs against a new rule, or to get evidence for repairing learned code. A judgement never changes the turns it judged.

Research preview: the request and response may change.

Request

POST /v1/agents/judge

json

{
  "turns": ["turn_vzjz62w7j7jvOilZ6zBL"],
  "criteria": [
    {
      "id": "title",
      "check": "The agent reported the title of the top story",
      "returnEvidence": true
    }
  ]
}

Field	Type	Required	Notes
`turns`	array	yes	Turn ids from one session, judged together as one task. Each must have ended. At most 20.
`criteria`	string or array	yes	A sentence for a model to judge against (at most 4,000 characters), or up to 20 checks `{ id, check, returnEvidence? }`, as in `success_criteria`.
`task`	string	no	The task to judge against. Default: the first turn's message.
`judges`	object	no	`{ criteria: true }`. Step-by-step judging of screenshots (`transitions`) and checking a live computer (`probe`) come later and return `400`.
`idempotency_key`	string	no	The same key again returns the earlier judgement.

The judge reads the turns' steps, their output and the final answers. It does not see screenshots yet.

Response

200:

json

{
  "judgementId": "judgement_8LmQ2rT0bV",
  "verdict": "passed",
  "summary": "1 of 1 checks passed.",
  "task": "Open https://news.ycombinator.com and read the title of the top story",
  "criteria": [
    {
      "id": "title",
      "check": "The agent reported the title of the top story",
      "verdict": "passed",
      "score": 0.97,
      "evidence": "The final answer gives the title \"Show HN: ...\" read from the front page."
    }
  ],
  "turns": ["turn_vzjz62w7j7jvOilZ6zBL"],
  "usage": { "inputTokens": 5120, "outputTokens": 140 },
  "researchPreview": true
}

Field	Notes
`verdict`	`passed`, `failed`, or `unverifiable` when a criterion could not be checked.
`criteria`	One entry per criterion. `score` is the probability that a check holds; `evidence` is always set for a sentence and, for a check, with `returnEvidence`.
`usage`	Tokens of the judge's model calls, billed to your account.

Treat the verdict as advice: text on the screen can try to steer a model judge.

Read a judgement later

GET /v1/agents/judge/{judgementId}

Errors

Status	Body
`400`	Validation failed, `mixed_sessions`, or `not_supported` for `transitions` and `probe`.
`402`	Not enough credits.
`404`	`{ "error": "turn_not_found" }`, or `judgement_not_found` on `GET`.
`409`	`{ "error": "turn_not_finished" }`.
`429`	More than 120 judge calls in an hour.