# Judge

POST /v1/agents/judge. Check recorded turns against criteria you choose now (research preview).

Checks what Sai did in one or more [turns](/documentation/api-reference/turns) against criteria you choose now, without running anything again. Use it to check old runs against a new rule, or to get evidence for [repairing learned code](/documentation/api-reference/learn). A judgement never changes the turns it judged.

Research preview: the request and response may change.

## Request

```
POST /v1/agents/judge
```

```json
{
  "turns": ["turn_vzjz62w7j7jvOilZ6zBL"],
  "criteria": [
    {
      "id": "title",
      "check": "The agent reported the title of the top story",
      "returnEvidence": true
    }
  ]
}
```

| Field             | Type            | Required | Notes                                                                                                                                              |
| ----------------- | --------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `turns`           | array           | yes      | Turn ids from one session, judged together as one task. Each must have ended. At most 20.                                                          |
| `criteria`        | string or array | yes      | A sentence for a model to judge against (at most 4,000 characters), or up to 20 checks `{ id, check, returnEvidence? }`, as in `success_criteria`. |
| `task`            | string          | no       | The task to judge against. Default: the first turn's message.                                                                                      |
| `judges`          | object          | no       | `{ criteria: true }`. Step-by-step judging of screenshots (`transitions`) and checking a live computer (`probe`) come later and return `400`.      |
| `idempotency_key` | string          | no       | The same key again returns the earlier judgement.                                                                                                  |

The judge reads the turns' steps, their output and the final answers. It does not see screenshots yet.

## Response

`200`:

```json
{
  "judgementId": "judgement_8LmQ2rT0bV",
  "verdict": "passed",
  "summary": "1 of 1 checks passed.",
  "task": "Open https://news.ycombinator.com and read the title of the top story",
  "criteria": [
    {
      "id": "title",
      "check": "The agent reported the title of the top story",
      "verdict": "passed",
      "score": 0.97,
      "evidence": "The final answer gives the title \"Show HN: ...\" read from the front page."
    }
  ],
  "turns": ["turn_vzjz62w7j7jvOilZ6zBL"],
  "usage": { "inputTokens": 5120, "outputTokens": 140 },
  "researchPreview": true
}
```

| Field      | Notes                                                                                                                                                    |
| ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `verdict`  | `passed`, `failed`, or `unverifiable` when a criterion could not be checked.                                                                             |
| `criteria` | One entry per criterion. `score` is the probability that a check holds; `evidence` is always set for a sentence and, for a check, with `returnEvidence`. |
| `usage`    | Tokens of the judge's model calls, billed to your account.                                                                                               |

Treat the verdict as advice: text on the screen can try to steer a model judge.

## Read a judgement later

```
GET /v1/agents/judge/{judgementId}
```

## Errors

| Status | Body                                                                                   |
| ------ | -------------------------------------------------------------------------------------- |
| `400`  | Validation failed, `mixed_sessions`, or `not_supported` for `transitions` and `probe`. |
| `402`  | Not enough credits.                                                                    |
| `404`  | `{ "error": "turn_not_found" }`, or `judgement_not_found` on `GET`.                    |
| `409`  | `{ "error": "turn_not_finished" }`.                                                    |
| `429`  | More than 120 judge calls in an hour.                                                  |