A computer use API lets your agent do work in apps that have no API, by handing the task to an agent that operates a real computer. With Sai you add one HTTP call, POST /v1/agents/message, as a tool in LangGraph, the OpenAI Agents SDK or the Vercel AI SDK. Your agent sends a task in plain English and gets the result back, without you building a VM or a screenshot loop.
What a computer use API is
A computer use API is an interface through which software can ask an AI model to operate a graphical computer: read the screen, move the pointer, click and type. Some computer use APIs return single actions that your own code must carry out and repeat. Others take a whole task and return the outcome.
That difference decides how much you build. The model-level tools from Anthropic and OpenAI are the first kind. A task-level API such as Sai is the second kind, so it fits in your agent as an ordinary tool.
When your agent needs a computer instead of an API
Use an API whenever a good one exists. It is faster, cheaper and easier to test. A computer use tool earns its place when:
- There is no API. Supplier and carrier portals, government sites, and older internal tools often have only a web or desktop screen.
- The API does not cover the step. The export, the setting or the approval queue you need exists only in the UI.
- It is desktop software. Some accounting, design or line-of-business apps run only on a Mac or Windows machine.
- The task crosses apps. Download from a portal, check a sheet, file the PDF: one task, three screens.
If a browser alone is enough and you want full control of the code, a browser automation library may be the better fit (see the comparison below).
Tasks agents hand to a computer use tool
These are the calls back-office and support agents make most often when an app has no API. Each asks for a result your agent can act on.
| Your agent | Task it hands to Sai | Result shape to ask for |
|---|---|---|
| Procurement agent | Check invoice status in a supplier portal | One row per invoice: status, amount, due date |
| Support agent | Look up a shipment in a carrier portal | Tracking status, last scan, delivery estimate |
| Finance agent | Download last month's statements from bank and vendor portals | File names and where they were saved |
| Operations agent | Update records in a legacy desktop app | Records changed, with before and after values |
| Sales agent | Fill CRM fields that the API does not expose | Fields updated per account, and any skipped |
| Research agent | Collect figures from a site behind a login | A table with the source page for each value |
How to wrap a computer use API as a tool for your agent
These rules apply whichever computer use API you use:
- One tool, one clear job. Name it something like
computer_taskand write a description that says when to use it: "Use only for apps with no API. Describe the task, the site, and the result you need." In LangChain the docstring becomes the description, and type hints define the input schema. The OpenAI Agents SDK also reads the docstring and type hints. The AI SDK takes adescriptionand aninputSchema. - Send a complete task. The tool takes one text field: the task. Include the URL, what to read or change, what is off limits, and the shape of the answer. A vague task returns a vague result.
- Ask for a structured result. Tell the tool to return a table or named fields, and validate them before your agent acts on them.
- Keep approvals with a person. Gate consequential calls in your framework too. LangChain's human-in-the-loop middleware can approve, edit or reject a tool call before it runs. The Agents SDK supports
needs_approvalon a tool, which pauses the run. The AI SDK has a tool approval flow. - Set a time limit. Screen work takes minutes, not milliseconds. The Agents SDK supports a
timeouton async function tools; the AI SDK forwards an abort signal into your tool. State a limit in the task as well. - Return failures as results. "Invoice not found" or "sign-in denied" should come back as text the agent can reason about, not a crash.
Where each setting lives in your framework:
| Framework | Define the tool | Gate it for approval | Limit the time |
|---|---|---|---|
| LangChain / LangGraph | A Python function; the docstring is the description and type hints are the schema | Human-in-the-loop middleware: approve, edit or reject the call | State the limit in the task |
| OpenAI Agents SDK | A function tool; the docstring and type hints become the schema | needs_approval on the tool pauses the run | timeout on async function tools |
| Vercel AI SDK | tool() with a description and an inputSchema | The SDK's tool approval flow | The abort signal passed into your tool |
| Claude Code, Codex, Cursor | No code: add the Sai MCP server | The client's own tool prompts, plus Sai's approvals | State the limit in the task |
In TypeScript-like pseudo-code, the whole tool is: computerTask(task: string) → POST /v1/agents/message with { message: task } → return the reply text. For authentication, the exact request and response fields and how to wait for a long task, see the API reference.
Ways to give an AI agent a computer, compared
| Approach | What you build | Reaches | Best for |
|---|---|---|---|
| Do the step by hand | A hand-off in your agent | Anything | Rare steps |
| Anthropic computer use tool | A sandboxed desktop, action handlers, screenshot capture and scaling, the agent loop, approvals | Whatever your environment runs | Teams that want full control of the harness |
| OpenAI computer use tool | The environment, the action handler, screenshots, the loop and session state | Browser or desktop you provide | Teams already on the Responses API that want control |
| Browser agents and cloud browsers (Browser Use, Browserbase) | Agent code and browser sessions | Websites | Web-only tasks with code you own |
| Sai via the Simular API | One tool that sends a task | Websites and desktop apps on a hosted computer or your own Mac or Windows machine | Delegating whole tasks from an existing agent |
If you need step-by-step control of every click, or the task never leaves a browser, the model-level tools or a browser library give you more control for more work.
How to give your agent a computer with Sai
- Get an API key and read the quick start.
- Wrap the call as a tool in your framework, with the description and limits above.
- Write the task like a brief. An example is in the Start with a prompt tab above.
- Handle approvals. Sai asks before sign-ins, payments, submits, sends and deletes. Credentials are approved through a link, so passwords never pass through your agent or its logs.
- Return the result to your agent and let it continue the run.
Sai ranks #1 on OSWorld, the public benchmark for agents that complete tasks on a real computer. Once Sai completes a task, its steps are kept as code that replays the same way, so the next call with the same task is faster and cheaper. Paid plans add dedicated computers, one task each, so your agent can run several tasks at once.
From your coding agent: the same capability is available as the Sai MCP server in Claude Code, Codex or Cursor. See GUI steps from coding agents.
Safety for agents that use a computer
Anthropic and OpenAI give similar advice for their computer use tools, and it applies here:
- Isolate the computer. Use a dedicated machine with minimal privileges and only the accounts the task needs.
- Treat screen content as untrusted. Text on a page can contain instructions. It must not override your task or grant permission.
- Confirm consequential actions. Purchases, sending data, accepting terms and deletions go to a person.
- Bound the run. Set time and step limits, and check the actual outcome, not only the final message.
- Respect site terms. Use portals you are authorized to use, at a human pace, and do not resell data you collect.
Related use cases: turn a runbook into a scheduled job and test your signup flow end to end.
Last reviewed October 7, 2026.