Test your Windows installer on a clean computer

To test a Windows installer properly, run it on a machine that has never had your app, click through what a real user sees, and check install, first launch, upgrade and uninstall. Sai does that on a fresh Windows cloud computer, from Claude Code or CI, and sends back a screenshot of every step and the installer log.

What installer testing covers

Installer testing checks that your setup package installs, upgrades, repairs and removes your application correctly on the systems your users have. It covers the files, shortcuts, registry entries and prerequisites the installer creates, and the prompts the user sees along the way.

Microsoft's Windows Installer team put the most common mistake plainly: developers often test only on their own systems, where they have full administrator rights and every dependency (Windows Installer team).

Why installers that work on your machine fail on clean PCs

• Missing prerequisites. Your machine has the Visual C++ runtime or .NET version the app needs; a new user's machine does not.
• Prompts nobody tested. Silent installs with /qn skip the UAC prompt, the wizard pages and the license screen, so CI never sees them.
• SmartScreen. EV certificates no longer bypass SmartScreen, and a new release can take weeks to build reputation (Microsoft). Users see the warning; your test machine, which already trusts the file, does not.
• "Clean" machines that are not. GitHub-hosted Windows runners start fresh for each job, but the images come with a lot of developer software preinstalled and updated weekly (GitHub).
• Upgrade and uninstall get skipped. "Test uninstall as thoroughly as install," the same Microsoft team wrote. Leftover files and broken upgrades are what existing users hit.

Windows installer test checklist

1. Clean start: no earlier version of your app, no developer runtimes. Microsoft's MSIX guidance is to test on a machine where the previous installer was never installed (Microsoft).
2. Download and first warning: the browser download, SmartScreen or Smart App Control, and the signature details.
3. Install with the UI: every UAC prompt and wizard page, default options and at least one custom option.
4. Silent install: msiexec /i MyApp.msi /qn /l*v install.log, or your EXE's silent switch.
5. What got installed: install folder, Start menu and desktop shortcuts, Apps & features entry and version, file associations, services.
6. First launch: from the Start menu, no error dialogs, and no process left running after you close it.
7. Upgrade: install the previous release, use it once, upgrade, and check settings and data are kept.
8. Repair: delete a key file and run repair from Apps & features.
9. Uninstall: from Apps & features and the command line; nothing left behind, and nothing removed that was not yours.
10. Logs: keep the verbose log of every run. When an MSI fails, search it for Return value 3 (Microsoft).

Clean machines for installer testing, compared

Keep silent installs and log checks in CI for every commit. Use Sai for the full user path on release candidates: the download, the prompts, first launch, upgrade and uninstall.

How to test a Windows installer with Sai

1. Add Sai to your coding agent. One command adds the @simular-ai/sai-mcp server to Claude Code; Codex and Cursor work the same way.
2. Point to the build and the previous release. A release or artifact link to each installer. MSI, MSIX and EXE installers from WiX, Inno Setup, NSIS, InstallShield or electron-builder all work, because Sai uses the installer's own screens.
3. List the checks. Start from the checklist above. The full prompt is in the Start with a prompt tab.
4. Watch or come back later. Claude Code shows a watch_url link with the live screen. Steps outside the test wait for your approval.
5. Fix and re-run. Claude Code reads the report and the log, fixes the packaging, and asks Sai to run the checks again on a clean computer.

On the official OSWorld 2.1 leaderboard, which tests agents on real desktop tasks, Sai has the highest partial score, 79.38%, at $14.34 per task. See the leaderboard and method.

Run it from CI

From a release pipeline, send the checks to POST /v1/agents/message and fail the job if any check fails. success_criteria asks Sai to judge each check and return evidence, and recordTrajectory keeps a screenshot of every step:

curl -s https://api.simular.ai/v1/agents/message \
  -H "Authorization: Bearer $SAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "message": "On a computer with no MyApp installed, install {{installer}}, launch it from the Start menu, then uninstall it.",
    "options": {
      "inputs": { "installer": "https://github.com/acme/myapp/releases/download/v2.5.0/MyApp-2.5.0-x64.msi" },
      "success_criteria": [
        { "id": "installed", "check": "MyApp 2.5.0 is listed in Apps & features", "returnEvidence": true },
        { "id": "launch", "check": "The main window opened with no error dialog", "returnEvidence": true },
        { "id": "removed", "check": "The install folder and Start menu shortcut are gone after uninstall", "returnEvidence": true }
      ],
      "recordTrajectory": true
    },
    "wait": false
  }'

Poll /turns for the result, and read the steps and screenshots from the turn's trajectory. The quick start covers keys and the first call, and computers explains which machine a task runs on.

Limitations

• A cloud computer is one configuration. It is a Windows machine without your app, not every Windows edition, language and hardware mix your users have.
• Slower than a silent install. A full UI run takes minutes; keep it for release candidates, not every commit.
• It does not replace package validation. Keep ICE validation for MSI tables and your own unit tests.
• SmartScreen results vary. Reputation depends on Microsoft's view of the file at that moment, so a warning on one run may not repeat.

Related use cases: test your desktop app from Claude Code, test a signup flow end to end and reproduce a bug report step by step.

Last reviewed October 17, 2026.