How we QA every pull request automatically
The QA agent we run on Valkor — connected to GitHub and our test environment. It checks out each PR, runs the suite, exercises the change, and posts the result.
Code review catches problems a person can find by reading a diff. It misses the ones you only find by running the change: a test that passes locally but flakes in CI, a path that works in the happy case and 500s on an edge case, a migration that reads fine but locks a table under load. These tend to surface in staging or production, after the PR is approved.
We run a QA agent on Valkor that does this checking when the PR opens. This is how we QA our own changes, including the connections and guardrails involved.
The problem
Some bugs don't show up in a diff. A reviewer reads the code and approves, but no one has checked out the branch, run the suite, hit the new endpoint, and watched what happens. CI runs the tests the author wrote; it doesn't cover the paths they missed.
The common fixes are incomplete. Green CI shows the existing tests pass, not that the change is correct. A manual QA pass is thorough but slow and lands after the review. A generic AI reviewer only sees the diff; it can't run the branch, so it misses failures that only appear at runtime.
What we built
On Valkor, each PR triggers an agent. On open and on every push, the PR spawns an isolated session (a cloud sandbox) with scoped access to the branch, the test suite, a deployable test environment, and the Cloudflare edge in front of it. The agent checks out the change, runs the suite, exercises the new behavior on a live deploy, and posts a pass/fail result on the PR.
How it works
Connect GitHub as the trigger
A signed GitHub webhook points at the project. Every PR opened or updated fires it, and each firing spawns a fresh session in its own sandbox, seeded with the branch under test. One PR maps to one session on one disposable machine, so nothing carries over between runs and concurrent PRs run in parallel.
Give the agent the branch and the test playbook
The session checks out the branch and installs it clean. Our QA conventions live as skills and memory that travel with the agent: how to run the suite, which flows are critical, edge cases that have caused problems before, and what a result should contain. When a bug slips through, we write it down and the agent picks it up on the next run.
Connect the systems QA needs
Through scoped connectors, brokered server-side so no raw token reaches the model, the agent can:
- Run the full suite — unit, integration, and e2e inside the sandbox, with the failure output captured in full.
- Deploy to the test environment — it stands the change up on an ephemeral deploy and exercises it end-to-end against the new paths.
- Reach the edge via Cloudflare — it checks behavior through the edge (routing, headers, caching, redirects), not just localhost.
- Report on GitHub — the pass/fail result and any reproduction post as a check and a comment on the PR.
Set the guardrails
The agent operates against the test environment only; production is out of scope. It does not merge or deploy to prod. Its output is a result, and a human owns the merge. Credentials are encrypted in the Secrets Manager and injected at runtime, never shown to the model or written to logs.
Let each PR arrive pre-QA'd
With that in place, a new PR checks itself out, runs the suite, deploys to the test environment, exercises the change through Cloudflare, and posts a result: green when it passes, or a red check with the failing command, the logs, and steps to reproduce. A flaky test is flagged with the evidence. A broken edge route is caught before staging.
Summary
A trigger on every PR spawns a session with scoped connectors into the branch, the test suite, the test environment, and Cloudflare. The QA playbook is encoded as skills and memory. The agent stays on the test environment and a human owns the merge.
Guardrails
The agent has access to the test environment and the edge, so the access is scoped and contained:
- Isolation. Every PR runs in its own microVM sandbox on its own branch. The session can install, deploy, and exercise the change to reproduce a failure; only the reported result leaves the sandbox.
- Scoped secrets. The GitHub, test-environment, and Cloudflare credentials are encrypted in the Secrets Manager and injected into the sandbox at runtime, never exposed to the model or the logs.
- Test environment only. The agent's access stops at the test plane: no production access, no prod deploy, no merge. It reports; the team decides.
- Everything is code. The agent's configuration, skills, and permissions are files in the repo, versioned and changed through a reviewed change request rather than a dashboard setting.
The outcome
Runtime failures that a diff can't show now surface on the PR when it opens, with the failing command, the logs, and the repro attached. Reviewers spend their time on design rather than checking out branches by hand, and the changes that reach staging have already been run against the test environment.
Read more
How we draft incident postmortems
When an incident resolves, an agent pulls the timeline from the incident channel, correlates deploys and log spikes, and drafts a structured postmortem as a doc PR for the team to review and edit.
How we generate release notes from merged PRs
The release-notes agent we run on Valkor — connected to GitHub. On each release it reads the merged PRs since the last one, groups them, writes the notes, and opens a changelog PR.
How we catch cloud cost spikes before the bill lands
The cloud-cost anomaly agent we run on Valkor — connected to AWS Cost Explorer and Slack. Every day it keeps a running spend baseline per service and account, flags whatever breaks out of that baseline, attributes the likely driver, and alerts with the delta. Read-only and alert-only; it never touches a resource or a budget.
Engineer your first loop
Give your team a workforce of agents that runs on a schedule, ships real deliverables, and improves each change through review.
Controlled delivery · SSO · RBAC · human review