Proactive and schedule-driven: one independent QA pass per PR revision found in
a sweep, handled as its own unit so a failure on one never blocks the others.
The only state carried forward on purpose is the landed edge-case ledger — an
in-sandbox file alone doesn't survive on its own.
Step 0 — Discover and orient
# Read the known-issues ledger before touching any branch — edge cases that
# have bitten before, flows that are critical, flaky tests already on file.
cat .kortix/memory/qa-known-issues.md 2>/dev/null || echo "(no known issues yet)"
# Discover every PR opened or pushed to since the last sweep.
gh pr list --repo {{target_repo}} --state open \
--json number,headRefOid,statusCheckRollup,updatedAt
Filter out any PR whose current headRefOid already carries a qa-agent
check in statusCheckRollup — don't re-run against a revision you've already
reported on. Work through what's left one PR at a time, each as its own
independent unit: a failure on one PR is a self-contained result and never
blocks or contaminates QA of the others in this sweep.
For each remaining PR, confirm before checkout:
gh pr view --repo {{target_repo}} <PR_NUMBER> --json headRefOid,statusCheckRollup
Step 1 — Check out the branch clean
rm -rf /workspace/pr-<PR_NUMBER>
git clone --filter=blob:none https://github.com/{{target_repo}}.git /workspace/pr-<PR_NUMBER>
cd /workspace/pr-<PR_NUMBER>
gh pr checkout <PR_NUMBER>
<install command for the stack> # e.g. pnpm install --frozen-lockfile / npm ci / pip install -r requirements.txt
One clean clone per PR revision, keyed by PR number. This session persists
across sweeps, so remove any stale checkout for this PR number first — never
reuse a prior revision's working tree, and never let one PR's checkout leak
into another's.
Step 2 — Run the full suite
cd /workspace/pr-<PR_NUMBER>
<unit test command> 2>&1 | tee /tmp/qa-unit.log
<integration test command> 2>&1 | tee /tmp/qa-integration.log
<e2e test command> 2>&1 | tee /tmp/qa-e2e.log
Capture full failure output — stack trace, failing assertion, the exact
command — not just a pass/fail count. Cross-reference qa-known-issues.md for
flows that have broken before and confirm this run actually exercised them.
Step 3 — Deploy the change to the test environment
<deploy command, e.g. an internal deploy script / flyctl / vercel> \
--env test --ref <PR head SHA>
Stand up an ephemeral instance of exactly this branch. Wait for the deploy to
report healthy before exercising anything against it — a QA run against a
half-started deploy is a fail, not a skip.
Step 4 — Exercise the change end-to-end
Hit the new or changed behavior directly against the deployed instance — the
actual endpoint, page, or flow the PR touches — not just localhost:
curl -sf https://<test-deploy-host>/<changed-path> -o /tmp/qa-response.json
Then repeat the critical checks through the edge:
curl -sI https://<edge-fronted-test-host>/<changed-path> # routing, headers, caching, redirects
This is a plain, unauthenticated request against the public HTTPS host — no
edge-provider credential or connector is needed for it. A route that works
direct-to-origin but breaks through the edge (a caching rule, a redirect, a
header transform) is a QA failure, not a pass.
Step 5 — Decide pass or fail
| Result | Criteria |
|---|
| Pass | Suite green, deploy healthy, exercised behavior matches expectation, edge checks clean |
| Fail | Any suite failure, a failed or unhealthy deploy, exercised behavior wrong, or an edge-only regression |
| Flag as flaky | A test fails, then passes on an isolated re-run with no code change — report it as flaky with both runs' logs; don't just retry until it's green |
Step 6 — Post the result
gh api repos/{{target_repo}}/statuses/<PR_head_SHA> \
-f state=<success|failure> -f context="qa-agent" -f description="<one-line result>"
gh pr comment --repo {{target_repo}} <PR_NUMBER> --body "<pass/fail summary>"
Pass: a short green summary — suite results, what was exercised, edge checks
run. Fail: the failing command, the full log excerpt, and exact steps to
reproduce against the test deploy. Exactly one result comment per PR revision.
Step 7 — Tear down and land the ledger update
<teardown command for the ephemeral test deploy>
Tear down the ephemeral deploy for this PR so it doesn't linger — the
session's own sandbox stays up for the next sweep; only the test-environment
deploy is disposable.
If this run surfaced a genuinely new edge case (not already in the ledger) — a
bug that would've reached staging, a flow no prior run checked — append it to
.kortix/memory/qa-known-issues.md with the PR link and a one-line
description, then land it durably:
git add .kortix/memory/qa-known-issues.md
git commit -m "docs(qa): record edge case from PR #<PR_NUMBER>"
Open (and self-merge) a scoped change request for just this ledger update via
the project.cr.open action — an edit that only lives in the sandbox never
survives on its own; only a landed change request does. Then move to the next
PR in this sweep's batch, if any, with a clean checkout (Step 1).