What Blocks a Deploy: Risk-Tiering Tests for a Production Site

Every test suite implicitly answers a question. Most only answer "does the code work?" The question that actually shapes a suite is different: which failure is bad enough to stop a release?

Answer that honestly and the rest of the architecture falls out — because the interesting decision isn't what you test, it's what you deliberately allow to break without blocking a deploy. That's a risk call, and this site's Playwright suite is small enough to show the whole call in one post.

(Updated August 2026 — rewritten around the release-gate design; the original tutorial content on page objects and debugging lives in the Playwright docs, where it belongs.)

Three Gates, Three Prices

Tests here run at three points, each with a different cost of failure and a different cost of running:

| Gate | When | Blocks | Price of the gate | |------|------|--------|-------------------| | PR gate | Every pull request | The merge | Minutes added to every change | | Main matrix | Push to main | Nothing (alerts) | CI time, no human wait | | Production smoke | Post-deploy + daily cron | Nothing (alerts + auto-filed issue) | Live-site traffic |

The asymmetry is the design. A PR gate failure stops a human in their tracks, so everything in it must be fast and deterministic. The other two tiers can afford breadth and flakiness because their failures page someone instead of blocking someone.

The Honest Part: the Gate Is a File List, Not a Tag

The suite uses @smoke / @infra / @prod tags, and it would be easy to tell you the PR gate "runs the smoke tests." It doesn't. The gate is an explicit testMatch list in the Playwright config:

// playwright.config.ts — the chromium project IS the PR gate
testMatch: process.env.E2E_BROWSERS === 'all' ? '**/*.spec.ts' : [
  '**/routing.spec.ts',
  '**/content.spec.ts',
  '**/atom.spec.ts',
  '**/terminal.spec.ts',
  '**/sandbox.spec.ts'
],

The full suite is eleven spec files. Five block a merge:

tests/e2e/
├── routing.spec.ts           # PR gate — navigation, URLs, 404s
├── content.spec.ts           # PR gate — post rendering, MDX
├── atom.spec.ts              # PR gate — feed validity
├── terminal.spec.ts          # PR gate — the easter-egg shell
├── sandbox.spec.ts           # PR gate — interactive tools + URL state
├── graph.spec.ts             # tagged @smoke — NOT in the gate
├── analytics.spec.ts         # tagged @smoke — NOT in the gate
├── ai-governed-data.spec.ts  # tagged @smoke — NOT in the gate
├── responsive.spec.ts        # full-matrix runs only
├── infrastructure.spec.ts    # @infra — headers, CDN behavior
└── production-smoke.spec.ts  # @prod — live-site monitoring

Notice the middle three: they carry the @smoke tag and still don't block a PR. The knowledge-graph tests drive a heavy canvas that's slow and occasionally unstable in constrained environments; blocking every merge on them would trade real developer time for coverage of a feature that degrades visibly and harmlessly. That's a defensible call — but only if it's visible. A tag that says "smoke" while the config says otherwise is how test suites quietly lie to their owners. The testMatch list is the truth; the tags are aspiration. I'd rather document the gap than pretend the tag is the policy.

The demotion isn't free, and I won't dress it up: those specs currently run in no automated environment — a gap on the follow-up list, not a feature.

What Earns a Place in the Blocking Tier

Three requirements, strictly enforced:

Deterministic. Anything that can fail for reasons other than the change under review is disqualified. Third-party calls are the usual offender, so every gated test blocks them at the network layer:

static async blockExternalCalls(page: Page): Promise<void> {
  await page.route('**/*', (route) => {
    const url = route.request().url();
    const blockedDomains = ['google-analytics.com', 'googletagmanager.com'];
    if (blockedDomains.some((d) => url.includes(d))) return route.abort();
    return route.continue();
  });
}

Fast. The gate runs against the static export served locally — no backend, files serve instantly, parallel workers don't contend. The whole blocking tier finishes in well under a minute of test time.

User-visible on failure. Every gated assertion maps to something a reader would actually hit: a post that doesn't render, a feed that doesn't parse, a broken route, a sandbox tool that loses its URL state. Internal refactors that don't change any of those sail through — which is the point of testing outcomes instead of implementation.

The Alerting Tiers

The main-branch matrix adds browsers and the demoted specs — breadth that would be too expensive per-PR but is cheap per-merge. Production smoke runs after every deploy and on a daily schedule, against the live CDN, where the failure modes are different in kind: edge-cache variance, header regressions, redirect breakage. Those tests get retries by design:

{
  name: 'production-smoke',
  retries: 3,          // CDN edges disagree briefly; three strikes is signal
  timeout: 15000,
}

A production failure doesn't roll anything back — it files a GitHub issue automatically. For a content site, alert-and-investigate beats block-and-page at this tier; on a revenue system, I'd draw that line differently, which is exactly the point: the tiering is the risk statement.

One operational footnote worth stealing: bare playwright test with no --project flag runs every project — including production smoke against the live site. The config makes it one typo to point your test run at production. pnpm e2e pins --project=chromium; use the script, not the raw CLI.

The Same Question at Any Scale

Deciding what blocks a deploy is the same decision as deciding what blocks a release of a revenue system: which failures are cheap enough to detect after the fact, and which must be impossible to ship? The tooling here is a personal-site miniature, but the discipline transfers — the deployment pipeline this suite gates applies it end to end, and operational complexity survives the build whether the thing you shipped is a static site or a sales-engagement platform.

A test suite is a policy document that happens to execute.