Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions .github/workflows/dashboard-e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,10 @@ on:
jobs:
playwright:
name: Dashboard Tests
runs-on: ubuntu-latest
# #1935: pinned, not ubuntu-latest. The floating label rolled to ubuntu-26.04, and the
# Playwright this repo resolves (1.59.x) ships no Chromium build for it (support landed in
# 1.61). Move this pin only together with a Playwright bump, verified on the new image.
runs-on: ubuntu-24.04
steps:
- name: Checkout repository
uses: actions/checkout@v4
Expand Down Expand Up @@ -38,6 +41,7 @@ jobs:
run: pnpm build

- name: Install Playwright browsers
id: install-browsers
working-directory: packages/codev
run: pnpm exec playwright install chromium

Expand All @@ -54,8 +58,10 @@ jobs:
# Spec 1819: the Decisions tab suite writes files, so it runs through its own config —
# no global setup activating the repository workspace — on a Tower that shares nothing
# with the one above: its own port, state dir and shellper socket dir.
# It runs even when the suite above fails, but not when there is no browser to launch
# (#1935): a failed install is reported once, at its own step, not as launch errors.
- name: Run Decisions tab tests
if: ${{ !cancelled() }}
if: ${{ !cancelled() && steps.install-browsers.outcome == 'success' }}
working-directory: packages/codev
env:
TOWER_TEST_PORT: '14101'
Expand Down
12 changes: 9 additions & 3 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,10 @@ jobs:

canvas-browser:
name: Artifact-Canvas Browser Tests
runs-on: ubuntu-latest
# #1935: pinned, not ubuntu-latest. The floating label rolled to ubuntu-26.04, and the
# Playwright this repo resolves (1.59.x) ships no Chromium build for it (support landed in
# 1.61). Move this pin only together with a Playwright bump, verified on the new image.
runs-on: ubuntu-24.04
# Backstop for #1502. The stall's root cause is removed below (see the install
# step), but every job stays bounded regardless: an unbounded job conceals its
# own failure, which is #1502's real lesson. Healthy runs finish in ~4 min.
Expand All @@ -147,7 +150,7 @@ jobs:
# system libraries, and that apt index fetch intermittently stalls indefinitely
# (the fast azure.archive.ubuntu.com mirror is Ign-ored, and apt has no network
# timeout on the fallback to archive.ubuntu.com). Those libraries are already
# present on the ubuntu-latest image: the sibling dashboard-e2e workflow installs
# present on the ubuntu-24.04 image: the sibling dashboard-e2e workflow installs
# Chromium the same way without `--with-deps` and launches it successfully on
# every scheduled run. Dropping the flag removes the stalling apt phase entirely.
- name: Install Playwright Chromium
Expand All @@ -162,7 +165,10 @@ jobs:

web-browser:
name: Web Terminal Browser Tests
runs-on: ubuntu-latest
# #1935: pinned, not ubuntu-latest. The floating label rolled to ubuntu-26.04, and the
# Playwright this repo resolves (1.59.x) ships no Chromium build for it (support landed in
# 1.61). Move this pin only together with a Playwright bump, verified on the new image.
runs-on: ubuntu-24.04
# Bounded like every job here (#1502). Healthy runs finish in ~3 min.
timeout-minutes: 15
steps:
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
id: bugfix-1935
title: ci-playwright-jobs-fail-on-ubu
protocol: bugfix
phase: pr
plan_phases: []
current_plan_phase: null
gates:
pr:
status: approved
requested_at: '2026-11-20T10:22:12.141Z'
approved_at: '2026-11-20T10:27:05.229Z'
iteration: 1
build_complete: false
history: []
started_at: '2026-11-20T10:11:54.134Z'
updated_at: '2026-11-20T10:27:05.230Z'
pr_ready_for_human: false
17 changes: 17 additions & 0 deletions codev/state/bugfix-1935_thread.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# bugfix-1935 thread

## Investigate (2026-11-20)
- Reproduced from CI logs: run 44264876229 (2026-11-18) ran on `Image: ubuntu-24.04`; run 44485513618 (2026-11-19) ran on `Image: ubuntu-26.04` (20261111.196.1). Install step: `Error: ERROR: Playwright does not support chromium on ubuntu26.04-x64`, exit 1.
- Lockfile resolves playwright/playwright-core 1.59.1. Checked playwright-core tarballs: 1.60.0 has no ubuntu26.04 entry; 1.61.0+ (through latest 1.65.1) do. So a Playwright bump is a real alternative, but it is a dependency + browser-revision jump; pinning the runner is the minimal, deterministic fix.
- Correction to the issue text: the install step DID fail its own step (bash -e, exit 1), and "Run Playwright tests" was skipped. The launch-error noise came from "Run Decisions tab tests", whose `if: ${{ !cancelled() }}` (meant to run it even if the main suite fails) also runs it after the browser install failed. Fix: gate it on the install step's outcome too.
- Other ubuntu-latest workflows (e2e, post-release-e2e, sdk-canary) do not use Playwright: out of scope.

## Fix (2026-11-20)
- Pinned dashboard-e2e `playwright`, test.yml `canvas-browser` and `web-browser` to `ubuntu-24.04` with a #1935 comment; Decisions tab step now also requires `steps.install-browsers.outcome == 'success'`.
- Regression test `bugfix-1935-playwright-runner-pin.test.ts` scans every workflow for jobs that run `playwright install`; 4 assertions fail on HEAD's workflows, pass with the fix.
- Adjacent fix: `protocol-drift-audit.test.ts` offline-lookup test failed deterministically on this machine because a user `~/.npmrc` `@cluesmith:registry=` outranks `npm_config_registry`; the test now overrides the scoped key too.

## PR (2026-11-20)
- PR #1936. Dashboard E2E dispatched on the branch (run 44718205786): all steps green on ubuntu-24.04. canvas-browser + web-browser green on the PR.
- CMAP: gemini skipped (agy exit 1); codex REQUEST_CHANGES only over porch-authored `chore(porch):` commit prefixes (not rewritten, porch owns those); claude APPROVE (test-header escape-hatch nit addressed).
- Suggested follow-up for the architect: bump Playwright >=1.61 and unpin before ubuntu-24.04 retires.
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ describe('bugfix-1502: the Chromium install does not run the stalling apt phase'
'canvas-browser must have a "playwright install" step',
).toBeDefined();
// `--with-deps` triggers the apt-get update that intermittently stalls (#1502).
// Chromium's system libraries are already present on the ubuntu-latest image
// Chromium's system libraries are already present on the pinned ubuntu-24.04 image
// (the dashboard-e2e workflow installs without it and launches Chromium fine).
// If a future runner image ever drops a library canvas needs, the fix is to
// restore --with-deps AND update this assertion with the reason — never to let
Expand Down
117 changes: 117 additions & 0 deletions packages/codev/src/__tests__/bugfix-1935-playwright-runner-pin.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
/**
* Regression test for bugfix #1935: Playwright jobs broke when `ubuntu-latest` rolled to
* Ubuntu 26.04.
*
* The Playwright this repo resolves (1.59.x) ships no Chromium build for ubuntu26.04, so
* `playwright install chromium` failed on the new image and every browser test with it.
* The fix is two-part, and this test pins both halves for every job that installs a
* Playwright browser, in every workflow:
* 1. The job runs on an explicitly versioned Ubuntu image, never the floating
* `ubuntu-latest`, so the image changes only by a deliberate edit.
* 2. A failed browser install is reported once, at its own step. A later step whose `if:`
* overrides the default success() gate (`!cancelled()`, `always()`, `failure()`) must
* also require the install step's success; otherwise it runs with no browser and
* buries the real cause under launch errors (the Decisions tab step did exactly that).
*
* If a later step legitimately must run after a failed install (an artifact or report upload
* that launches no browser), exempt it here by name AND state the reason, never by dropping
* the install gate silently. Unpinning the image belongs with a Playwright bump verified on
* the new image (Chromium on ubuntu26.04 needs Playwright 1.61+).
*/

import { describe, it, expect } from 'vitest';
import * as fs from 'node:fs';
import * as path from 'node:path';
import * as yaml from 'js-yaml';

const repoRoot = path.resolve(__dirname, '..', '..', '..', '..');
const workflowsDir = path.join(repoRoot, '.github/workflows');

interface WorkflowStep {
id?: string;
name?: string;
if?: string;
run?: string;
}

interface WorkflowJob {
'runs-on'?: string | string[];
steps?: WorkflowStep[];
}

interface Workflow {
jobs?: Record<string, WorkflowJob>;
}

interface PlaywrightJob {
label: string;
job: WorkflowJob;
installIndex: number;
}

const playwrightJobs: PlaywrightJob[] = fs
.readdirSync(workflowsDir)
.filter((file) => file.endsWith('.yml') || file.endsWith('.yaml'))
.flatMap((file) => {
const workflow = yaml.load(
fs.readFileSync(path.join(workflowsDir, file), 'utf-8'),
) as Workflow;
return Object.entries(workflow.jobs ?? {}).flatMap(([jobId, job]) => {
const installIndex = (job.steps ?? []).findIndex((step) =>
step.run?.includes('playwright install'),
);
if (installIndex === -1) {
return [];
}
return [{ label: `${file}:${jobId}`, job, installIndex }];
});
});

const overridesSuccessGate = (condition: string): boolean =>
/\b(always|failure)\(\)|!\s*cancelled\(\)/.test(condition);

describe('bugfix-1935: Playwright jobs survive runner image rollouts', () => {
it('finds the known Playwright jobs', () => {
const labels = playwrightJobs.map((entry) => entry.label);
expect(labels).toEqual(
expect.arrayContaining([
'dashboard-e2e.yml:playwright',
'test.yml:canvas-browser',
'test.yml:web-browser',
]),
);
});

it.each(playwrightJobs.map((entry) => [entry.label, entry] as const))(
'%s runs on a pinned Ubuntu image, not ubuntu-latest',
(label, { job }) => {
expect(
job['runs-on'],
`${label} installs a Playwright browser, so its image must change only by a ` +
'deliberate edit verified against Playwright support (see #1935)',
).toMatch(/^ubuntu-\d+\.\d+$/);
},
);

it.each(playwrightJobs.map((entry) => [entry.label, entry] as const))(
'%s runs no browser step after a failed install',
(label, { job, installIndex }) => {
const steps = job.steps ?? [];
const installId = steps[installIndex].id;
const ungated = steps
.slice(installIndex + 1)
.filter((step) => step.if !== undefined && overridesSuccessGate(step.if))
.filter(
(step) =>
installId === undefined ||
!step.if?.includes(`steps.${installId}.outcome == 'success'`),
)
.map((step) => step.name ?? step.run);
expect(
ungated,
`${label}: steps that run past a failure must also require the browser install ` +
'to have succeeded, so a missing browser is reported at the install step (#1935)',
).toEqual([]);
},
);
});
8 changes: 8 additions & 0 deletions packages/codev/src/__tests__/protocol-drift-audit.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -180,9 +180,15 @@ describe('protocol-drift-audit', () => {
// registry, asserting the non-functional guarantee from the plan: it returns null and
// completes within a bound. ECONNREFUSED is immediate; the spawnSync timeout
// (NPM_LATEST_TIMEOUT_MS) is the hard backstop, so a generous ceiling never flakes.
// A scoped registry (`@cluesmith:registry=` in a user .npmrc) outranks
// npm_config_registry for this package, so it is overridden too, or the lookup
// still reaches the live registry on such a machine.
const scopedRegistryKey = 'npm_config_@cluesmith:registry';
const prevRegistry = process.env.npm_config_registry;
const prevScopedRegistry = process.env[scopedRegistryKey];
const prevRetries = process.env.npm_config_fetch_retries;
process.env.npm_config_registry = 'http://127.0.0.1:1';
process.env[scopedRegistryKey] = 'http://127.0.0.1:1';
process.env.npm_config_fetch_retries = '0';
try {
const start = performance.now();
Expand All @@ -193,6 +199,8 @@ describe('protocol-drift-audit', () => {
} finally {
if (prevRegistry === undefined) delete process.env.npm_config_registry;
else process.env.npm_config_registry = prevRegistry;
if (prevScopedRegistry === undefined) delete process.env[scopedRegistryKey];
else process.env[scopedRegistryKey] = prevScopedRegistry;
if (prevRetries === undefined) delete process.env.npm_config_fetch_retries;
else process.env.npm_config_fetch_retries = prevRetries;
}
Expand Down
Loading