test(cli-app): close the branch-coverage gap and add the package to CI - #2402
Conversation
@percy/cli-app was the only one of 18 packages missing from both the test.yml and windows.yml matrices, so its 81 specs have never run in CI on any platform. Adding it as-is would have turned CI red: the package sits at 98.44% branch coverage against the repo's 100% threshold. The gap is `err.code || err.message`, interpolated into the warning at maestro-inject.js:157 and the debug line at :273. Every existing spec throws an error carrying a code (EACCES, EROFS, EEXIST, ENOENT), so the `err.message` arm was unreachable — a trap for whoever writes the next fallback spec, since the obvious reading is that the `log?.` optional call is what's uncovered. Two specs throw codeless errors to cover it, then cli-app joins both matrices in the same commit so CI never observes a failing job. Verified on Node 14 (what these workflows currently run): 83/83, 100% statements/branches/functions/lines, exit 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude Code PR ReviewPR: #2402 • Head: fd29784 • Reviewers: stack-code-reviewer SummaryAdds Review Table
Findings
Dismissed after verification
Verdict: PASS — test-and-CI-only change, correctly targeted and green on CI; the two Low items are pre-existing and out of scope for this diff. |
The Test workflow's package list is hardcoded, so @percy/cli-pdf's suite was never running in CI at all -- the same gap #2402 closed for cli-app. Added to the matrix. CI runs test:coverage, which enforces the repo's 100% threshold, so two dead spots had to go first: - loadDocument used `mod.default ?? mod` for CJS interop, but the pdfjs legacy build always exposes `.default` (verified: `mod.default` is an object while `mod.getDocument` is undefined), leaving `?? mod` unreachable. It now reads `mod.default` directly. - pages.js skips empty segments in a string selection and nothing exercised that path. Added coverage for '1,,3' and '2,', plus the case where a selection resolves to no pages at all. Verified the rasterizer really does work on Node 14, the matrix version: pdfjs 2.16.105 plus the @napi-rs/canvas native binding render all three fixture pages with identical non-white pixel counts to Node 22. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
) * feat(cli-pdf): native PDF visual testing via POST /percy/pdf/snapshot Adds first-class PDF support to the CLI so an SDK can hand over PDF bytes and get one Percy snapshot per page, with synchronous comparison results. This is the replacement for the external percy-pdf solution, which wrapped the CLI from outside by serving a pdf.js viewer and driving Percy's renderer through the viewer's DOM with per-page `execute` scripts. New package @percy/cli-pdf is a leaf library: PDF bytes in, page rasters and their root DOM out. It holds no reference to @percy/core, which is what lets core list it as an optionalDependency without a cycle -- users who never snapshot a PDF do not install pdfjs-dist or the @napi-rs/canvas prebuilds. @percy/core gains the POST /percy/pdf/snapshot route and pdf-snapshot.js, which validates the request, decodes the base64 document, lazily imports @percy/cli-pdf, and pushes one snapshot per selected page through percy.upload() with `resources` as a function so rasterizing happens inside the queue task and inherits its concurrency. Each page carries resources and no `tag`, so createSnapshotsQueue routes it via client.sendSnapshot -- these are real web snapshots, not comparisons. This mirrors cli-upload's web-token path. The document travels as base64 in an ordinary JSON body and the sync response is always a JSON object (never a bare array) carrying a per-page array. Both are deliberate: every SDK, including the .NET wrapper's Dictionary-to-JSON helper and its JObject.Parse of the response, can call this with the HTTP client it already has, with no multipart or streaming code. Page snapshots are named `<name> | Page N`, matching percy-pdf exactly so teams migrating keep their approved baselines instead of orphaning them. Oversized pages are fitted rather than rejected: Legal (1224x2016 at scale 2) and A3 exceed Percy's 2000px cap and are exactly the documents this targets, so fitScale reduces the scale deterministically from the page's own dimensions and warns. pdfjs-dist is pinned to 4.x rather than 6.x, which requires Node >=22.13. @percy/sdk-utils exports postPdfSnapshot as the shared seam every SDK wraps. Note that sync mode is only reachable through this endpoint under `percy exec`: percy.syncMode() force-disables sync under skipUploads/deferUploads/delayUploads, which the `snapshot` and `upload` commands set. A `percy pdf <dir>` command could therefore never return comparison results, so none is added. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(cli-pdf): pin pdfjs-dist to 2.x so yarn install works on Node 14 CI installs on Node 14.21.3 and yarn enforces `engines` across the whole tree, so `yarn install` aborted with: error @percy/cli-pdf@1.32.8: The engine "node" is incompatible with this module. Expected version ">=18". Got "14.21.3" Dropping cli-pdf's own `engines` field is not sufficient: pdfjs-dist declares `node: ">=18"` from 3.x onward, so yarn would fail on the dependency instead. pdfjs-dist 2.16.105 is the last line that declares no engines constraint, and @napi-rs/canvas is already `>= 10`, so 2.x is what keeps the repo installable on its current Node floor. Rasterization output is equivalent -- verified end to end: unchanged document gives zero diffs on every page, and a document changed on page 2 only reports a diff on page 2 while pages 1 and 3 stay at zero. The 2.x legacy build is CommonJS rather than ESM, so the import moves to pdfjs-dist/legacy/build/pdf.js with `mod.default ?? mod` interop. cli-pdf's engines now matches its sibling packages at >=14. Also removes source comments across the PDF changes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(cli-pdf): don't bind createRequire to the name `require` packages/cli-command/test/noRequireBinding.test.js guards every packages/*/src file against `const require = createRequire(...)`: the name collides with Babel's transforms and crashes the packaged pkg binary with "_require is not a function". rasterize.js needed it to resolve pdfjs-dist's on-disk standard_fonts and cmaps directories, so the binding is renamed to cjsRequire as the guard suggests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(cli-pdf): add the package to CI and close two coverage gaps The Test workflow's package list is hardcoded, so @percy/cli-pdf's suite was never running in CI at all -- the same gap #2402 closed for cli-app. Added to the matrix. CI runs test:coverage, which enforces the repo's 100% threshold, so two dead spots had to go first: - loadDocument used `mod.default ?? mod` for CJS interop, but the pdfjs legacy build always exposes `.default` (verified: `mod.default` is an object while `mod.getDocument` is undefined), leaving `?? mod` unreachable. It now reads `mod.default` directly. - pages.js skips empty segments in a string selection and nothing exercised that path. Added coverage for '1,,3' and '2,', plus the case where a selection resolves to no pages at all. Verified the rasterizer really does work on Node 14, the matrix version: pdfjs 2.16.105 plus the @napi-rs/canvas native binding render all three fixture pages with identical non-white pixel counts to Node 22. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(cli-pdf): rasterize in the discovery browser, drop @napi-rs/canvas The discovery Chromium is already launched eagerly by percy.start() and sits completely idle for the whole of a PDF run (verified: every discovery queueInfo line reports total 0). Shipping @napi-rs/canvas alongside it meant paying 25MB of platform-specific native prebuilds, plus a native-binary dependency in a widely distributed CLI, to duplicate a renderer already present and running. pdf.js now runs inside a browser page instead of in the Node process: - @percy/cli-pdf drops @napi-rs/canvas entirely and becomes pure helpers plus pdf.js assets: page selection, the page DOM, pdfjs-dist asset paths, and the functions that execute in the page context. It remains an optionalDependency so nobody pays for pdfjs-dist's 34MB unless they snapshot a PDF. - @percy/core gains pdf-rasterize.js, which owns the browser work: a throwaway loopback origin (Server.serve) exposing pdf.js, its worker, standard_fonts, cmaps and the document itself, then a page that injects pdf.js and renders each selected page to a canvas, returning a PNG data URL per page. Serving the assets over a real origin is what makes standard fonts work: pdf.js fetches standardFontDataUrl/cMapUrl over HTTP, and base-14 fonts such as Helvetica are not embedded in most documents. isEvalSupported stays false -- PDFs are untrusted input. The rasterizer now calls percy.browser.launch() explicitly. It is idempotent, and this makes PDF snapshots work under skipDiscovery, where the eager launch does not happen. Two side effects worth noting. Rendering is now the CLI's pinned Chromium rather than a separately versioned Skia, so page rasters are as reproducible as the rest of Percy's pipeline instead of tracking a native dependency's version. And existing PDF baselines must be regenerated, since the rasterizer changed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(cli-pdf): cover the page-context scripts to meet the 100% threshold CI reported browser-scripts.js at 23.81% statements: openDocument, measurePages, renderPage and destroyDocument are serialized and executed in the browser page, so nothing in the Node suite ever ran them. pages.js:59 also had one uncovered branch, the singular form of the out-of-range message. Rather than mark the page scripts ignored, the suite now stands up fake `window` and `document` globals and invokes them directly. That covers the code and asserts behaviour that was genuinely untested: - the exact URLs pdf.js is handed (worker, document, standard_fonts, cmaps) and that isEvalSupported stays false - the window.pdfjsLib fallback, and the error when pdf.js never initialised - fractional viewports rounding up - the white canvas pre-fill, without which transparent PDF regions rasterize to alpha-0 black and diff against anything - page handles being released even when rendering rejects Adds a 1-page-document case so both arms of the pluralisation in the out-of-range message are exercised. 44 specs, all passing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(core): cover the PDF error paths to meet the 100% threshold All 1278 core specs passed, but the job failed on coverage: pdf-rasterize.js | 95.24 | 70.00 | 100 | 95.24 | 36,64 pdf-snapshot.js | 95.89 | 95.12 | 100 | 95.71 | 34,52,118 Every gap was an error or warning path. Now covered: the invalid-scale guard (each of its three arms), the fitScale warning via a Legal-size page, the too-short-base64 and non-object-pdf branches of decodePdf, a blank name, a non-object request body, an empty body, a browser failure surfacing as a rasterization error, and a page returning a failure status. Two small production changes fell out of writing them: - loadPdfModule takes an injectable loader, defaulting to the real dynamic import, so the 501 "package is not installed" path is reachable from a test instead of only when the optional dependency is genuinely absent. - The body guard now also rejects Buffers. api.js leaves req.body as raw bytes when JSON.parse fails, and `typeof Buffer === 'object'`, so a malformed body slipped past the check and produced a confusing "Missing required `name`" rather than "Expected a JSON object body". Found by the test. rasterizePdf's `options = {}` default is dropped: its single caller always passes options, so the default arm was unreachable branch weight. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(core): share the image-snapshot wrapper and take the extraction path Two problems, found by comparing against example-non-rendering-project. 1. buildPageHtml duplicated cli-upload's wrapper DOM. That shape is a contract with percy-api, not cosmetics: extract_and_process_upload_snapshot recovers the image by matching /<img\s+src="([^"]+)"\s+width="(\d+)px"\s+height="(\d+)px"/ against the root resource. A mismatch is not an error -- extraction raises, percy-api rescues, and the snapshot silently falls back to being rendered. Two divergent copies of that was a latent bug, and mine had already drifted (a trailing alt="" plus extra CSS; harmless only because the regex is unanchored). The wrapper now lives once, in core's utils as buildImageSnapshotHtml / createImageSnapshotResources. cli-upload's getImageResources delegates to it, and the PDF path uses it, so buildPageHtml is gone. core cannot import cli-upload (cli-upload -> cli-command -> core would cycle), but cli-upload already reaches core's utils via @percy/cli-command/utils, so this needs no new dependency either way. image-snapshot-resources.test.js pins percy-api's regex verbatim, so drift is caught in CI rather than degrading silently in production. 2. PDF pages were not taking the extraction path at all. Comparison#upload_snapshot? gates on `user_agent&.include?('@percy/cli-upload')` plus a root resource URL under http://local/. The PDF endpoint only ever forwarded the SDK's own clientInfo, so every page was fully re-rendered by the renderer fleet despite the CLI already having produced the exact PNG. Measured per page, same document: 19s/11s/9s rendered versus 1s/1s/1s extracted. The endpoint now also tags the build with @percy/cli-pdf and @percy/cli-upload. The percy-api check is a substring match, so naming cli-pdf alongside keeps the User-Agent honest about which code ran instead of impersonating the upload command. Note for operators: upload_extraction_allowed? only short-circuits on a project's default base branch. Elsewhere it mirrors the base comparison, so existing baselines need regenerating before comparison builds will extract. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(regression): compare rendered PDF pages byte-for-byte Adds Track P to the regression suite: rasterizes the PDFs in test/regression/assets/pdfs/ through @percy/core's rasterizePdf — the same path POST /percy/pdf/snapshot takes, pdf.js rendering each page in the discovery browser — and asserts every produced PNG is byte-identical to a committed golden. Byte equality catches anything that changes what reaches Percy: scale selection, canvas size, pixel output, PNG encoding. Token-free and build-free, so it runs on every PR. Goldens are platform-scoped under expected/<platform>-<arch>/ because PNG bytes are only reproducible for one platform and Chromium build — Percy pins a different Chromium snapshot per platform, and glyph rasterization goes through CoreText on macOS versus FreeType on Linux. Each set carries a manifest recording the browser build it came from, and a failing run reports a browser mismatch so a Chromium bump is not mistaken for a regression. Pages with raster images are not byte-reproducible even on one machine: Chromium picks between two anti-aliasing paths for a clipped image edge from run to run. Measured on jack sparrow resume.pdf as 6 of 15 runs differing, always the same 270 pixels in the same 193x193 box around the circular photo crop, never more than 52 per channel, out of 2,005,644 (0.013%). Such pages declare a pixel budget in TOLERANCES set at ~2x the measured worst case; the byte comparison still runs first and only falls back to the budget when it fails, a dimension change is never tolerated, and everything not listed must match byte-for-byte. The CI step is a temporary bootstrap: linux-x64 goldens have to be produced by the Linux Chromium build, so it generates and uploads them as an artifact. Once those are committed the step collapses to a plain compare run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(regression): assert PDF page bytes against linux-x64 goldens in CI Adds the goldens the previous commit's bootstrap step generated on the CI runner and collapses that step into a plain compare run, so the regression job now asserts on PDF page bytes instead of recording them. The macOS and Linux renders confirm why the goldens have to be platform scoped: page dimensions match exactly, but 9.7% of pixels differ on single-page-sample and 12.6% on multipage-sample-pdf page 2, at a max channel delta of 255 across the whole text area. Glyph rasterization is delegated to the platform font backend — CoreText on macOS, FreeType on Linux — so the glyph bitmaps themselves differ rather than just the PNG encoding. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(core): address PR review on the PDF snapshot path Resolves the CodeRabbit review on #2418. config: `pages` and `excludePages` spelled the selection grammar out as a regex that disagreed with parseSelection() in @percy/cli-pdf — the pattern rejected "1,,3" and "2,", which the parser accepts by skipping empty parts. The shape is now defined once as `pageSelection` and its pattern constrains only the character set, leaving the grammar and every semantic rule to the parser that actually reads the value. Note this drift was latent rather than user-visible: shouldHideError() in @percy/config suppresses every `oneOf` error unless the schema carries a custom `error`, so neither pattern has ever produced a warning. pdf-snapshot: a buffer shorter than the %PDF- magic reported "not valid base64-encoded data", which misdescribes input that decoded fine and was merely too short. It now says so, matching what the test was already named. pdf-snapshot tests: save and restore jasmine.DEFAULT_TIMEOUT_INTERVAL around the suite so the 240s bump does not leak into suites that run after it, and assert the /scale schema warning separately from the rasterizer's 400 — the existing assertion would still pass if schema validation were dropped. regression: the PDF byte track wrote each golden inside the render loop, so a run that failed its own sanity checks still overwrote the baseline and exited 0 in update mode. Writes are buffered and flushed only after the failure check, so a bad render leaves the committed set untouched. semgrep: the track's path.join() calls are flagged as path traversal. They join process.platform/arch, a slug already reduced to [a-z0-9-], and filenames read from the committed fixture directory — no external input reaches them — so both files are suppressed at the file level with that rationale, as the existing entries are. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(core): address PDF review findings on #2418 Four fixes from review, plus one change the third one forces. escapeAttr: the wrapper's `img src` went through escapeHtml, which rewrites `'` to '. encodeURIComponent leaves `'` literal, so for a document named "Jack's Resume" the src diverged from the registered resource URL, percy-api's extractor found no match, and the page silently fell back to being rendered at ~10x the cost with nothing surfaced. This also regressed @percy/cli-upload, which interpolated the URL raw before the wrapper was shared. Attribute context is double-quoted, so only & and " need escaping; `'`, < and > must be left alone. escapeHtml still guards the <title> text. Page#eval threw exceptionDetails.exception.description -- a bare string -- so every caller's error.message was undefined and an in-page failure surfaced as "Could not rasterize PDF: undefined". It now throws a real Error whose message is the description's first line, with the remote stack preserved verbatim as `stack` so nothing is lost. Limits: MAX_PAGES (250) enforced after exclusions so it counts what is actually selected; a 30s timeout on every in-page call, since Page.TIMEOUT only covers navigation and Runtime.callFunctionOn with awaitPromise waits forever; and the 50MB cap checked against the encoded length so Buffer.from never allocates for an oversized body. Rasterization failures answer 400 only when the caller can fix them -- the rasterizer tags those. A browser launch failure, an OOM or a CDP disconnect is a 500, not a report that the SDK sent a malformed request. The render timeout makes a leak reachable that was theoretical before: a timed-out page is exactly when close() rejects, which would strand the asset server still holding the customer's PDF. Both closes are now settled together. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(core): close the high findings from the second review Page#eval's Error broke the logging contract. @percy/logger renders a thrown Error as Error.prototype.toString and only falls back to `stack` at debug level, so a default `name` both doubled the prefix ("Error: Error: test error") and dropped the remote frames the user needs to debug their own execute script. snapshot.test.js "logs execute errors and does not snapshot" pinned exactly that text and went red in CI. Blanking `name` makes toString return the description verbatim, so the logged output is byte-identical to the string this replaced. Parsing the name off the first line does NOT work -- it strips the frames. withTimeout had no test at all, on a package with a 100% coverage gate, which means the fix for "no render timeout" shipped with no evidence it worked. It is now exported and covered: timeout, pass-through, early rejection, timer cleanup, and that a late rejection never surfaces as unhandled. `timer.unref?.()` became `timer.unref()` -- the optional call's false branch is unreachable in Node and would have failed the branch gate on its own. The asset server binds 127.0.0.1 instead of inheriting Server's "::" default. It serves the customer's PDF unauthenticated and its only client is the local discovery browser. An explicit host now beats PERCY_SERVER_HOST, so widening the API server cannot widen this one. Two evals were still un-raced despite the comment claiming every in-page call was: the pdf.js injection and destroyDocument. Either one reproduced the original hang exactly. Both now go through withTimeout. Cleanup failures are logged rather than silently discarded -- a socket that will not drain is still holding that PDF. And the `promise.catch(() => {})` guard is gone: Promise.race attaches handlers to every input, so the loser's late rejection was already handled and the comment described a hazard that cannot occur. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds
@percy/cli-appto CI. It was the only one of 18 packages missing from both thetest.ymlandwindows.ymlmatrices, so its 81 specs have never run in CI on any platform.Adding it as-is would have turned CI red — the package sits at 98.44% branch coverage against the repo's 100% threshold:
The uncovered branch is not the one it looks like
Both lines contain a
log?.optional call, which is the obvious suspect. It isn't that. The gap iserr.code || err.message, interpolated into the warning atmaestro-inject.js:157and the debug line at:273. Every existing spec throws an error carrying a code (EACCES,EROFS,EEXIST,ENOENT), so theerr.messagearm is unreachable.Worth stating explicitly because it is a trap for the next fallback spec: adding another coded-error case moves no coverage at all.
Two specs throw codeless errors to cover it, and
cli-appjoins both matrices in the same commit so CI never observes a failing job.Verification
Run the way these workflows currently do — Node 14:
Found via, but deliberately separate from, the Node 20 work
Surfaced while auditing package coverage for #2386. It is unrelated to that migration — the gap is version-agnostic and would fail identically on any Node — so it is kept off that branch rather than widening a release-bound PR.
One note for reviewers of #2386: running this same suite on master + Node 20 reports
All files | 0 | 0 | 0 | 0and still exits 0. That is the vacuous-coverage failure mode #2386 fixes, reproduced here incidentally. It is why the verification above was run on Node 14.🤖 Generated with Claude Code