fix(scanner): bound memory and accelerate full scans

[alxxjohn]

Summary

This fixes repository-wide full scans that could remain CPU-active for more than ten minutes while growing to multiple GiB of memory on mixed Go and TypeScript/AWS CDK repositories.

The change makes full-scan work bounded, removes repeated TypeScript regex compilation, adds progress and heap diagnostics, and introduces repeatable cold/warm full-scan and clone-index benchmarks.

Root-cause evidence

A repeated cold full-scan profile showed approximately 39 percent of allocation space in regexp compilation, primarily from recompiling the same child_process and vm TypeScript binding patterns for every file. Repository-wide corpus and clone structures also retained work without aggregate byte, token, window, or entry limits.

Changes

Bounded repository corpus

• Keep dependency and generated trees named node_modules and cdk.out hard-pruned at any depth.
• Keep vendor pruned by default, with bounded source analysis available through an explicit scan_vendored_source opt-in.
• Skip individual files larger than 32 MiB.
• Cap retained repository paths at 100,000 files.
• Cap retained file contents at 128 MiB and 50,000 entries.
• Cap retained Go AST source admission at 64 MiB and 25,000 entries.
• Limit concurrent uncached file reads and Go parses to two.
• Singleflight concurrent reads of the same overflow file.
• Reject further uncached work after a corpus budget is exhausted instead of repeatedly allocating it across concurrent sections.
• Emit deduplicated scan.corpus-budget informational diagnostics when analysis is truncated.

Bounded and faster clone analysis

• Cap clone source at 64 MiB, normalized tokens at 2,000,000, and indexed windows at 1,000,000.
• Cap clone pair comparisons and emitted candidates.
• Replace per-window full hashing with an O(1) rolling hash while retaining token-by-token collision verification.
• Bucket merge work by document pair.
• Emit quality.duplicate-code-budget diagnostics when the bounded analysis is truncated.

TypeScript security scan optimization

• Precompile the closed ten-pattern child_process and vm scanner pattern set once.
• Short-circuit module-binding analysis when a file does not reference the module.
• Keep source-derived module names uncached and regexp-quoted, preventing attacker-controlled global cache growth.

Operator diagnostics

• Stream scan start and completed-section progress to stderr without contaminating JSON, SARIF, GitHub, or CycloneDX stdout.
• Emit a 30-second heartbeat with elapsed time, current Go heap, and the active GOMEMLIMIT.
• Add -memprofile for rolling atomic heap profiles at startup, heartbeat intervals, and completion.
• Preserve report output and combine findings/profile errors when final profile writing fails.

Configurable vendored-source analysis

• Add top-level scan_vendored_source: true for repositories that commit or patch vendored code.
• Preserve the existing file-count, file-size, retained-byte, AST, and concurrency limits when vendored scanning is enabled.
• Preserve repository-relative exclusion semantics for normal targets and folder-scoped scans.
• Prevent -folder and configured target roots from bypassing vendor, node_modules, or cdk.out policy.
• Keep configured exclude patterns authoritative when vendored-source scanning is enabled.
• Include the option in JSON/YAML configuration round-trip coverage and scan-cache transition tests.

scan_vendored_source: true

Benchmarks

Synthetic mixed repository benchmark on Apple M4 arm64 with 64 Go files, 64 TypeScript files, and generated node_modules, vendor, and cdk.out trees:

Cold full scan	Baseline	This change	Difference
Time	27.00 ms/op	21.07 ms/op	about 22 percent faster
Allocated bytes	48.30 MB/op	29.65 MB/op	about 39 percent lower
Allocations	264,578/op	138,289/op	about 48 percent lower

Current warm-cache result: 14.09 ms/op, 15.10 MB/op, and 55,199 allocations/op.

Follow-up full-scan matrix after adding configurable vendored-source analysis (Apple M4 arm64, median of three runs):

Mode	Time	Allocated bytes	Allocations
Default cold	28.70 ms/op	29.42 MB/op	138,364/op
Default warm cache	30.34 ms/op	17.19 MB/op	61,701/op
Cold with 257 vendored Go files enabled	76.89 ms/op	60.59 MB/op	235,485/op

The default path stays within the bounded post-fix allocation profile. The opt-in case performs and retains more analysis in proportion to the explicitly admitted vendored source while remaining subject to the same hard corpus limits.

High-entropy clone window benchmark:

• 10,000 tokens: 1.14 ms/op, 1.47 MB/op, 9,994 allocations/op
• 100,000 tokens: 6.45 ms/op, 12.09 MB/op, 100,443 allocations/op

Edge and regression coverage

Added coverage for:

• Nested dependency and CDK output pruning
• Default and opt-in vendored-source traversal
• Configured and absolute dependency-tree target roots
• Folder-scoped scans that attempt to narrow directly into excluded trees
• Target-relative default/custom exclude preservation
• Scan-cache transitions from default to vendored opt-in and back
• Oversized individual files
• Aggregate retained-read and Go AST byte budgets
• Zero-byte file floods and independent entry caps
• Bounded file listings with surfaced diagnostics
• Concurrent over-budget Go parses
• Concurrent overflow read singleflight behavior
• Clone source, token, window, pair-comparison, and candidate limits
• Runtime-derived TypeScript module names remaining uncached and quoted
• Progress output separation and heap-profile success/failure paths
• Cold and warm full-scan performance
• High-entropy clone index scaling

Verification

• go test ./... passed
• Race-enabled tests passed for CLI, corpus support, quality, security, and public integration packages
• go vet ./... passed
• go build ./cmd/codeguard passed
• Full self-scan passed under GOMEMLIMIT=1GiB with a heap profile
• Synthetic full-scan and clone benchmarks passed
• Independent review found no remaining Critical or Important issues

Operational note

If a repository exceeds a hard corpus or clone budget, CodeGuard completes with an informational diagnostic indicating degraded analysis rather than risking unbounded memory growth. Repository-specific generated paths should still be added to exclude where appropriate.