@@ -59,12 +59,12 @@ The Bazel build produces two archives: `:typescript` (`javascript.zip`) contains
5959the compiled ESM wrapper and JavaScript API; ` :native ` (` native.zip ` ) contains the
6060matching native packages for the target CodeQL distribution:
6161
62- | Distribution | Native packages |
63- | --- | --- |
64- | ` linux64 ` | ` @typescript/typescript-linux-x64 ` |
65- | ` linux-arm64 ` | ` @typescript/typescript-linux-arm64 ` |
66- | ` osx64 ` | ` @typescript/typescript-darwin-arm64 ` and ` @typescript/typescript-darwin-x64 ` |
67- | ` win64 ` | ` @typescript/typescript-win32-x64 ` |
62+ | Distribution | Native packages |
63+ | ------------- | -------------------------------------------------------------------------- --- |
64+ | ` linux64 ` | ` @typescript/typescript-linux-x64 ` |
65+ | ` linux-arm64 ` | ` @typescript/typescript-linux-arm64 ` |
66+ | ` osx64 ` | ` @typescript/typescript-darwin-arm64 ` and ` @typescript/typescript-darwin-x64 ` |
67+ | ` win64 ` | ` @typescript/typescript-win32-x64 ` |
6868
6969The build installs the selected packages explicitly with ` --force --no-save ` ,
7070using the pinned versions and integrities in the lockfile. Merely passing
@@ -97,12 +97,13 @@ Linux arm64, and Windows x64, including Windows 2022.
9797Native lifecycle and memory
9898---------------------------
9999
100- Each file owns a fresh native API/process. AST projection and JSON serialization
101- finish before the program is disposed and the API is closed, including on
102- failure. No native process or compiler cache is retained between files, during
103- stdout backpressure, or across reset/quit. This deliberately favors bounded
104- resource lifetime over reusing a compiler process; the measured cost is described
105- below.
100+ A single native API/process is created lazily and reused across files and
101+ prepare-files batches. Each file still owns a separate program: AST projection
102+ and JSON serialization finish before the program is disposed, including on
103+ failure. Program disposal releases its snapshot and reference-counted source-file
104+ caches without restarting the compiler. Reset closes the API, and the next parse
105+ creates a new one. Quit, EOF, failure, and completion of single-file mode also
106+ close the API.
106107
107108` SEMMLE_TYPESCRIPT_MEMORY_THRESHOLD ` remains a positive megabyte budget, defaulting
108109to 1000 when the Java extractor does not supply it. At native-process creation,
@@ -111,9 +112,9 @@ and passes the remaining bytes, with a minimum of one, as the child's
111112` GOMEMLIMIT ` . The parent's original ` GOMEMLIMIT ` is restored after spawning.
112113This is a Go garbage-collector soft limit, not a hard combined RSS ceiling:
113114Node allocations during extraction, native non-Go allocations, and oversized
114- individual files can exceed the budget. Closing the native process after every
115- file prevents native caches and heap capacity from accumulating across files.
116- Node RSS exceeding the budget produces a warning on stderr.
115+ individual files can exceed the budget. The child's budget is fixed until the
116+ API is closed; later changes to Node RSS do not resize it. Node RSS exceeding
117+ the budget at native-process creation produces a warning on stderr.
117118
118119Native stderr is inherited separately from the JSON stdout protocol. Syntax
119120diagnostics remain successful AST responses; native, I/O, and protocol failures
@@ -139,7 +140,7 @@ spelling. Namespace classification and variable declaration kinds are derived
139140in Java without synthetic flags or properties.
140141
141142The tests cover AST projection, metadata, UTF-16 source positions, syntax
142- diagnostics, TSX, repeated parses, native process disposal, the child memory
143+ diagnostics, TSX, repeated parses, native process reuse and disposal, the child memory
143144budget, reset/quit/EOF, cancelled prefetch, runtime overrides, and fatal native
144145failures. Run them with Node.js 16.20.0 to verify the minimum supported runtime.
145146
@@ -193,19 +194,56 @@ codeql test run -j4 --search-path=javascript/extractor-pack:. \
193194Performance comparison
194195----------------------
195196
196- On macOS arm64 with Node.js 18.15.0, a comparison against the pre-migration
197- wrapper from ` 1971c368e1d ` compiled with TypeScript 5.9.2 gave these medians over three
198- runs:
197+ The current shared-API implementation was compared with refreshed ` main ` at
198+ ` fb0ba8c8a8a ` on October 21, 2026. Both ran on macOS arm64 with Node.js 18.15.0;
199+ ` main ` was built and run with TypeScript 5.9.2, and this branch used the pinned
200+ 7.1 native compiler. These are medians of five fresh-process runs after one
201+ warm-up per variant and workload, alternating variant order:
199202
200- | Workload | 5.9.2 time | 7.1 time | 5.9.2 combined RSS | 7.1 combined RSS |
203+ | Workload | Main time | Branch time | Main combined RSS | Branch combined RSS |
201204| --- | --- | --- | --- | --- |
202- | 103 TypeScript fixtures (33,589 bytes) | 0.237 s | 1.349 s | 124.1 MiB | 113.9 MiB |
203- | Five parses of TypeScript's 588,128-byte declaration file | 0.991 s | 1.116 s | 269.3 MiB | 249.1 MiB |
205+ | 104 TypeScript fixtures (33,779 bytes) | 0.081 s | 0.120 s | 105.5 MiB | 110.4 MiB |
206+ | Five parses of TypeScript's 588,128-byte declaration file | 0.886 s | 0.992 s | 263.7 MiB | 247.2 MiB |
207+
208+ The branch takes 47.3% more time on the fixtures and 11.9% more on the declaration
209+ workload. Sampled combined RSS is 4.7% higher and 6.3% lower, respectively.
210+ Both variants parse identical inputs. Timing starts after the metadata handshake
211+ and includes prepare-files, AST extraction, serialization, and protocol responses,
212+ including the branch's first native-process startup. It excludes Java conversion
213+ and database import. RSS is the maximum sampled sum of Node and its native
214+ descendants, sampled at a nominal 5 ms interval; short-lived peaks may be missed.
215+ Current ` main ` includes parser cleanup absent from the older baseline below.
216+
217+ ### Historical and API-lifetime comparisons
218+
219+ The initial implementation created a native process per file. On macOS arm64
220+ with Node.js 18.15.0, a comparison against the pre-migration wrapper from
221+ ` 1971c368e1d ` compiled with TypeScript 5.9.2 gave these medians over three runs:
222+
223+ | Workload | 5.9.2 time | 7.1 time | 5.9.2 combined RSS | 7.1 combined RSS |
224+ | --------------------------------------------------------- | ---------- | -------- | ------------------ | ---------------- |
225+ | 103 TypeScript fixtures (33,589 bytes) | 0.237 s | 1.349 s | 124.1 MiB | 113.9 MiB |
226+ | Five parses of TypeScript's 588,128-byte declaration file | 0.991 s | 1.116 s | 269.3 MiB | 249.1 MiB |
204227
205228Timing covers prepare-files, extraction, serialization, and protocol responses
206229after the metadata handshake, not Java conversion or database import. Combined
207230RSS is the maximum sampled sum of Node and its native descendants, sampled every
2082315 ms; short-lived peaks may be missed. The per-file native process costs about
2092325.7x on tiny files and 1.13x on the declaration workload, while measured memory is
210- slightly lower. The bounded lifecycle is retained deliberately; this migration
211- does not claim a speedup.
233+ slightly lower.
234+
235+ A follow-up comparison on October 21, 2026, using the same platform and Node.js
236+ version, measured the per-file and shared-API lifetimes directly (medians of three
237+ runs):
238+
239+ | Workload | Per-file API time | Shared API time | Per-file combined RSS | Shared combined RSS |
240+ | --- | --- | --- | --- | --- |
241+ | 104 TypeScript fixtures (33,779 bytes) | 1.017 s | 0.118 s | 115.8 MiB | 109.4 MiB |
242+ | Five parses of the same declaration file | 1.044 s | 0.985 s | 250.2 MiB | 260.2 MiB |
243+
244+ AST responses were byte-identical between lifetimes. A longer shared-API run on
245+ Node.js 26.7.0 processed 50,000 distinct generated TypeScript files: native RSS
246+ was 27.9 MiB after 1,000 files and 30.4 MiB after 50,000, while combined RSS
247+ stabilized around 176 MiB. Another run completed 500 parses of the declaration
248+ file with peak native RSS of 56.5 MiB. These workloads showed no unbounded
249+ per-file retention, so the wrapper reuses the API without a fixed batch limit.
0 commit comments