fix: JavaScript extensions scale to a large corpus - #1319
alandefreitas merged 9 commits into
Conversation
✨ Highlights
🧾 Changes by Scope
🔝 Top Files
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #1319 +/- ##
========================================
Coverage 83.12% 83.12%
========================================
Files 35 35
Lines 3662 3662
Branches 844 844
========================================
Hits 3044 3044
Misses 410 410
Partials 208 208
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
An automated preview of the documentation is available at https://1319.mrdocs.prtest2.cppalliance.org/index.html If more commits are pushed to the pull request, the docs will rebuild at the same URL. 2026-09-24 03:10:00 UTC |
`jerry_init` asks the port for a block to hold the context and the heap, and takes the *size* of that block from what the call returns. We returned a pointer, instead, so the engine read an address as a length and believed it had a much larger heap than actually available. Return the size, instead, as the port function is asked to. The heap is now the size it was allocated, and running out of it ends the way the engine means it to, rather than by corrupting memory. Refs cppalliance#1294.
A script that required more memory than the interpreter had available caused the process to end with a bare failure code, which is what the engine's default behavior on a fatal condition is. Say what happened instead, and where the limit is, so the author of the script can see what they ran into. Refs cppalliance#1294.
This fixes a bug for which an extension on a large project ran out of memory before it reached its first line of output. Reading `ctx.corpus.symbols` was enough to trigger the bug. The root cause was that an array was converted to JavaScript element by element, up front. Each element that is an object cost an interpreter object of its own, and the interpreter's heap is a fixed 512 KiB that a build cannot simply enlarge, since 16-bit compressed pointers cap it there. About two thousand symbols filled it. An array now reaches JavaScript through a proxy, as an object already did, and an element is converted when a script reads it, so what a script holds has to fit rather than the whole corpus. The proxy wraps a real array, so `Array.isArray` still answers `true` and the methods on `Array.prototype` still work, and a script's own writes go to the wrapped array, where they stay out of the DOM as copying used to keep them. Fixes cppalliance#1294.
JerryScript 3.0.0's parser reads an uninitialized field of its context while parsing a `let` declaration, which MemorySanitizer reports from any test that uses `let`. That is upstream's bug, not ours, so the JerryScript build now gets a sanitizer ignorelist covering jerry-core and jerry-port when compiled with -fsanitize=memory. The code stays instrumented and uninitialized values are still tracked into mrdocs; only reports originating inside those sources are silenced.
9ebb3c4 to
5a74d0c
Compare
|
The general direction is correct, but it failed CI and was missing components for a more stable fix. I've pushed extra commits to consider that. |
JerryScript addresses its arena through compressed pointers, and with the 16-bit variant the heap could not exceed 512 KB, so an extension that kept a few thousand symbol objects alive ran out of memory even with lazy arrays. JerryScript is now built with JERRY_CPOINTER_32_BIT=ON. Each context reserves just under 4 GB. On POSIX that is mmap(MAP_NORESERVE), so pages cost memory only when written. Windows has no demand-zero pages, so the block is only reserved and a vectored exception handler commits it 1 MB at a time when the engine first touches a page, which gives the same lazy behavior without an upstream change. A failed reservation is halved down to 512 KB. The out-of-memory message names the heap size and leaves through _Exit so static destructors do not call back into the dead engine. The bootstrap patch excludes jerry-port-process.c along with the context port file so the jerry_port_fatal override is portable to MSVC. Lua needs no change: it allocates from the system heap and reports out-of-memory as a catchable error.
Every DOM proxy used to build its own handler object and its own set of five or six trap functions, so an element read from ctx.corpus.symbols cost about eight engine objects and 330 bytes, and creating it dominated loops over the corpus. The traps only need the wrapped value, which now lives as a native pointer on the proxy target (their first argument), so a context builds two handlers once, one for objects and one for arrays, and every proxy is just a target plus the Proxy object. On the MrDocs corpus (6414 symbols) a count-by-kind loop went from 154 ms to 65 ms, collecting function names from 302 ms to 119 ms, nested property reads from 630 ms to 186 ms, and 102,624 live symbol proxies add 5.4 MB instead of 33.6 MB.
With JERRY_EXTERNAL_CONTEXT every read the engine makes of its own state goes through jerry_port_context_get(), several times per bytecode instruction. Our port answered that with pthread_once plus pthread_getspecific, two library calls per engine field access, which a profile showed as 98% of the samples of any JavaScript loop. The pointer is now a constant-initialized thread_local, one load. On top of that, the JerryScript build force-includes mrdocs-context.h (a compile option set from the patched CMakeLists, no upstream file edited), which defines jerry_port_context_get() as a macro reading that variable, so the engine makes no call at all: the built library references the variable and not the function. An engine built without the header still works through the call, since the variable and the function always agree. On the MrDocs corpus a count-by-kind loop went from 62 ms to 29 ms and nested property reads from 182 ms to 116 ms.
A page under Extensions that compares Lua and JavaScript as script engines on the technical side: measured speed on six equivalent transforms over the MrDocs corpus (Lua is 2 to 20 times faster), how symbols reach a script and what a live one costs, how each engine uses memory and what happens on out of memory, and the state of the standard libraries in each engine as probed in the built binary. Ends with guidance on which language to pick. The corpus transforms and Handlebars extensions pages name both languages and link to it. tests/benchmarks/ keeps the benchmarks the page quotes so they can be re-measured: the six transforms in both languages over MrDocs's own corpus, a runner that averages 30 runs and prints the table (Markdown or AsciiDoc), and the mrdocs-benchmark-script-engines custom target. Not CTest tests, since timing is machine-dependent.
5a74d0c to
c7a0050
Compare
|
Thanks. A few comments: Do the benchmarks really support the conclusions drawn in the docs?
Is the MSan ignorelist too broad?
Exit paths
Commit-on-demand
Stale comments
|
A JavaScript extension could not read the corpus of a large project. Reading
ctx.corpus.symbolswas enough to cause the process to die with an access violation and no diagnostic. Two bugs were behind it, one in how the interpreter is set up and one in how an array reaches it, and a third change makes the limit the script ran into visible.Changes
jerry_port_context_allocnow returns the allocated size, not a pointer to the allocated block.jerry_initasks the port for a block to hold the context and the heap, and takes the size of that block from what the call returns. We returned a pointer, instead, so the engine read an address as a length, believed it had a much larger heap than it had, and wrote past the 512 KiB it really had.jerry_port_fatalnow says what happened and where the limit is, so the author of the script can see what they ran into.Testing
The new unit test in tests/unit/Engines/JavaScript.cpp covers the lazy conversion. On develop, it crashes with an access violation; with the first two commits and not the third, it ends with the out-of-memory message; with all three it passes. The existing unit tests that pin the snapshot semantics of a converted array, including that a script's writes do not reach the DOM, pass unchanged.
The out-of-memory message itself ends the process, so no test asserts it in-process; it was checked by running the test above without the third commit.
Our own schema generator, docs/mrdocs/extensions/schema.js, now runs within the enforced heap for the first time. Run as the
schema-checktest runs it, it still produces artifacts byte-identical to the committed ones.Documentation
Not needed. The documentation never described the heap or the workaround, and the obvious loop over
ctx.corpus.symbolsthat the example extensions use now works on a large corpus. The out-of-memory message states the limit where a script meets it.Fixes #1294.