Make configured memory-limit exhaustion optionally terminal

[yuhaouno]

Summary

Fixes #1773

• JS_SetMemoryLimitTermination() opts in (off by default) so a rejection from the engine's own JS_SetMemoryLimit check is a runtime-scoped terminal condition.
• The condition is recorded at that check, not from the exception object. It stays set when the out-of-memory Error cannot be allocated and the pending exception is null.
• JavaScript catch does not resume, and JS_ExecutePendingJob() returns failure instead of running a rejection continuation. JS_GetTerminationStatus() reports JS_TERMINATION_MEMORY_LIMIT without parsing exception text.
• Arbitrary allocator NULL, ordinary exceptions, and the previous catchable memory-limit behavior are unchanged. After termination the runtime is not reusable and should be freed.

Test plan

• api-test in Debug and ASan+UBSan, including catchable exhaustion, terminal sync (including 1-byte headroom / null), promise reaction, async resumption, promise constructor, thenable job, ordinary Error/RangeError/TypeError/syntax/rejection, custom-allocator NULL, and a fresh runtime
• lre-test in Debug and ASan+UBSan
• JavaScript suite (run-test262 -c tests.conf): 0/117 errors, 9 excluded, in Debug and ASan+UBSan
• ctest, cxxtest, and jscheck (Apple Clang needs -Wno-unused-command-line-argument for jscheck)
• JS_ABORT_ON_LEAKS via api-test; macOS leaks --atExit reported 0 leaks
• Full Test262 not run; the default path is unchanged and the new behavior is opt-in

Made with Cursor

[saghul]

What's the use case here? Can't the host app check the error and decide whether it allows the app to continue or stop it?

[josefguenther]

Use case: a host running untrusted scripts under a memory limit. The script can catch the out-of-memory error itself (try { … } catch {} around the allocation) and keep going, so the host never gets to decide. Making it terminal leaves that decision with the host, the same way an interrupt can't be caught.

[saghul]

Im trying to come up with a case in which a he catch would be the right thing to do...

@bnoordhuis You have more experience here. What do other engines do? Perhaps memory errors should always be uncatchable?

[bnoordhuis]

Other JS engines just terminate the process. :-)

IMO, the only change we need to make is to turn OOM errors uncatchable by JS. Easy:

diff --git a/quickjs.c b/quickjs.c
index bc72af2..8be4dd3 100644
--- a/quickjs.c
+++ b/quickjs.c
@@ -8563,6 +8563,7 @@ JSValue JS_ThrowOutOfMemory(JSContext *ctx)
     if (!rt->in_out_of_memory) {
         rt->in_out_of_memory = true;
         JS_ThrowInternalError(ctx, "out of memory");
+        JS_SetUncatchableError(ctx, ctx->rt->current_exception);
         rt->in_out_of_memory = false;
     }
     return JS_EXCEPTION;
@@ -49777,7 +49778,7 @@ static JSValue js_regexp_exec(JSContext *ctx, JSValueConst this_val,
                 JS_ThrowInterrupted(ctx);
                 break;
             case LRE_RET_MEMORY_ERROR:
-                JS_ThrowInternalError(ctx, "out of memory in regexp execution");
+                JS_ThrowOutOfMemory(ctx);
                 break;
             case LRE_RET_BYTECODE_ERROR:
                 JS_ThrowInternalError(ctx, "corrupted bytecode in regexp execution");

Agree/disagree?

[saghul]

Agree, that's what I was thinking indeed!

[bnoordhuis]

#1810 - and maybe there's a broader discussion to be had of making all InternalError exceptions uncatchable.