Discuss opt-in terminal semantics for configured memory limits

[yuhaouno]

An embedder can cap engine-accounted memory with JS_SetMemoryLimit, but cannot currently request that reaching this cap terminate evaluation with a host-identifiable reason. Is an opt-in terminal memory-limit contract something the project would support?

I tested unmodified master 2f0aa72 (fresh clone, macOS arm64). This is a design question before proposing a new API or changing existing catchable OOM behavior.

Reproduced behavior

The attached public-C-API-only reproducer initializes the context and compiles the source before arming a limit inside fail(). It then requests a 1 MiB ArrayBuffer with 64 KiB of runtime headroom. Its custom-allocator variant deterministically refuses requests >= 1 MiB once armed, leaving smaller allocations available. Every case uses an independent runtime.

Failure / path	Synchronous try/catch	Promise executor	Promise reaction	After await	Thenable call	then getter
JS_SetMemoryLimit refusal	caught; success	caught; success	caught; success	caught; success	caught; success	caught; success
Custom allocator refusal	caught; success	caught; success	caught; success	caught; success	caught; success	caught; success
Trusted native operation throws a preallocated, explicitly uncatchable Error after its allocator refuses	host exception	caught; success	host exception	host exception	caught; success	caught; success

“Caught; success” means the source's catch/rejection handler runs and the following success marker executes; the host receives no exception from evaluation or draining jobs. The native-operation row isolates propagation: the host knows the refusal reason from its own allocator state and throws a preallocated Error with JS_SetUncatchableError. It does not claim that current engine OOM is uncatchable. Where this error reaches the host, object identity is preserved and the catch/success markers remain unset.

With just one byte of headroom, the synchronous memory-limit case even catches null and continues: allocation of the OOM Error itself fails. Ordinary Error, RangeError and Promise rejection remain catchable, and a fresh independent runtime evaluates successfully after all cases.

Why an OOM flag alone is insufficient

• The runtime can distinguish its own configured-limit check from an allocator returning NULL at the allocation layer. Both currently return NULL without retaining a cause. Context allocation wrappers call JS_ThrowOutOfMemory; other runtime-only allocation callers handle NULL separately. JS_ComputeMemoryUsage is not a failed-allocation reason query.
• Custom allocator callbacks receive opaque and allocation arguments, not the initiating context or a typed failure channel. The embedder can record its own refusal reason in opaque; QuickJS cannot infer whether NULL means a policy refusal or actual system OOM. Storing a context pointer in opaque does not supply a supported contract for throwing from an allocator callback, and ordinary OOM handling can overwrite a pending exception.
• OOM handling allocates an Error. Error construction falls back to null if allocation fails. The uncatchable flag only applies to Error objects, so simply marking the resulting OOM value cannot cover that case.
• Fix uncatchable error inside a promise (#810) #811 already preserves uncatchable state in Promise reactions and async-function resumption. On this HEAD, Promise construction, thenable jobs, and the resolution getter failure path still convert it into a rejection. This is a representative propagation check, not an exhaustive audit of all asynchronous machinery.

Proposed scope for discussion

An opt-in contract would need to preserve an engine-authenticated reason for configured runtime-limit exhaustion even when no Error can be allocated, bypass source catch handlers, and propagate to the C host through evaluation/jobs without turning into recoverable Promise rejection. Ordinary JavaScript exceptions would retain their existing behavior. Actual system OOM and arbitrary custom-allocator refusal should not silently be reclassified as a configured-limit event.

Would a private runtime termination state plus a small host-facing query/opt-in be an appropriate direction, or is this deliberately outside the engine's contract? A reason query alone can help host diagnostics but does not stop JavaScript from catching the failure and continuing. Before implementation, the lifetime of such state, treatment of runtime-only allocation calls, and whether teardown is required need to be specified. No private allocator structures need to become public, and no promise of runtime reuse or cancellation of pending jobs is proposed.

I checked open issues/PRs and relevant history before writing this. Related but distinct: #1145 concerns per-context accounting; #1181 concerns context initialization under a low limit; #1516 discussed InternalError; #1745 discusses queued-job disposal and runtime reuse. Recent #1711 fixes WASI accounting rather than termination. This report neither repeats #811's fixed paths nor proposes changing all OOM exceptions.

Reproducer

Save the following as resource-repro.c outside the source tree. From the QuickJS-NG checkout:

cmake -B build -DCMAKE_BUILD_TYPE=Debug
cmake --build build -j 8
cc -std=c11 -g -I. /path/to/resource-repro.c build/libqjs.a -lm -lpthread -o resource-repro
./resource-repro

The assertions characterize the current behavior, including the undesired successes; they are not assertions that hard termination is implemented. The allocator tracks outstanding backing allocations and checks that teardown frees all of them. Failure classification never uses error messages.

Complete C reproducer (memory cap, allocator refusal, propagation, and controls)

/* Public-API-only characterization of QuickJS-NG master 2f0aa72a. */
#include <assert.h>
#include <stddef.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "quickjs.h"

typedef union Block { max_align_t alignment; size_t size; } Block;
typedef struct State {
    int mode, armed, refused, caught, continued, success, null_caught;
    size_t live;
    JSValue terminal;
} State;

static void *alloc(void *opaque, size_t size) {
    State *s = opaque;
    Block *b;
    if (s->armed && size >= 1024 * 1024) { s->refused++; return NULL; }
    b = malloc(sizeof(*b) + size);
    if (!b) return NULL;
    b->size = size;
    s->live++;
    return b + 1;
}
static void release(void *opaque, void *ptr) {
    State *s = opaque;
    if (ptr) { s->live--; free((Block *)ptr - 1); }
}
static void *resize(void *opaque, void *ptr, size_t size) {
    void *p;
    if (!ptr) return alloc(opaque, size);
    if (!size) { release(opaque, ptr); return NULL; }
    p = alloc(opaque, size);
    if (!p) return NULL;
    size_t old = ((Block *)ptr - 1)->size;
    memcpy(p, ptr, old < size ? old : size);
    release(opaque, ptr);
    return p;
}
static void *zero_alloc(void *opaque, size_t count, size_t size) {
    if (size && count > SIZE_MAX / size) return NULL;
    void *p = alloc(opaque, count * size);
    if (p) memset(p, 0, count * size);
    return p;
}
static size_t usable(const void *ptr) { return ptr ? ((const Block *)ptr - 1)->size : 0; }
static const JSMallocFunctions mf = { zero_alloc, alloc, release, resize, usable };

static JSValue arm(JSContext *ctx, JSValueConst this_val, int argc, JSValueConst *argv) {
    State *s = JS_GetContextOpaque(ctx);
    if (s->mode == 0 || s->mode == 3) {
        JSMemoryUsage usage;
        JS_ComputeMemoryUsage(JS_GetRuntime(ctx), &usage);
        JS_SetMemoryLimit(JS_GetRuntime(ctx), usage.malloc_size + (s->mode == 0 ? 65536 : 1));
    } else {
        s->armed = 1;
    }
    return JS_UNDEFINED;
}
static JSValue terminal_fail(JSContext *ctx, JSValueConst this_val, int argc, JSValueConst *argv) {
    State *s = JS_GetContextOpaque(ctx);
    s->armed = 1;
    /* A trusted native operation that knows why its allocator refused. */
    void *p = js_malloc(ctx, 1024 * 1024);
    assert(p == NULL && s->refused == 1);
    JS_FreeValue(ctx, JS_GetException(ctx));
    return JS_Throw(ctx, JS_DupValue(ctx, s->terminal));
}
static JSValue mark(JSContext *ctx, JSValueConst this_val, int argc, JSValueConst *argv) {
    State *s = JS_GetContextOpaque(ctx);
    int32_t kind;
    assert(argc >= 1 && JS_ToInt32(ctx, &kind, argv[0]) == 0);
    if (kind == 0) { s->caught++; if (argc > 1 && JS_IsNull(argv[1])) s->null_caught++; }
    if (kind == 1) s->continued++;
    if (kind == 2) s->success++;
    return JS_UNDEFINED;
}
static void install(JSContext *ctx, const char *name, JSCFunction *fn) {
    JSValue g = JS_GetGlobalObject(ctx);
    assert(JS_SetPropertyStr(ctx, g, name, JS_NewCFunction(ctx, fn, name, 0)) >= 0);
    JS_FreeValue(ctx, g);
}
static const char *paths[] = {
    "try { fail(); } catch(e) { mark(0,e); } mark(1); mark(2);",
    "new Promise(() => { fail(); }).catch(e => { mark(0,e); }).then(() => mark(2)); mark(1);",
    "Promise.resolve().then(() => { fail(); mark(1); }).catch(e => { mark(0,e); }).then(() => mark(2));",
    "(async () => { await 0; try { fail(); } catch(e) { mark(0,e); } mark(1); })().then(() => mark(2));",
    "Promise.resolve({then() { fail(); }}).catch(e => { mark(0,e); }).then(() => mark(2)); mark(1);",
    "Promise.resolve({get then() { fail(); }}).catch(e => { mark(0,e); }).then(() => mark(2)); mark(1);"
};
static const char *names[] = {"sync", "executor", "reaction", "await", "thenable", "then-getter"};
static void run(int mode, int path) {
    State s = {.mode = mode};
    JSRuntime *rt = (mode == 0 || mode == 3) ? JS_NewRuntime() : JS_NewRuntime2(&mf, &s);
    assert(rt);
    JS_SetDumpFlags(rt, JS_ABORT_ON_LEAKS);
    JSContext *ctx = JS_NewContext(rt), *job_ctx = ctx;
    assert(ctx);
    JS_SetContextOpaque(ctx, &s);
    install(ctx, "arm", arm); install(ctx, "mark", mark);
    s.terminal = JS_NewError(ctx);
    assert(!JS_IsException(s.terminal));
    JS_SetUncatchableError(ctx, s.terminal);
    if (mode == 2) install(ctx, "fail", terminal_fail);
    const char *setup = mode == 2 ? "0" : "function fail() { arm(); new ArrayBuffer(1048576); }";
    JSValue ret = JS_Eval(ctx, setup, strlen(setup), "setup", 0);
    assert(!JS_IsException(ret)); JS_FreeValue(ctx, ret);
    ret = JS_Eval(ctx, paths[path], strlen(paths[path]), "repro", 0);
    int host = JS_IsException(ret), steps = 0;
    JS_FreeValue(ctx, ret);
    while (!host && JS_IsJobPending(rt)) {
        assert(++steps < 20);
        host = JS_ExecutePendingJob(rt, &job_ctx) < 0;
    }
    int uncatchable = 0, identity = 0;
    if (host) {
        JSValue e = JS_GetException(job_ctx);
        uncatchable = JS_IsUncatchableError(e);
        identity = JS_IsObject(e) && JS_VALUE_GET_PTR(e) == JS_VALUE_GET_PTR(s.terminal);
        JS_FreeValue(job_ctx, e);
    }
    printf("%-12s %-11s host=%d uncatchable=%d identity=%d caught=%d continued=%d success=%d refused=%d null_caught=%d\n",
        (const char *[]){"memory-limit", "allocator", "terminal", "tight-limit"}[mode], names[path],
        host, uncatchable, identity, s.caught, s.continued, s.success, s.refused, s.null_caught);
    if (mode == 0 || mode == 1) assert(!host && s.caught == 1 && s.success == 1);
    if (mode == 2 && (path == 0 || path == 2 || path == 3))
        assert(host && uncatchable && identity && !s.caught && !s.success);
    if (mode == 2 && (path == 1 || path == 4 || path == 5))
        assert(!host && s.caught == 1 && s.success == 1);
    JS_SetMemoryLimit(rt, 0); s.armed = 0;
    JS_FreeValue(ctx, s.terminal);
    JS_FreeContext(ctx); JS_FreeRuntime(rt);
    assert(s.live == 0);
}
static void controls(void) {
    JSRuntime *rt = JS_NewRuntime();
    JSContext *ctx = JS_NewContext(rt), *job_ctx;
    State s = {0};
    JS_SetDumpFlags(rt, JS_ABORT_ON_LEAKS);
    JS_SetContextOpaque(ctx, &s); install(ctx, "mark", mark);
    const char *code = "try { throw Error(); } catch(e) { mark(0); }"
        "try { throw RangeError(); } catch(e) { mark(0); }"
        "Promise.reject(42).catch(() => mark(0)).then(() => mark(2)); 42";
    JSValue ret = JS_Eval(ctx, code, strlen(code), "controls", 0);
    int32_t n;
    assert(!JS_IsException(ret) && JS_ToInt32(ctx, &n, ret) == 0 && n == 42);
    JS_FreeValue(ctx, ret);
    while (JS_IsJobPending(rt)) assert(JS_ExecutePendingJob(rt, &job_ctx) == 1);
    assert(s.caught == 3 && s.success == 1);
    JS_FreeContext(ctx); JS_FreeRuntime(rt);
    puts("ordinary Error, RangeError, rejection and fresh-runtime controls: PASS");
}
int main(void) {
    for (int mode = 0; mode < 3; mode++)
        for (int path = 0; path < 6; path++) run(mode, path);
    run(3, 0);
    controls();
    return 0;
}

Validation on unmodified master

• Reproducer: identical outcomes in Debug and ASan+UBSan builds; no sanitizer diagnostics. Native allocator outstanding-allocation assertions and JS_ABORT_ON_LEAKS passed. macOS leaks --atExit reported 0 leaks / 0 bytes.
• Existing api-test, lre-test, and JavaScript tests passed in Debug and ASan+UBSan; JavaScript suite: 0/117 errors, 9 excluded.
• Full Test262 under ASan+UBSan: 82,220 tests, 49 existing expected errors, 4,802 excluded, 5,736 skipped; exit status 0, with no new/changed/fixed errors relative to the checked-in expectations and no sanitizer diagnostics. Pinned Test262 revision: 5ef1e5723be95296f36afb0386676fed0205869c.
• C/C++ header checks and make jscheck passed (Apple Clang needs -Wno-unused-command-line-argument for the latter).

No engine changes are included. If this contract is wanted, its implementation should add regression coverage to api-test.c, including allocation of the diagnostic failing, host-visible reason, skipped catch/success markers, and ordinary exception controls.