Crash in HAMT during garbage collection

Crash report

What happened?

We discovered a crash in the https://github.com/datadog/dd-trace-py library we maintain that actually seems to be rooted in CPython, not our own code.

Error UnixSignal: Process terminated with SEGV_MAPERR (SIGSEGV)
#0  0x00007fc92c3dd27f PyObject_GC_UnTrack
#1  0x00007fc92c4e1448 PyContextVar_Set
#2  0x00007fc92c3f3546 _PyEval_EvalFrameDefault
#3  0x00007fc92c3fbce7 PyObject_Vectorcall
#4  0x00007fc92c3f039b _PyEval_EvalFrameDefault
#5  0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#6  0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#7  0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#8  0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#9  0x00007fc92c44e25f PyIter_Send
#10 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#11 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#12 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#13 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#14 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#15 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#16 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#17 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#18 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#19 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#20 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#21 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#22 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#23 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#24 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#25 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#26 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#27 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#28 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#29 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#30 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#31 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#32 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#33 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#34 0x00007fc8f1f854ff __Pyx_PyObject_Call (/project/uvloop/loop.c:191431:15)
#35 0x00007fc8f204f948 __pyx_f_6uvloop_4loop_6Handle__run (/project/uvloop/loop.c:66901:25)
#36 0x00007fc8f2053f7c __pyx_f_6uvloop_4loop_4Loop__on_idle (/project/uvloop/loop.c:17975:25)
#37 0x00007fc8f204f9a6 __pyx_f_6uvloop_4loop_6Handle__run (/project/uvloop/loop.c:66927:24)
#38 0x00007fc8f2051047 __pyx_f_6uvloop_4loop_cb_idle_callback (/project/uvloop/loop.c:87335:19)
#39 0x00007fc8f20687d1 uv__run_idle (/project/build/libuv-x86_64/src/unix/loop-watcher.c:68:1)
#40 0x00007fc8f2065b07 uv_run (/project/build/libuv-x86_64/src/unix/core.c:439:5)
#41 0x00007fc8f1fb83a0 __pyx_f_6uvloop_4loop_4Loop__Loop__run (/project/uvloop/loop.c:18458:23)
#42 0x00007fc8f1ff612d __pyx_f_6uvloop_4loop_4Loop__run (/project/uvloop/loop.c:18876:18)
#43 0x00007fc8f1ff1e85 __pyx_pf_6uvloop_4loop_4Loop_24run_forever (/project/uvloop/loop.c:31528:18)
#44 0x00007fc8f1ff1e85 __pyx_pw_6uvloop_4loop_4Loop_25run_forever (/project/uvloop/loop.c:31331:13)
#45 0x00007fc92c3ec26e PyObject_VectorcallMethod
#46 0x00007fc92c3f039b _PyEval_EvalFrameDefault
#47 0x00007fc92c47474c PyEval_EvalCode
#48 0x00007fc92c4a24a7 _PyRun_SimpleFileObject
#49 0x00007fc92c4a1b38 _PyRun_AnyFileObject
#50 0x00007fc92c49c3be Py_RunMain
#51 0x00007fc92c46461b Py_BytesMain
#52 0x00007fc92c067d65 __libc_start_main
#53 0x0000559068986071 _start

We fixed the crash on our side with DataDog/dd-trace-py#17407 but figured it might still be worth reporting upstream.

The crash happens in PyObject_GC_UnTrack, called from the dealloc path of a HAMT node that PyContextVar_Set is trying to free via Py_DECREF. If that HAMT node's memory is already unmapped, the PyObject_GC_UnTrack write faults.

We discovered that crash in dd-trace-py accidentally because of the following code:

Store a fresh {} dict in the context's HAMT under self._storage
Put the Token returned by set() back into that same dict

def __enter__(self) -> "BaseWrappingContext":
    token: Token = self._storage.set({})          # 1. create new HAMT entry
    self._storage.get()["__dd_wrapping_context_token__"] = token  # 2. store Token in dict
    return self

Token objects holds tok_ctx, which is a strong reference to the PyContext that was active when set was called -- i.e. ts->context.

I unfortunately don't have a proposed fix yet, but I'm looking into it.

CPython versions tested on:

~~3.12~~

EDIT: this is happening on 3.12 but the stack I originally reported (and most crashes I see) are coming from Python 3.10 and 3.11.

Operating systems tested on:

Linux

Linked PRs

gh-148891: prevent crash in HAMT #148928

@KowalskiThomas

@KowalskiThomas, thank you for your crash report.

Can you provide a self-contained repro? That would go a long way towards helping us understand the bug -- it doesn't seem easy to repro given that HAMT is nearly a decade old and it's only just now being reported.

Also: There's a small amount of C++ and Cython in your package. Have you absolutely ruled out that the cause could be there? (A missed INCREF in user C/C++ code could cause any number of crashes that appear to be coming from inside CPython.)

Finally, can you reproduce the crash on 3.14 or (ideally) 3.15 latest alpha? It would be a shame to spend time investigating this only to find that it's already been fixed in a newer CPython release.

Alternatively, if you can come up with a fix in HAMT (that's not too convoluted), that would also go a long way towards helping us feel it's worth investigating this more.

It seems that we at least should add checking for this value PyContextToken *tok = token_new(ctx, var, old_val);.

token_new may return NULL, that not checked before DECREF. And I think we shouldn't set contextvar if new token is NULL.

Thanks for the quick replies!

Can you provide a self-contained repro?

I unfortunately haven't been able to make one so far, but I haven't tried very hard yet.
We store crash reports from our apps instrumented by dd-trace-py and that's how we know it happens, but that's about it... I'll try to though!

Also: There's a small amount of C++ and Cython in your package. Have you absolutely ruled out that the cause could be there? (A missed INCREF in user C/C++ code could cause any number of crashes that appear to be coming from inside CPython.)

That's a good point of course -- it's nearly impossible to completely rule it out; however if I can find a repro then I guess that question will be answered! I'll let you know here. (We run on Python 3.12 internally which is why I've reported it as 3.12.)

Alternatively, if you can come up with a fix in HAMT (that's not too convoluted), that would also go a long way towards helping us feel it's worth investigating this more.

I was also looking into this but I don't really know it so I don't have anything (yet!)

Okay so after some more work (I'll admit, lots of it was LLM-assisted...), I do have a reproducer that makes it crash (segmentation fault) almost consistently with pure Python code in 3.10 and 3.11.

from __future__ import annotations

import contextvars
import gc
import weakref

N: int = 200_000

storage = contextvars.ContextVar("storage")

_pad_vars: list[contextvars.ContextVar[object]] = [
    contextvars.ContextVar(f"pad_{i}") for i in range(20)
]

def _init_current_context() -> None:
    """Set padding vars in the CURRENT context so the HAMT is multi-level.
    This ensures hamt_node_bitmap_assoc operates at deeper tree levels
    where node allocations (and thus GC triggers) happen."""
    for v in _pad_vars:
        v.set(object())
    storage.set({})


def _build_cycle() -> None:
    """Build a reference cycle: ctx → HAMT → dict → Token → ctx."""
    token = storage.set({})
    storage.get()["__token__"] = token
    for v in _pad_vars:
        v.set(object())


def repro(n: int = N) -> None:
    """Trigger use-after-free via re-entrant ContextVar.set() in a weakref
    callback, exactly matching the production crash mechanism.

    Critical timing detail: GC must fire DURING _PyHamt_Assoc (on a node
    allocation), not before it (on token_new or bound-method creation).
    We ensure this by:
      - Caching the method descriptor to avoid bound-method allocation
      - Pre-filling the Token freelist so token_new doesn't allocate
      - Promoting the cycle to gen1, then letting gen0+gen1 collection
        happen inside _PyHamt_Assoc's first node allocation
    """
    reentrant_val = {"reentrant": True}
    cb_count = 0

    def _reentrant_cb(_ref):
        nonlocal cb_count
        cb_count += 1
        _direct_set(storage, reentrant_val)

    # Cache the method descriptor - avoids bound-method GC allocation.
    _direct_set = type(storage).set

    # Pre-fill the Token freelist so token_new() inside PyContextVar_Set
    # does NOT call PyObject_GC_New (which would trigger GC too early).
    for _ in range(8):
        _t = _direct_set(storage, None)
        del _t
    _direct_set(storage, {})
    gc.collect()

    for i in range(n):
        gc.disable()
        new_val = {}

        ctx = contextvars.copy_context()
        ctx.run(_build_cycle)
        wr = weakref.ref(ctx, _reentrant_cb)
        del ctx  # orphan - cycle objects stay in gen0

        # gc.get_count() itself allocates a GC-tracked tuple (+1 to real
        # count), and token_new inside PyContextVar_Set is another +1.
        # We need GC to fire INSIDE _PyHamt_Assoc (at the node allocation),
        # so we skip those by adding an offset.  Rotate offsets across
        # iterations so at least some hit the sweet spot regardless of
        # exact internal alloc count.
        c0 = gc.get_count()[0]
        offset = (i % 6) + 2  # try offsets 2..7
        gc.enable()
        gc.set_threshold(c0 + offset, 1, 1)

        _direct_set(storage, new_val)
        gc.set_threshold(700, 10, 10)
        del wr

    if cb_count > 0:
        print(f"[{cb_count} callbacks]", end=" ", flush=True)
    else:
        print("[WARNING: 0 callbacks]", end=" ", flush=True)


_init_current_context()
repro()

I double checked on the crashes we are getting, most of them are coming from Python <= 3.11. Some (more rare) are coming from 3.12 but I can't seem to reproduce them with my reproducer and wasn't able to find one that reproduces consistently.
I'll attach those crashes anyway if that can be any helpful.

Details

Error UnixSignal: Process terminated with SI_KERNEL (SIGSEGV)
#0   0x0000603b6c072b68 _Py_Dealloc (/usr/src/python/Objects/object.c:2626:16)
#1   0x0000603b6bfee52f Py_DECREF (/usr/src/python/./Include/object.h:705:9)
#2   0x0000603b6bfee52f hamt_tp_clear (/usr/src/python/Python/hamt.c:2636:5)
#3   0x0000603b6bfee4ef hamt_tp_dealloc (/usr/src/python/Python/hamt.c:2664:11)
#4   0x0000603b6bfe6144 Py_DECREF (/usr/src/python/./Include/object.h:705:9)
#5   0x0000603b6bfe6144 contextvar_set (/usr/src/python/Python/context.c:748:5)
#6   0x0000603b6bfe5f61 PyContextVar_Set (/usr/src/python/Python/context.c:284:9)
#7   0x0000603b6c08b0f6 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:3094:19)
#8   0x0000603b6c066ba6 _PyEval_EvalFrame (/usr/src/python/./Include/internal/pycore_ceval.h:89:16)
#9   0x0000603b6c066ba6 gen_send_ex2 (/usr/src/python/Objects/genobject.c:230:14)
#10  0x00007e1be814bdc7 task_step_impl (/usr/src/python/./Modules/_asynciomodule.c:2869:22)
#11  0x00007e1be814c5a2 task_step (/usr/src/python/./Modules/_asynciomodule.c:3188:11)
#12  0x0000603b6c072877 cfunction_vectorcall_O (/usr/src/python/Objects/methodobject.c:509:24)
#13  0x00007e1be50f7f69 __Pyx_PyObject_Call (/project/uvloop/loop.c:191431:15)
#14  0x00007e1be50f7f69 __pyx_f_6uvloop_4loop_6Handle__run (/project/uvloop/loop.c:66901:25)
#15  0x00007e1be50fd96b __pyx_f_6uvloop_4loop_4Loop__on_idle (/project/uvloop/loop.c:17975:25)
#16  0x00007e1be50f7e52 __pyx_f_6uvloop_4loop_6Handle__run (/project/uvloop/loop.c:66927:24)
#17  0x00007e1be50f9c88 __pyx_f_6uvloop_4loop_cb_idle_callback (/project/uvloop/loop.c:87335:19)
#18  0x00007e1be5115311 uv__run_idle (/project/build/libuv-x86_64/src/unix/loop-watcher.c:68:1)
#19  0x00007e1be5112647 uv_run (/project/build/libuv-x86_64/src/unix/core.c:439:5)
#20  0x00007e1be5033db5 __pyx_f_6uvloop_4loop_4Loop__Loop__run (/project/uvloop/loop.c:18458:23)
#21  0x00007e1be509be50 __pyx_f_6uvloop_4loop_4Loop__run (/project/uvloop/loop.c:18876:18)
#22  0x00007e1be50accf0 __pyx_pf_6uvloop_4loop_4Loop_24run_forever (/project/uvloop/loop.c:31528:18)
#23  0x00007e1be50accf0 __pyx_pw_6uvloop_4loop_4Loop_25run_forever (/project/uvloop/loop.c:31331:13)
#24  0x0000603b6c06459c _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11)
#25  0x0000603b6c06459c PyObject_VectorcallMethod (/usr/src/python/Objects/call.c:887:24)
#26  0x00007e1be50b0d60 __pyx_pf_6uvloop_4loop_4Loop_44run_until_complete (/project/uvloop/loop.c:33768:23)
#27  0x00007e1be50b2591 __pyx_pw_6uvloop_4loop_4Loop_45run_until_complete (/project/uvloop/loop.c:33318:13)
#28  0x0000603b6c063a18 _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11)
#29  0x0000603b6c063a18 PyObject_Vectorcall (/usr/src/python/Objects/call.c:325:12)
#30  0x0000603b6c08a807 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:2715:19)
#31  0x0000603b6c066ba6 _PyEval_EvalFrame (/usr/src/python/./Include/internal/pycore_ceval.h:89:16)
#32  0x0000603b6c066ba6 gen_send_ex2 (/usr/src/python/Objects/genobject.c:230:14)
#33  0x00007e1be814bdc7 task_step_impl (/usr/src/python/./Modules/_asynciomodule.c:2869:22)
#34  0x00007e1be814c5a2 task_step (/usr/src/python/./Modules/_asynciomodule.c:3188:11)
#35  0x0000603b6c0636fe _PyObject_MakeTpCall (/usr/src/python/Objects/call.c:240:18)
#36  0x0000603b6bfe680c _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:90:16)
#37  0x0000603b6bfe680c context_run (/usr/src/python/Python/context.c:668:29)
#38  0x0000603b6c0d5d7b cfunction_vectorcall_FASTCALL_KEYWORDS (/usr/src/python/Objects/methodobject.c:438:24)
#39  0x0000603b6c08e1a9 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:3263:26)
#40  0x0000603b6c10d4b9 PyEval_EvalCode (/usr/src/python/Python/ceval.c:578:21)
#41  0x0000603b6c12b52c run_eval_code_obj (/usr/src/python/Python/pythonrun.c:1722:9)
#42  0x0000603b6c12b4a4 run_mod (/usr/src/python/Python/pythonrun.c:1743:19)
#43  0x0000603b6c12b061 pyrun_file (/usr/src/python/Python/pythonrun.c:1643:15)
#44  0x0000603b6c12aea7 _PyRun_SimpleFileObject (/usr/src/python/Python/pythonrun.c:433:13)
#45  0x0000603b6c12acc7 _PyRun_AnyFileObject (/usr/src/python/Python/pythonrun.c:78:15)
#46  0x0000603b6c135230 pymain_run_file_obj (/usr/src/python/Modules/main.c:360:15)
#47  0x0000603b6c135230 pymain_run_file (/usr/src/python/Modules/main.c:379:15)
#48  0x0000603b6c135230 pymain_run_python (/usr/src/python/Modules/main.c:633:21)
#49  0x0000603b6c135230 Py_RunMain (/usr/src/python/Modules/main.c:713:5)
#50  0x0000603b6c134dbd Py_BytesMain (/usr/src/python/Modules/main.c:767:12)
#51  0x00007e1be8c00e40 __libc_start_main
#52  0x0000603b6c0ad2d5 _start

Error UnixSignal: Process terminated with SEGV_MAPERR (SIGSEGV)
#0   0x0000603b6c0a5f01 _PyGCHead_PREV (/usr/src/python/./Include/internal/pycore_gc.h:68:15)
#1   0x0000603b6c0a5f01 _PyObject_GC_UNTRACK (/usr/src/python/./Include/internal/pycore_object.h:251:23)
#2   0x0000603b6c0a5f01 PyObject_GC_UnTrack (/usr/src/python/Modules/gcmodule.c:2265:9)
#3   0x0000603b6bfee54f hamt_node_bitmap_dealloc (/usr/src/python/Python/hamt.c:1139:5)
#4   0x0000603b6bfee52f Py_DECREF (/usr/src/python/./Include/object.h:705:9)
#5   0x0000603b6bfee52f hamt_tp_clear (/usr/src/python/Python/hamt.c:2636:5)
#6   0x0000603b6bfee4ef hamt_tp_dealloc (/usr/src/python/Python/hamt.c:2664:11)
#7   0x0000603b6bfe6144 Py_DECREF (/usr/src/python/./Include/object.h:705:9)
#8   0x0000603b6bfe6144 contextvar_set (/usr/src/python/Python/context.c:748:5)
#9   0x0000603b6bfe5f61 PyContextVar_Set (/usr/src/python/Python/context.c:284:9)
#10  0x0000603b6c08b0f6 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:3094:19)
#11  0x0000603b6c064972 _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11)
#12  0x0000603b6c064972 method_vectorcall (/usr/src/python/Objects/classobject.c:61:18)
#13  0x0000603b6c063a18 _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11)
#14  0x0000603b6c063a18 PyObject_Vectorcall (/usr/src/python/Objects/call.c:325:12)
#15  0x0000603b6c08a807 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:2715:19)
#16  0x0000603b6c066ba6 _PyEval_EvalFrame (/usr/src/python/./Include/internal/pycore_ceval.h:89:16)
#17  0x0000603b6c066ba6 gen_send_ex2 (/usr/src/python/Objects/genobject.c:230:14)
#18  0x00007e1be814bdc7 task_step_impl (/usr/src/python/./Modules/_asynciomodule.c:2869:22)
#19  0x00007e1be814c5a2 task_step (/usr/src/python/./Modules/_asynciomodule.c:3188:11)
#20  0x0000603b6c072877 cfunction_vectorcall_O (/usr/src/python/Objects/methodobject.c:509:24)
#21  0x00007e1be50f7f69 __Pyx_PyObject_Call (/project/uvloop/loop.c:191431:15)
#22  0x00007e1be50f7f69 __pyx_f_6uvloop_4loop_6Handle__run (/project/uvloop/loop.c:66901:25)
#23  0x00007e1be50fd96b __pyx_f_6uvloop_4loop_4Loop__on_idle (/project/uvloop/loop.c:17975:25)
#24  0x00007e1be50f7e52 __pyx_f_6uvloop_4loop_6Handle__run (/project/uvloop/loop.c:66927:24)
#25  0x00007e1be50f9c88 __pyx_f_6uvloop_4loop_cb_idle_callback (/project/uvloop/loop.c:87335:19)
#26  0x00007e1be5115311 uv__run_idle (/project/build/libuv-x86_64/src/unix/loop-watcher.c:68:1)
#27  0x00007e1be5112647 uv_run (/project/build/libuv-x86_64/src/unix/core.c:439:5)
#28  0x00007e1be5033db5 __pyx_f_6uvloop_4loop_4Loop__Loop__run (/project/uvloop/loop.c:18458:23)
#29  0x00007e1be509be50 __pyx_f_6uvloop_4loop_4Loop__run (/project/uvloop/loop.c:18876:18)
#30  0x00007e1be50accf0 __pyx_pf_6uvloop_4loop_4Loop_24run_forever (/project/uvloop/loop.c:31528:18)
#31  0x00007e1be50accf0 __pyx_pw_6uvloop_4loop_4Loop_25run_forever (/project/uvloop/loop.c:31331:13)
#32  0x0000603b6c06459c _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11)
#33  0x0000603b6c06459c PyObject_VectorcallMethod (/usr/src/python/Objects/call.c:887:24)
#34  0x00007e1be50b0d60 __pyx_pf_6uvloop_4loop_4Loop_44run_until_complete (/project/uvloop/loop.c:33768:23)
#35  0x00007e1be50b2591 __pyx_pw_6uvloop_4loop_4Loop_45run_until_complete (/project/uvloop/loop.c:33318:13)
#36  0x0000603b6c063a18 _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11)
#37  0x0000603b6c063a18 PyObject_Vectorcall (/usr/src/python/Objects/call.c:325:12)
#38  0x0000603b6c08a807 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:2715:19)
#39  0x0000603b6c066ba6 _PyEval_EvalFrame (/usr/src/python/./Include/internal/pycore_ceval.h:89:16)
#40  0x0000603b6c066ba6 gen_send_ex2 (/usr/src/python/Objects/genobject.c:230:14)
#41  0x00007e1be814bdc7 task_step_impl (/usr/src/python/./Modules/_asynciomodule.c:2869:22)
#42  0x00007e1be814c5a2 task_step (/usr/src/python/./Modules/_asynciomodule.c:3188:11)
#43  0x0000603b6c0636fe _PyObject_MakeTpCall (/usr/src/python/Objects/call.c:240:18)
#44  0x0000603b6bfe680c _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:90:16)
#45  0x0000603b6bfe680c context_run (/usr/src/python/Python/context.c:668:29)
#46  0x0000603b6c0d5d7b cfunction_vectorcall_FASTCALL_KEYWORDS (/usr/src/python/Objects/methodobject.c:438:24)
#47  0x0000603b6c08e1a9 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:3263:26)
#48  0x0000603b6c10d4b9 PyEval_EvalCode (/usr/src/python/Python/ceval.c:578:21)
#49  0x0000603b6c12b52c run_eval_code_obj (/usr/src/python/Python/pythonrun.c:1722:9)
#50  0x0000603b6c12b4a4 run_mod (/usr/src/python/Python/pythonrun.c:1743:19)
#51  0x0000603b6c12b061 pyrun_file (/usr/src/python/Python/pythonrun.c:1643:15)
#52  0x0000603b6c12aea7 _PyRun_SimpleFileObject (/usr/src/python/Python/pythonrun.c:433:13)
#53  0x0000603b6c12acc7 _PyRun_AnyFileObject (/usr/src/python/Python/pythonrun.c:78:15)
#54  0x0000603b6c135230 pymain_run_file_obj (/usr/src/python/Modules/main.c:360:15)
#55  0x0000603b6c135230 pymain_run_file (/usr/src/python/Modules/main.c:379:15)
#56  0x0000603b6c135230 pymain_run_python (/usr/src/python/Modules/main.c:633:21)
#57  0x0000603b6c135230 Py_RunMain (/usr/src/python/Modules/main.c:713:5)
#58  0x0000603b6c134dbd Py_BytesMain (/usr/src/python/Modules/main.c:767:12)
#59  0x00007e1be8c00e40 __libc_start_main
#60  0x0000603b6c0ad2d5 _start

According to the LLM analysis, the reason this isn't crashing on 3.12+ is the following -- I have no knowledge at all on the GC so I don't know if it is true...

Python 3.12+ moved GC from inline (fires during _PyObject_GC_Alloc) to deferred (fires at the eval breaker between bytecodes). On 3.10/3.11, GC fires during hamt_node_bitmap_new inside _PyHamt_Assoc, which allows a weakref callback to re-enter contextvar_set and free the HAMT mid-traversal. On 3.15, GC never runs during C function execution, so the pure-Python reentrancy vector can't trigger the UAF.

If I'm not mistaken, though, the latent bug is stil in the code, it's just not being triggered by the current version of the GC. I think the following branch should fix it (increase reference count before the possibly re-entrant call): main...KowalskiThomas:cpython:kowalski/fix-prevent-crash-in-hamt.

It also seems like a similar crash was fixed in the past in #142829 in December last year.

Understood. Thanks for the effort! Can you fix the latent bug? Is the one-line fix that Sergey suggested maybe all we need to do?

I don't think so. As far as I can tell, Sergey's proposed change addresses a separate concern from the change on my branch:

Sergey's proposal is to make sure we check the return value of token_new before decref-ing it (since it could be NULL)
My proposal is to ensure a strong reference to vars is held during the call to _PyHamt_Assoc to make sure it does not get freed.

To confirm, I tried running my Python reproducer with my changes and Sergey's separately, after cherry-picking my commit on top of v3.10.20.

Running the reproducer on a freshly-built v3.10.20 crashes
Running the reproducer on v3.10.20 + my changes does not crash
Running the reproducer on v3.10.20 + Sergey's changes crashes
Running the reproducer on v3.10.20 + my changes + Sergey's changes crashes (expectedly)

I took the liberty of opening #148928 as a draft since it does seem to make the problem go away. (I do think it would be worth adding Sergey's fix as well since it's extra safety for cheap.)

Thanks. I hope someone else can review that code, I'm not closely familiar with HAMT internals. Your approach of using INCREF/DECREF around the offending calls certainly makes sense to me!

Navigation Menu

Repository navigation

Crash in HAMT during garbage collection #148891

Crash report

What happened?

CPython versions tested on:

Operating systems tested on:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Crash in HAMT during garbage collection #148891

Description

Crash report

What happened?

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

gvanrossum commented on Apr 22, 2026

sergey-miryanov commented on Apr 22, 2026

KowalskiThomas commented on Apr 22, 2026

KowalskiThomas commented on Apr 23, 2026

gvanrossum commented on Apr 23, 2026

KowalskiThomas commented on Apr 23, 2026

gvanrossum commented on Apr 23, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions