Summary
AutoBatcherTest.concurrencySmokeTest intermittently aborts the macOS nightly with an AddressSanitizer stack-overflow on a hz_thread_pool worker thread. The overflow is caused by unbounded nesting of boost::future unwrap() states in impl::auto_batcher::new_id() (hazelcast/src/hazelcast/client/proxy.cpp), introduced in #1454. This is a bug in the client, not a test or CI problem: any application that calls flake_id_generator::new_id() from several threads under contention can hit it.
Failing runs
nightly-macOS-x86_64 (runner image is actually macos-26-arm64), all with the identical signature:
Roughly half of the macOS nightlies since 2026-09-06 fail this way. Linux and Windows nightlies pass.
Log excerpt
[ RUN ] AutoBatcherTest.concurrencySmokeTest
AddressSanitizer:DEADLYSIGNAL
==37550==ERROR: AddressSanitizer: stack-overflow on address 0x00016cf1bd20 (pc 0x00010c01ff04 bp 0x00016cf1c550 sp 0x00016cf1bd20 T848)
#0 _Znwm+0x28 (libclang_rt.asan_osx_dynamic.dylib)
...
#8 boost::detail::shared_state<long long>::do_continuation(boost::unique_lock<boost::mutex>&) future.hpp:590
#9 boost::detail::shared_state_base::mark_finished_internal(boost::unique_lock<boost::mutex>&) future.hpp:344
#10 boost::detail::shared_state<long long>::mark_finished_with_result_internal(long long&&, boost::unique_lock<boost::mutex>&) future.hpp:611
#11 boost::detail::future_unwrap_shared_state<boost::future<boost::future<long long>>, long long>::launch_continuation() future.hpp:5432
#12 boost::detail::shared_state<long long>::do_continuation(boost::unique_lock<boost::mutex>&) future.hpp:590
#13 boost::detail::shared_state_base::mark_finished_internal(boost::unique_lock<boost::mutex>&) future.hpp:344
#14 boost::detail::shared_state<long long>::mark_finished_with_result_internal(long long&&, boost::unique_lock<boost::mutex>&) future.hpp:611
#15 boost::detail::future_unwrap_shared_state<boost::future<boost::future<long long>>, long long>::launch_continuation() future.hpp:5432
... (same 4-frame cycle repeated; trace truncated at 255 frames)
Thread T848 created by T0 here:
#0 pthread_create+0x60 (libclang_rt.asan_osx_dynamic.dylib)
#12 hazelcast::util::hz_thread_pool::hz_thread_pool(unsigned long) util.cpp:1246
#14 hazelcast::client::test::AutoBatcherTest_concurrencySmokeTest_Test::TestBody() HazelcastTests1.cpp:2567
SUMMARY: AddressSanitizer: stack-overflow allocate.h:61 in ... std::__libcpp_allocate<boost::shared_ptr<boost::detail::shared_state_base>>
==37550==ABORTING
./scripts/test-unix.sh: line 67: 37550 Abort trap: 6
Test FAILED. Result:134
Root cause
The slow path of auto_batcher::new_id() is:
return fetch
.then(boost::launch::sync,
[this, gen](boost::shared_future<boost::shared_ptr<block>> f)
-> boost::future<int64_t> {
...
int64_t v = nb->next();
if (v != INT64_MIN) {
return boost::make_ready_future(v);
}
return new_id(); // lost the race: retry
})
.unwrap();Every call to new_id() returns a new future<int64_t> built as then(...).unwrap(). When a coalesced waiter loses the race for an ID from the freshly fetched batch, its continuation calls new_id() again and returns that future as its value. unwrap() then has to forward the inner future's result to the outer one. After N lost rounds a single caller holds a tower of N live future_unwrap_shared_state objects, each waiting on the one below:
unwrap( then(fetch1) -> new_id() )
= unwrap( then(fetch2) -> new_id() )
= unwrap( then(fetch3) -> new_id() )
= ...
When the innermost future finally gets a value, Boost marks it finished, which fires the next layer's continuation, which marks it finished, and so on. That completion cascade is synchronous and recursive on the thread that completed the last fetch (an executor worker). Stack depth is proportional to the number of consecutive lost rounds, which is unbounded.
Why waiters lose repeatedly. The fetch lambda publishes the new batch via block_.store(nb) before the fetch future is marked ready. A caller thread that obtains one ID immediately loops back into the lock-free fast path and drains the remaining IDs of the (3-element) batch while the coalesced continuations are still being run one after another on the executor thread. With batch_size = 3, 4 threads and 100k IDs per thread, a waiter can lose hundreds of consecutive rounds.
Why only macOS Debug. macOS gives secondary threads a 512 KiB stack (Linux: 8 MiB), and AddressSanitizer frames are large. In Release builds on Linux the same chain still grows without bound; it just needs far more lost rounds before it overflows, and there is no sanitizer to report it cleanly.
The Java AutoBatcher has the same for (;;) retry but blocks synchronously in a loop, so it has no equivalent stack growth.
Proposed fix
Flatten the retry so the caller's result lives in a single boost::promise<int64_t> and each retry attaches a fresh, non-nested continuation to whatever fetch is current:
new_id() creates one promise, returns promise.get_future(), and calls an internal try_get_id(p).try_get_id(p) tries the fast path, otherwise joins/elects the in-flight fetch and registers a void continuation on it.- On a won race the continuation does
p->set_value(v). On a supplier error it does p->set_exception(...).
- On a lost race the continuation posts the retry to the executor (
boost::asio::post(executor_, [this, p] { try_get_id(p); })) instead of calling new_id() inline. Posting guarantees every retry starts from an empty stack even when the next fetch is already complete (a launch::sync continuation on an already-ready future runs immediately on the subscribing thread).
No layer is left waiting on the layer below, so completion never cascades and stack depth is constant regardless of how many rounds a waiter loses. The externally visible behavior of new_id() is unchanged: one future that resolves to a valid ID or rethrows the supplier's exception; single-flight fetching is preserved.
A regression test should exercise the worst case for the old code (tiny batch, many threads, long-lived contention) so that a Debug/ASan build on macOS catches any reintroduction of nested completion.
Summary
AutoBatcherTest.concurrencySmokeTestintermittently aborts the macOS nightly with an AddressSanitizer stack-overflow on ahz_thread_poolworker thread. The overflow is caused by unbounded nesting ofboost::futureunwrap()states inimpl::auto_batcher::new_id()(hazelcast/src/hazelcast/client/proxy.cpp), introduced in #1454. This is a bug in the client, not a test or CI problem: any application that callsflake_id_generator::new_id()from several threads under contention can hit it.Failing runs
nightly-macOS-x86_64(runner image is actuallymacos-26-arm64), all with the identical signature:Roughly half of the macOS nightlies since 2026-09-06 fail this way. Linux and Windows nightlies pass.
Log excerpt
Root cause
The slow path of
auto_batcher::new_id()is:Every call to
new_id()returns a newfuture<int64_t>built asthen(...).unwrap(). When a coalesced waiter loses the race for an ID from the freshly fetched batch, its continuation callsnew_id()again and returns that future as its value.unwrap()then has to forward the inner future's result to the outer one. After N lost rounds a single caller holds a tower of N livefuture_unwrap_shared_stateobjects, each waiting on the one below:When the innermost future finally gets a value, Boost marks it finished, which fires the next layer's continuation, which marks it finished, and so on. That completion cascade is synchronous and recursive on the thread that completed the last fetch (an executor worker). Stack depth is proportional to the number of consecutive lost rounds, which is unbounded.
Why waiters lose repeatedly. The fetch lambda publishes the new batch via
block_.store(nb)before the fetch future is marked ready. A caller thread that obtains one ID immediately loops back into the lock-free fast path and drains the remaining IDs of the (3-element) batch while the coalesced continuations are still being run one after another on the executor thread. Withbatch_size = 3, 4 threads and 100k IDs per thread, a waiter can lose hundreds of consecutive rounds.Why only macOS Debug. macOS gives secondary threads a 512 KiB stack (Linux: 8 MiB), and AddressSanitizer frames are large. In Release builds on Linux the same chain still grows without bound; it just needs far more lost rounds before it overflows, and there is no sanitizer to report it cleanly.
The Java
AutoBatcherhas the samefor (;;)retry but blocks synchronously in a loop, so it has no equivalent stack growth.Proposed fix
Flatten the retry so the caller's result lives in a single
boost::promise<int64_t>and each retry attaches a fresh, non-nested continuation to whatever fetch is current:new_id()creates one promise, returnspromise.get_future(), and calls an internaltry_get_id(p).try_get_id(p)tries the fast path, otherwise joins/elects the in-flight fetch and registers avoidcontinuation on it.p->set_value(v). On a supplier error it doesp->set_exception(...).boost::asio::post(executor_, [this, p] { try_get_id(p); })) instead of callingnew_id()inline. Posting guarantees every retry starts from an empty stack even when the next fetch is already complete (alaunch::synccontinuation on an already-ready future runs immediately on the subscribing thread).No layer is left waiting on the layer below, so completion never cascades and stack depth is constant regardless of how many rounds a waiter loses. The externally visible behavior of
new_id()is unchanged: one future that resolves to a valid ID or rethrows the supplier's exception; single-flight fetching is preserved.A regression test should exercise the worst case for the old code (tiny batch, many threads, long-lived contention) so that a Debug/ASan build on macOS catches any reintroduction of nested completion.