Hub startup BuildupAction fails with HubDisposingException when parent teardown races initialization — hub marked FAILED for an expected shutdown race

[systemorph-com]

What is failing

During startup of the mkleiner/_Install/SocialMedia plugin hub, the DataExtensions.StartDataSourcesAndOpenGate BuildupAction faulted because the SynchronizationStream constructor refused to bind: the host hub was already shutting down (HubDisposingException: Hub ... is shutting down — cannot create 'MeshNode'). The hub was consequently put in FAILED state and logged at fail level, even though the underlying condition is an expected teardown/init race on a hub that was going away anyway.

Probable cause (medium confidence)

This is the constructor refusal path added by the stream-liveness work (Stream Liveness and the Hub Reference, #3321 step 3): when the hub is already past accepting RegisterForDisposal registrants, it disposes the release hook inline and the SynchronizationStream constructor re-checks and refuses with HubDisposingException (src/MeshWeaver.Data/Serialization/SynchronizationStream.cs, ctor / ReleaseHub() branch). Here the parent (portal circuit or recycle) tore down the hub while its data-source initialization was still in flight — DataContext.InitializeDataSources() → SynchronizationStream ctor hit the refusal. The exception is deliberately classified transient (ErrorType.ShuttingDown, “the address may reactivate; retry”), but HandleInitialize treats any BuildupAction fault as a hard initialization failure and marks the hub FAILED. So the defect is arguably in the error handling: a disposal-in-progress refusal during init should not be surfaced as a hub-initialization failure.

Impact

One occurrence on one pod (memex-portal-deployment-…-mvzb5), one timestamp. The hub was being disposed regardless, and the exception itself tells callers to retry. No data loss, no user-visible path affected — this is noise-shaped but a real classification gap worth a cheap fix.

Where to look

src/MeshWeaver.Data/Serialization/SynchronizationStream.cs (constructor refusal / ReleaseHub()), reached from DataContext.InitializeDataSources() via DataExtensions.StartDataSourcesAndOpenGate (see Initialization Gates and Hub Initialization Failure). Consider catching/absorbing HubDisposingException in the BuildupAction path when the hub is already disposing, instead of failing initialization.

No existing issue or incident covers this exact shape; the closest related work is the stream-liveness program (#3321/#3380/#3386), all closed.

---

Evidence

Fingerprint	6f9895a929bc8b1b
Category	MeshWeaver.Messaging.MessageHub
Severity	Error
Exception	MeshWeaver.Messaging.HubDisposingException
Top frame	MeshWeaver.Data.Serialization.SynchronizationStream1..ctor(StreamIdentity StreamIdentity, IMessageHub Host, Object Reference, ReduceManager1 ReduceManager, Func2 configuration)`
Namespace	memex-cloud
Pods	memex-portal-deployment-7d7f4d84f9-mvzb5
Occurrences	1
First seen	2026-09-23 17:50:06Z
Last seen	2026-09-23 17:50:06Z
Routing	not determined — no configured route matches the category MeshWeaver.Messaging.MessageHub. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.

Recent log lines

2026-09-23 17:50:06Z memex-portal-deployment-7d7f4d84f9-mvzb5 fail: MeshWeaver.Messaging.MessageHub[0]
      Hub mkleiner/_Install/SocialMedia initialization failed — BuildupAction 1 of 2 (DataExtensions.StartDataSourcesAndOpenGate) faulted (HubDisposingException: Hub mkleiner/_Install/SocialMedia is shutting down — cannot create "MeshNode". The address may reactivate (recycle / restart); retry to get the authoritative answer.
      Object name: 'mkleiner/_Install/SocialMedia'.). Hub is now in FAILED state.
      MeshWeaver.Messaging.HubDisposingException: Hub mkleiner/_Install/SocialMedia is shutting down — cannot create "MeshNode". The address may reactivate (recycle / restart); retry to get the authoritative answer.
      Object name: 'mkleiner/_Install/SocialMedia'.
         at MeshWeaver.Data.Serialization.SynchronizationStream`1..ctor(StreamIdentity StreamIdentity, IMessageHub Host, Object Reference, ReduceManager`1 ReduceManager, Func`2 configuration) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Data/Serialization/SynchronizationStream.cs:line 1383
         at MeshWeaver.Data.DataContext.InitializeDataSources() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Data/DataContext.cs:line 365
         at MeshWeaver.Data.DataExtensions.<>c__DisplayClass5_0.<StartDataSourcesAndOpenGate>b__0() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Data/DataExtensions.cs:line 130
         at System.Reactive.Linq.ObservableImpl.Defer`1._.Run()

_{Opened automatically from Admin/_LogIncident/6f9895a929bc8b1b. Recurrences are folded into this issue rather than opening new ones.}

[rbuergi]

The premise of this issue does not hold. The hub was NOT shutting down, and the exception was mislabelled. Fixed in #5609 (open).

The line reads Hub … initialization failed — BuildupAction 1 of 2 … faulted (HubDisposingException …). Hub is now in FAILED state. That comes from the arm of MessageHub.HandleInitialize that runs only when IsShuttingDown || IsTerminatedByScopeTeardown(ex) is false. The shutdown arm logs Debug and records no failure. So at that moment this hub was not disposing, its creation was not frozen, and its DI scope still resolved.

SynchronizationStream's constructor throws HubDisposingException whenever GetHostedHub(sync/…) returns null. The null outcomes are:

outcome	IsShuttingDown at the catch
creation freeze (CloseCreation / collection disposing)	true, the same monotonic volatile flags
container disposed (IsHostContainerDisposed)	scope probe true. Checked: Autofac 9.3.4 disposes child scopes with their parent, so the two probes agree
RunLevel > Started	true, reached only after Dispose()
HostedHubOutcome.ConstructionFaulted: the sub-hub's build ran and threw on a live host	false

Only the last one fits this line. The constructor turned a construction fault into "is shutting down — retry", and absorbing it during init, as proposed here, would have hidden a real fault behind a teardown label.

Fix (#5609). The constructor reads the outcome through TryGetHostedHub. On ConstructionFaulted it throws InvalidOperationException, naming the root fault and carrying it as the inner exception. It is deliberately not an ObjectDisposedException, so nothing reads it as retryable. Every other null is still the transient HubDisposingException. Pinned by SyncHubConstructionFaultIsNotAShutdownTest, which injects a real fault inside the sub-hub's construction. Its control case shows the freeze still refuses as a shutdown. With the fix line disabled, the test fails.

After the roll, the next occurrence names the actual fault. HostedHubsCollection.CreateHub already logs Failed to create hosted hub for address sync/… at Error with that exception, on the same pod and at the same instant.

Not established: WHAT faulted on memex-portal-deployment-7d7f4d84f9-mvzb5 at 17:50:06Z. The same deployment had heap-exhaustion bursts that day: a 9.7 GiB burst in 107 s on …777677694-cc6zz, with OutOfMemoryException inside HostedHubsCollection; see #5555. An OOM inside the sub-hub build is a candidate, not a reading. I did not read that pod's Failed to create hosted hub line, because the control instance's Logs action is an owner step.

[rbuergi]

#5609 is merged: main at fe33e13a5b, through the merge queue, with the build and the review check both green. It is not yet running: memex.meshweaver.cloud serves 9a61fed964 and memex.systemorph.com serves 39f49494eb, neither of which contains it. It takes effect when an image containing it is rolled out (an owner step). No manual recycle is needed. Leaving this open until a post-roll reading shows the fingerprint has gone quiet on an image that contains fe33e13a5b.

[rbuergi]

Post-roll sweep 2026-09-24 ~06:50Z: the fix is not live on either portal yet

What is running.

• memex-cloud: /api/version returns 9a61fed964. Ops/Status/memex-cloud (06:42:00Z) shows 4/4 replicas on 3.0.0-ci.9263. CD run 9263 built core 9a61fed964 with Plugins 41bcfe968f.
• memex: /api/version returns 39f49494eb. Ops/Status/memex shows the serving pods on 3.0.0-ci.9218, which is core 39f49494eb with Plugins e9d9283abe. A third pod, on ci.9260, is unready, so the roll is converged: false.

This issue's fix: #5609 (fe33e13a5b). It is not an ancestor of either running build (git merge-base --is-ancestor, exit 1).

Where it is. 3.0.0-ci.9287 (CD run 35956222018) is core bf5ac85526 with Plugins c4795a8d5b, and it contains this fix. The registry has sealed it. memex-cloud's self-updater logged [SelfUpdate] handing 3.0.0-ci.9287 to the control lane (https://memex.systemorph…) at 06:47Z. So the roll now waits on the control instance, which is an owner step and was not done here.

Still open. It closes on a post-roll reading once a ci.9287-or-later image serves the namespace this fingerprint fires in.