Skip to content

custom_mg: an unsplit placed-ribbon coarse tail exhausts node memory before the first iteration #893

Open

Description

@lmoresi

Reported by kaiju through the science hub (2026-10-29); not yet reproduced elsewhere.

What happens

custom_mg.set_custom_fmg on a hierarchy whose coarse tail is unsplit placed-ribbon ladder levels (place_fault_ribbon_2d(split=False)) exhausts node memory: 170-377 GB at np 16 on a 127k-cell annulus. The growth comes after the solver's "Setup solver" stage and before the first iteration.

Split ladder levels and an unplaced coarse tail are fine on the same model.

What it is not

The case reproduces with each of these taken out, so none of them is the cause:

  • the TI model;
  • the FAC patch smoother: the band off, and FAC disabled;
  • a plain viscoplastic host instead of the campaign rheology;
  • all but one placed coarse level (it reproduces with only one).

That leaves the hierarchy or transfer build between unsplit placed levels (barycentric transfer, cross_partition="auto").

Reproduction (kaiju)

runs/slab_ti_check2b: the TAf checkpoint 40 with -uw_fault_realisation ti -uw_ti_check 2, library build 547c2e5 (bugfix/faulted-rebuild-leak, on feature/box-faults-warm). The campaign works around it with -uw_chain 0 for the TI runs.

Bearing

This is the hierarchy path #889 and #890 ship. A memory probe of the transfer build between two unsplit placed levels at np 4 would show whether this is the build or a later stage.

Underworld development team with AI support from Claude Code

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions