custom_mg at np 32 and 52: the nested interpolation fails on preallocation, the FMG build falls back and the ranks deadlock #902

Activity

1. lmoresi

   added

   bugSomething isn't workingSomething isn't working

   on Oct 30, 2026
2. lmoresi

   added 8 commits that reference this issue on Oct 31, 2026

   Interpolate within one mesh cell by cell, not through PETSc's matrix (#…

   ...

   37ac408

   Interpolate within one mesh through UW3's copy of PETSc's nested inte…

   ...

   89037a1

   Mark the summed discontinuous-to-continuous projection in project_to_…

   ...

   96d39f7

   test_1107: custom-P FMG over a coarse level where a rank owns nothing (…

   ...

   fcef03f

   #902: the mesh's coordinate space is built through UW3's interpolator…

   ...

   57c0de9

   Text that still named PETSc's createInterpolation where UW3's copy no…

   ...

   b148115

   #901: test that a rotated solve sets its coarse level up once

   ...

   6c06a76

   #902: the coordinate space keeps PETSc's route only for isoperiodic m…

   ...

   57d8778
3. lmoresi commented on Oct 31, 2026

   

   lmoresi

   on Oct 31, 2026

   MemberAuthor

   More actions

   Mechanism

   DMCreateInterpolation between two DMs on one plex goes to PETSc's DMPlexComputeInterpolatorNested (plexfem.c).

   • Its preallocation loop runs only on a rank with local rows or columns. Its fill loop runs on every rank that holds cells.
   • A shared point belongs to the highest rank that holds it. A low rank whose cells sit inside higher ranks' regions therefore owns nothing, and it skips the preallocation.
   • It still sends its cells' entries to their owners. An entry that only its cells generate was never preallocated: the owner raises "New nonzero ... caused a malloc" in MatAssemblyEnd, and the other ranks wait in the assembly's collective.
   • P1 to P1 escapes, because the neighbours' cells cover every vertex pair. P2 fails.

   Custom-P multigrid reads each level's node coordinates through Mesh._get_coords_for_basis, which builds this matrix. On kaiju, the 124-cell coarsest level at np 32 and 52 left a rank owning nothing.

   On the laptop, natural partitions did not reproduce it: an annulus, plain and faulted, at np 8, and the 124-cell annulus at np 32 oversubscribed. A shell partition at np 2 reproduces it every time, with rank 0 given one interior cell of a 4 x 4 grid.

   The guard came in with PETSc b4ec2976 ("Plex: Fix interpolator for empty mesh", v3.24.0). By source, it is still there in v3.25.6, v3.26.0 and main. A petsc4py reproducer and the report text are ready for upstream.

   Notch draft 02 (the 3-D notch at np 16) is the same defect.

   Fix, on bugfix/custom-mg-many-ranks (57d8778, no PR yet)

   • The interpolation. UW3 now builds this matrix with its own copy of the nested interpolator, for the same-plex finite-element case (UW_DMCreateInterpolationWithinPlex in petsc_compat.h). The preallocation and fill loops run over the same cells.
     • It is used by _get_coords_for_basis, project_to_degree and boundary_flux._trace_interior_coords.
     • It also builds a mesh's coordinate space. createCoordinateSpace projected degree >= 2 coordinates through PETSc's own interpolator, so Mesh(degree=2) failed during setup on such a rank. The space is now installed without projection, and the coordinates are copied or interpolated through the copy. Periodic coordinates keep PETSc's projection.
     • Wherever PETSc's version assembles, the copy gives the same matrix: max |A - B| = 0 and the same nonzeros. That was checked on triangles, quads, tets, hexes, a degree-2 annulus and a spherical shell, for continuous and discontinuous degree 1 to 3, at np 1 and 4.
   • The deadlock (custom_mg.build_transfers: a rank-local exception decides a collective retry #886). build_transfers caught a builder failure on the rank where it happened and retried there alone, so the ranks met different collectives.
     • The barycentric builder's verdict is now agreed across ranks as soon as it returns.
     • The checks on allreduced sizes, in custom_mg and in fault_mg's provenance transfer, raise CollectiveBuildError on every rank.
     • In parallel, only that error is retried or degraded.
     • The fault split's replay down the tail agrees on any exception, not only its own refusal types, and every verdict runs on a communicator of its own, so it cannot pair with a collective another rank is still inside.
     • A PETSc error raised inside the interpolation's own assembly cannot be made collective after the fact. The fix for that path is that the error no longer occurs.
   • Tests.
     • test_1107 uses two shell partitions: rank 0 on an interior cell (no vertex or edge owned) and on a boundary cell (only its wall edge owned). It covers node coordinates at degree 1 to 3, projection between degrees, a degree-2 mesh, and a custom-P FMG solve whose coarse level is the interior partition. With PETSc's route put back, one test at a time at np 2, these fail: interior P2 coordinates, the projection, the degree-2 mesh and the FMG case (inside CustomMGHierarchy.build, "New nonzero at (88,32)"); and boundary P1 coordinates and the projection. P1 reaches the bug when a rank's cell has a wall edge.
     • test_1108 covers a builder failing on one rank, and a check failing on every rank. Both hang on development.

   Still open here

   • Kaiju's np 32 and 52 run of the faulted annulus, compared against np 16. This is the "Done when" above.
   • A rank holding no cells at all, on a degree-2 mesh: the FVM cell-size call fails there and its error is ignored, so the ranks part company. It is a different mechanism, filed as Mesh(degree=2): the FVM cell-size call fails and its error is ignored, so a rank with no cells hangs the others #908.

   Underworld development team with AI support from Claude Code