Rotated free-slip annulus: the FMG coarse level is a dense SVD on one rank, so more ranks make the solve slower (caps kaiju at np 16) #901

Activity

1. lmoresi

   added

   bugSomething isn't workingSomething isn't working

   on Oct 30, 2026
2. lmoresi commented on Oct 31, 2026

   

   lmoresi

   on Oct 31, 2026 – with Claude

   Last edited by lmoresi

   MemberAuthor

   More actions

   Direction (Louis, 2026-10-31) and what the code does today

   Louis: np 16 cannot be the ceiling, since no 3-D problem is reachable from there. SVD being slow is why the coarse solve moved to redundant LU. The coarse level was also found not to be coarse enough, and GAMG may serve as the coarse solve where no coarser mesh is available.

   What development (fc67b65) does. The move to redundant LU applies only where the rotated problem has no rigid-rotation mode. rotated_bc.py around line 2305 picks coarse="svd" whenever _rotated_velocity_null_modes is non-zero, because redundant LU zero-pivots on the mode (#306). That covers the annulus (one mode) and the spherical shell (three). So the cases that need scale still gather a dense SVD onto one rank, and every 3-D shell will too.

   Options to measure. Same fixture, np 4 to 8 locally, then kaiju at np 16, 32 and 52:

   1. The rotation mode removed explicitly, then redundant LU. Either a nonzero shift with the null space attached to the coarse KSP, or the coarse system bordered by the rotation constraint so it is nonsingular. The prototype on feature/coarse-solve-nullspace measured the first (sweep 2).
   2. GAMG as the coarse solve. A Krylov coarse KSP with GAMG, given the rigid-body near-null space and the exact rotation mode as a null space. It stays distributed, with no gather. This is for coarse levels that cannot be made coarser.
   3. A coarser bottom level where the geometry allows it: the hierarchy extended down, so a gathered direct solve stays small.
   4. PCTELESCOPE on the coarse level, combined with 1 or 2, so that only a subset of ranks takes part.

   Done when: the coarse cost per Newton iteration is flat or falling from np 16 to np 52 on the rollback rig; the answer matches the SVD path to solver tolerance; macOS passes; and the choice is a default selected by the presence of null modes, not another user-facing option.

   Underworld development team with AI support from Claude Code
3. lmoresi commented on Oct 31, 2026

   

   lmoresi

   on Oct 31, 2026 – with Claude

   Last edited by lmoresi

   MemberAuthor

   More actions

   Refinement (Louis, 2026-10-31)

   • The ruling already in place (2026-10-28) is to handle the null space explicitly and use direct solves.
   • SVD is fine if the coarse level is coarse enough. The defect is a bottom level of 124-448 cells gathered to one rank, not SVD as such.
   • GAMG on the coarse mesh with SVD at GAMG's own base is worth investigating, as an option and not a commitment.

   The order of work is therefore:

   1. A coarser bottom level wherever the geometry allows it, so the gathered SVD or direct solve stays small at any rank count. Measure how far the annulus and shell hierarchies can be extended downward, and what that does to iterations.
   2. The null space handled explicitly, with a direct solve. Use this where a coarser mesh is not available, or as the general default if it is cheaper than SVD at the same size.
   3. Investigation: GAMG below the geometric coarse level, with SVD at GAMG's base. The rigid-body near-null space and the exact rotation mode are supplied. This is for coarse meshes that cannot be coarsened. Report it whether it wins or loses.
   4. Telescope only if a gathered solve still dominates after 1-3.

   Underworld development team with AI support from Claude Code
4. lmoresi commented on Oct 31, 2026

   

   lmoresi

   on Oct 31, 2026 – with Claude

   Last edited by lmoresi

   MemberAuthor

   More actions

   Scope of option 3 (Louis, 2026-10-31): GAMG only as the coarse-level solve of the geometric FMG (mg_coarse), with SVD at GAMG's base. It does not replace geometric multigrid on any level, and it is not a general preconditioner choice.

   Underworld development team with AI support from Claude Code
5. lmoresi

   added 2 commits that reference this issue on Oct 31, 2026

   #901: the rotated FMG's coarse solve handles the rotation modes, then…

   ...

   0502a9f

   CI reaches test_1111 (#901)

   ...

   018546d
6. lmoresi commented on Oct 31, 2026

   

   lmoresi

   on Oct 31, 2026 – with Claude

   Last edited by lmoresi

   MemberAuthor

   More actions

   GAMG as the coarse solve: kept for one case (Louis, 2026-10-31)

   GAMG as the geometric FMG's coarse solve stays available for one case only: when the coarsest mesh we can reach is too large for an efficient gathered redundant LU. It is not a default and not chosen automatically.

   What was measured (np 4; the answer matches the redundant-LU default to 1e-9):

   • GAMG's own base is at most 6 % of its coarse work, whether SVD, LU or pinned. Plain LU at the base hit no zero pivot and was cheapest.
   • Its cost is 15-40 GAMG V-cycles per geometric coarse solve: 4-10 times a gathered LU at 600-cell bottom levels (shell 4.9 s of coarse work against about 1.25 s).

   It goes in as an explicit, tested variant (coarse="gamg"), so it cannot rot. Settings: coarse rtol 1e-2, an LU base, and the rigid-body near-null space attached to the coarse operator. The guide states when to use it.

   Underworld development team with AI support from Claude Code
7. lmoresi

   added 6 commits that reference this issue on Oct 31, 2026

   #901: test that a rotated solve sets its coarse level up once

   ...

   6c06a76

   #901: the rotated route no longer sets its coarse level up twice per …

   ...

   5a30a08

   #901: GAMG as the multigrid coarse solve, when asked for

   ...

   9db23e5

   #901: the rotated coarse solve is redundant LU with a zero-pivot shif…

   ...

   b704140

   #901 review round 2: the shift on the live PC, tests that read it, ho…

   ...

   77de8d8

   #906: only an exact rotation mode selects the SVD coarse solve

   ...

   964dc3c