Repository navigation
Rotated path: FAC patch blocks run ILU(0), not the configured LU #892
Description
On the rotated free-slip path the FAC patch smoother's blocks are not solved by the LU that custom_mg._configure_pcmg configures; they run PCASM's default ILU(0).
_configure_pcmg (composite mode, the default) makes a patch level's PC composite (whole-level SOR, then ASM over the band blocks) and writes mg_levels_<l>_sub_1_sub_pc_type lu and mg_levels_<l>_sub_1_sub_pc_factor_shift_type nonzero into the options DB. solve_rotated_freeslip then calls setUp() on the level smoother and deletes the keys (rotated_bc.py, after "Force the FAC levels' smoother setup NOW", at both call sites). PCSetUp_Composite only hands its children their operators; the ASM child is set up at its first PCApply, after the keys are gone, so its block KSPs read nothing and take ILU(0). The same code is on development (bfd07b7).
Measured on feature/fault-split-integration (799ef02), PETSc 3.25.0. Hierarchy: welded base levels with the split ladder as the finest level, the case that gets the patch. The block PC is read back from the live sub-KSPs.
| case | block PC | velocity iterations |
|---|---|---|
| straight frictionless fault in a unit box, simple shear, h_b/S = 24, np 1 (all applications of the velocity solve) | ilu (as installed) | 344 |
| same, blocks by LU | lu | 67 |
| 2-D yielding-lid convection box with a split fault, one solve from rest, h_b/S = 16, np 4 (per Newton increment) | ilu (as installed) | 73-200, one cap |
| same, blocks by LU | lu | 7 |
On the box the solve time falls from 1014 s to 267 s (h_b/S 16) and from 1346 s to 193 s (h_b/S 25); the Newton counts and the peak slip are unchanged. With LU blocks the count no longer depends on h_b/S (linear: 66, 65, 67 at 6, 12, 24; box: 7 per increment at 8, 16, 25).
Keeping the _sub_1_sub_ keys in the DB (dropping them from the list _configure_pcmg returns, so the rotated path does not delete them) makes the blocks read LU; that is how the LU rows above were measured. A library fix would set up the composite's children before the keys are deleted, for example after vel_pc.getMGSmoother(_l).setUp():
spc = vel_pc.getMGSmoother(_l).getPC()
if spc.getType() == "composite":
spc.getCompositePC(1).setUp()This is untested in the library. It changes the block solver of every rotated solve that gets a FAC patch, so it needs the solver benchmarks before it merges.
Underworld development team with AI support from Claude Code
Activity
lmoresi commentedon Oct 29, 2026
Plan (dev hub, 2026-10-31)
Louis has made this a priority. Below: what the history shows, then the plan.
History, from the code
NUMERIC_ZEROPIVOTon the rotated Galerkin block. The rotated path forces the level smoother'ssetUp()before it deletes the keys, and for plain ASM that works.custom_mg.py1302 and 1375 on development) never delete the keys, so their blocks should read LU. That is still to be confirmed below.custom_mg.py:1556at 4698938). The tail sweep found the same.custom_mg.pystill says "Subdomain solver: SOR ... no factorization" directly above a default of"lu".Plan
Confirm the scope. One fixture, reading the live block sub-PC type at each point:
ilu;lu; if it is not, this issue is wider than its title;sor, thenilu.The regression test first, shown failing. It reads back the live block PC of every FAC level and fails unless it is the configured type. It covers composite and plain ASM, rotated and non-rotated, serial and
tests/parallelat np 2. An options-DB check would not catch this, because the keys were right all along.The fix. One helper in
custom_mg, called afterpc.setUp()from every call site, rather than a third copy of the loop:C has two rotated call sites (
rotated_bc.py2340 and 2493); development has one. So the fix is cleanest stacked on The add-on partitioned once, custom multigrid on empty ranks, and the rotated solve's switch and record #890. If C stalls, rebase onto development.Benchmarks before the PR.
tests/parallelat np 2 and 4.PC_FAILEDand zero pivots, the reason SOR was chosen in August.fac_block_rows(default 2000).After the merge, re-measure what rested on ILU blocks. Louis then rules on the production tail. What follows from that ruling:
uw_chaindefault (science hub);custom_mg.py:1556).3-D and parallel cost of the LU blocks. The 2-D result does not carry over: about 32k block rows summed at ratio 25 on the box.
Write the intent where the next refactor will meet it.
The wider class. The same test should read back every MG level's live configuration (smoother, coarse solve) against what was configured. A key deleted before its lazy consumer runs is the general failure, and Solver introspection does not report the numerical configuration a solve ran with #806 (solvers do not report the configuration they ran with) is its observability half.
Proposed owner: the laptop integration session, which measured this issue and ran the tail sweep. Steps 1 to 4 wait for Louis's go; the merge waits on #863, #882, #889 and #890 in that order.
Underworld development team with AI support from Claude Code
Generated by Claude Code
FAC patch blocks run the configured LU on the rotated path (#892)
lmoresi commentedon Oct 29, 2026
Step 1: scope
Only the rotated path runs ILU. The non-rotated FAC installs read LU.
Method. One fixture: the composed ribbon stack of
test_1022(base L0 + L1, an unsplit mid ribbon, the finest ribbon cut and split), serial, PETSc 3.25.0. After the solve we read every level's smoother back from the live PC objects, not from the options database; the factor shift comes from PCView. In every build the installedcustom_mg.pyandrotated_bc.pywere checked against the source.Paths.
solve_with_fault.stokes.solve(). This is the velocity-block install,_install_velocity_block_transfers.test_1017's NVB band hierarchy. This is the top-level PC install,_install_transfers."its" is velocity sub-KSP iterations on the last application (KSP iterations for Poisson).
What the table shows.
_sub_1_sub_keys in the database after the solve; the rotated path deletes them before its ASM sets up.rotated_bc.py(B, Split-level fault multigrid, the thin-sheet solver, and sheet extrusion #889) clears every coarse level's patch rows on that path, whereCustomMGHierarchy.installkeeps the rows of any level the restriction leaves whole. This is unchanged by the fix and is measured separately.The regression test's fixture (a fault split in place over a welded 1/8, 1/16 base) shows the same split by path: the rotated saddle-point and rotated thin-sheet solves read ilu in the composite form, serial and np 2; plain ASM (
UW_FAC_MODE=asm) and both non-rotated installs read lu.The probe and records are in
~/+Simulations/fac_892/on the laptop (scope_probe.py,records/*.json).Underworld development team with AI support from Claude Code
#892 review: the helper's docstring says what it does, the pin is a c…