Fix monomial comparison and tail bound in sparse-FGLM-col matrix

[wegank]

This fixes two defects in copy_poly_in_matrixcol_no_zero, which builds the multiplication matrix for the sparse-FGLM-col algorithm (option -C).

Tail bound

The index k walks through the tail of the polynomial from its smallest term, at index end - 1 - k. It was bounded by end, a position in the whole coefficient array, instead of by the length pos of the polynomial. Hence, when the polynomial is not the first one in the array, k could go past the tail to the leading term and then to the terms of the previous polynomial, whose coefficients could then be written into the row. The memory is valid, so AddressSanitizer does not detect it. The bounds are now k < pos - 1 and k < pos - 2.

Monomial comparison

is_larger_exponent breaks ties between monomials of the same degree with for (i = nvars-1; i>1; i--), so it never compares the exponents of x_1 (and hence x_0). Monomials of the same degree which differ only in x_0 and x_1 were thus considered equal; with two variables, all monomials of the same degree were. The loop now goes down to i = 1, which suffices since the degrees are equal. The same loop is fixed in is_larger_exponent_bs, used by copy_extrapoly_in_vector_no_zero and copy_extrapoly_in_matrixcol_no_zero, also in the -C path.

Testing

No test in make check reaches this code, and the -C path currently crashes before it on every system I tried (see below). I therefore tested the function directly with a small harness, calling copy_poly_in_matrixcol_no_zero on hand-built data where the target polynomial follows another polynomial in the array:

Case	Before	Tail bound only	Both fixes
A: y^2 - x, 2 variables	3 wrong entries, one from the previous polynomial	coefficient of x in the y column	OK
B: z^2 + 3x + 7z + 11 (no y), 3 variables	coefficient of x in the y column	same	OK
C: full tail, 3 variables	OK	OK	OK
D: basis contains monomials larger than the leading term	2 entries from the previous polynomial	OK	OK

make check passes (68/68) with a normal -O2 build.

Remaining issues in the -C path (not addressed here)

• src/msolve/msolve.c:4305 reads tbr->lmps[1] while tbr->lml == 1; under AddressSanitizer every zero-dimensional system I generated crashes there.
• src/msolve/msolve.c:4189 allocates mul with gens->nvars entries, while insert_multiplied_poly_in_hash_table reads ht->evl = nvars + 1 of them (line 4636 uses bht->evl). src/neogb/nf.c:40 uses the same size; I have not checked whether it is read past its end there.
• In these two copy_extrapoly_*_no_zero functions, while (b < 0 && k < len-2) looks one short, but I have not confirmed it.

Related: #379 fixes a similar out-of-bounds read in copy_poly_in_matrix (default path); the two changes are independent.

This change was made with the assistance of generative AI (Claude Code).

Co-authored-by: Claude Opus 5.5 noreply@anthropic.com

🤖 Generated with Claude Code