Skip to content

Draw sprand column by column again for 2^63 or more entries - #1143

Open
ViralBShah wants to merge 2 commits into
mainfrom
vs/sprand-hypersparse
Open

ViralBShah wants to merge 2 commits into
mainfrom
vs/sprand-hypersparse

Conversation

@ViralBShah

Copy link
Copy Markdown
Member

#1137 made sprand throw for a matrix of 2^63 or more entries, to save compile time. That rejects calls with a correct and affordable answer, including density 0, and it reached release-1.14 through #1136.

Before (main)

julia> sprand(Xoshiro(1), 2^62, 4, 0.0)
ERROR: OverflowError: a 4611686018427387904×4 sparse matrix has more entries than `sprand` can draw from

julia> sprand(Xoshiro(1), 2^40, 2^24, 1e-12)
ERROR: OverflowError: a 1099511627776×16777216 sparse matrix has more entries than `sprand` can draw from

After

julia> nnz(sprand(Xoshiro(1), 2^62, 4, 0.0))
0

julia> sprand(Xoshiro(1), 2^40, 2^24, 1e-12)
1099511627776×16777216 SparseMatrixCSC{Float64, Int64} with 18446866 stored entries

This restores the branch of #1128 that draws the rows of each column separately, since an Int cannot index the m*n entries. This keeps #1137's widelength and sizehint! changes, which are most of its saving. Invalid arguments throw ArgumentError before the size is looked at again, and test/issues.jl now pins that order.

The branch is not free. The TTFX workload, median of 16 fresh processes interleaved and pinned to one CPU of an AMD EPYC 9354, with the same --trace-compile count of 11 on both:

cold run warm run
main 485.5 ms 219.1 ms
this PR 498.5 ms 222.3 ms

That is 2.7% cold and 1.5% warm, kept because the alternative is a throw on working code. A matrix of this shape still needs one column pointer per column, so sprand(2^40, 2^40, 0.0) runs out of memory like spzeros(2^40, 2^40).

Backport candidate for 1.14, which carries the throw.

Left out: measuring 32-bit builds, where m*n overflows at 2^31 entries, so that this branch serves ordinary sizes there.

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.68%. Comparing base (9bfad27) to head (448d2ba).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1143   +/-   ##
=======================================
  Coverage   94.67%   94.68%           
=======================================
  Files          16       16           
  Lines       10422    10437   +15     
=======================================
+ Hits         9867     9882   +15     
  Misses        555      555           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant