Prefetch index lookups

[ksahlin]

Looking up a randstrobe reads its bucket_starts entry and then its
bucket in the randstrobes vector. For large indexes, both are usually
cache misses, and the bucket can only be located once the bucket_starts
entry has been read. find_all_hits waited for these misses one lookup
at a time.

find_all_hits now prefetches the index data for 64 randstrobes at a
time before looking them up: first all bucket_starts entries, then all
buckets. The CPU then waits for many cache misses at the same time
instead of one after the other. hash_revcomp is prefetched as well
because is_too_frequent looks it up.

Buckets of up to 64 randstrobes are prefetched completely. For larger
buckets, which come from repetitive seeds, the first and last randstrobe
and approximately those that the binary search reads in its first three
steps are prefetched.

Output is unchanged.

Speedup compared to v0.19.0 on an Apple M4 Max, mapping only (-x),
1 thread, median of 5 runs (time finding hits, total mapping time):

• CHM13, 150 nt reads: -21%, -12%
• CHM13, 10 kb reads: -25%, -12%
• maize, 150 nt reads: -23%, -12%
• maize, 10 kb reads: -23%, -11%
• rye, 150 nt reads: -22%, -13%
• rye, 10 kb reads: -23%, -12%
• fruit fly, 5 kb reads: -26%, -11%
• chrY, 150 nt reads: -4%, -1%
• chrY, 10 kb reads: -10%, -2%

Speed on x86-64 was not measured.

[marcelm]

I’ve benchmarked this and get a 7% average speedup for single-end reads in mapping-only mode, which is not too bad. I do not want to merge this without understanding what the code does, however, which I don’t have the time for right now. I also hope the code doesn’t need to be that complicated, but that will also need some time to figure out.

[ksahlin]

Thanks for benchmarking! I have simplified the PR. It now adds 78 lines instead of 124, and a good part of them are comments.

What changed:

• Instead of looking ahead by 16 and 8 randstrobes (two distances and two warm-up loops), find_all_hits now calls a small helper, prefetch_lookups(), once per 64 randstrobes. It prefetches first all bucket_starts entries and then all buckets. The lookups themselves are unchanged.
• prefetch_bucket() uses ordinary slice indexing, like get_masked_from(), so the pointer arithmetic, the cache-line size constant and the defensive bounds handling are gone. Small buckets (up to 64 randstrobes) are prefetched completely. For large ones, the entries at len * j / 8 are prefetched, which are approximately those read in the first three steps of the binary search.
• The unit test is gone.
• The only platform-specific part left is prefetch() itself. It is _mm_prefetch on x86-64 and a single prfm instruction via asm! on aarch64, because neither the aarch64 intrinsic nor std::hint::prefetch_read is stable yet.

I also tried some even simpler versions, but they were slower:

• Prefetching only the first, middle and last randstrobe of a bucket lost 3-5 percentage points on maize and rye. Lookups there often land in large repeat buckets, so the binary search entries need to be prefetched.
• Dropping the separate bucket_starts prefetch lost about 2 points.

On an M4 Max, the new version is as fast as the previous one or slightly faster on all nine datasets I tried. Total mapping time is down 9.8% compared to v0.19.0 (geometric mean), versus 8.6% for the previous version. Output is identical. I could not measure on x86-64, so it would be great if you could rerun your benchmark.