[venkat1701]

Statistics subscribes to the change feed and rebuilt its catalog histograms after every 10,000 member and slot changes. Each rebuild streams the whole catalog and the reverse index. The benchmark loads hyperedges in transactions of 1,000, which comes to more than 5,000 changes per commit, so the load paid for a full scan every second commit. The number of scans grows with the load and each scan grows with the store, which makes the total work quadratic.

The subscriber now waits for max(10,000, size / 10) changes, where size is the atoms plus memberships the last refresh counted (atoms + cardinality.sum() × stride). Those are the same units the change counter uses: in the engine, an edge with k members is one catalog change and k member changes. Small stores behave as before and refresh after their first 10,000 changes. Large stores refresh about every 10% of growth, so a bulk load does O(log N) scans. The tradeoff is that catalog statistics on a large store can be up to a tenth of its size out of date. STATS still calls refresh() first, and catalog() still refreshes when nothing has been computed yet. The planner reads only the cardinality feedback, not the catalog, so plans do not change.

What was measured

The probe runs the same steps as the first two Comparison run workloads with the same options (--store hstore --durability async --history 64, default cache): it ingests nodes, then edges, in transactions of 1,000 using the benchmark's Dataset. Before reading counters it waits until the statistics subscriber has acknowledged the last commit, so the async refresh work is counted. Allocation is getTotalThreadAllocatedBytes across all threads.

"main" is origin/main with the same package-private counter and subscriber hook as this branch, and the threshold fixed at 10,000. The branch is this PR. Both builds used the same probe, heap (4 GB at scale 1, 8 GB at scale 4) and dataset seed.

	scale 1 main	scale 1 branch	scale 4 main	scale 4 branch
full scans during ingest.edges	50	24	200	28
bytes allocated during ingest.edges (median)	3,839,403,712	3,586,258,704	21,417,608,352	18,925,662,688
range	3,760,522,072 to 3,965,029,072	3,558,835,032 to 3,588,191,472	21,360,114,776 to 21,475,101,928	18,626,881,104 to 19,224,444,272
full scans during ingest.nodes	12	12	50	43

Three runs each at scale 1, two at scale 4 (the scale 4 median is the mean of the two). Scan counts were the same in every run. Allocation during ingest.edges drops by about 7% at scale 1 and 12% at scale 4. Scale 1 has 50,000 nodes, 100,000 edges and 362,757 incidences. Scale 4 has 200,000 nodes, 400,000 edges and 1,453,427 incidences. I did not compare timings because the machine was busy with other builds.

Tests

StatisticsTest (new, database/src/test/java/io/hstore/db/stats):

• aSmallStoreGetsStatisticsFromItsFirstChanges: a fresh store has no refresh after 4,500 changes, refreshes once 10,800 changes are in, and catalog() then returns that refresh without scanning again.
• aLargeStoreWaitsForATenthOfItsSizeToChange: a store of 450,000 atoms and memberships takes 45,000 changes without refreshing, then refreshes exactly once when it goes past that. On main it refreshes twice in the same window.
• aBulkLoadRefreshesLogarithmicallyOften: 1,350,000 changes in commits of 9,000 give 28 refreshes. The test allows at most 38 (10 plus one per 10% growth after 100,000) and fewer than a quarter of changes / 10,000. On main the same load gives 75.

With the threshold line reverted, the second and third tests fail and the first still passes. The first test is there to make sure small stores keep getting statistics quickly. To let the tests see the subscriber, Statistics has two package-private methods, refreshes() and acknowledged().

./mvnw install passes: 179 engine, 38 database and 15 server tests.

Docs

docs/database/views-signals-stats.md explains the new rule and what it costs. docs/database/planner.md and docs/storage/change-feed.md have the updated one-line summaries.