Repository navigation
docs: benchmark report, refreshed results and README - #89
Merged
Merged
Conversation
…ability HyperGraphDB 1.4 begins every JE transaction with WRITE_NO_SYNC, which overrides the COMMIT_SYNC the adapter set on the environment. In sync mode its commits were never forced: single-update commits took about 0.1 ms while one forced write on the test machine takes about 3.8 ms. The adapter now calls Environment.flushLog(true) after every commit in sync mode.
DatasetStats prints node, hyperedge and incidence counts, the cardinality distribution and the incidence-set size distribution at scales 1 and 4, for the dataset table in the benchmark report.
Replaces the tables with runs of commit 459b8be at scale 1 (async and sync) and scale 4, both caches at 64 MiB. The analysis now covers where HyperGraphDB leads (member enumeration once the data outgrows HStore's node cache), the results that are too close to call, and what HStore spends. The setup explains the sync fix for HyperGraphDB and notes that sync results published before it measured HyperGraphDB without syncing.
The report describes both systems, the dataset, workloads and method, presents the results for three configurations, discusses them, and lists the threats to validity. Appendices hold every table, every run and the steps to reproduce. gen.py builds all tables, figures and quoted numbers from the raw results in benchmarks/report/results.
Decoding a leaf used to work out the encoded size of every key and value straight away, for inline member lists inside catalog records too. Only writes read those sizes. A frozen leaf now measures itself the first time bytes() or heapBytes() is called, and writable leaves still measure when they are created.
Codecs that write each value without reference to the others (the catalog, posting lists and row codecs) now say so with independentValues(), and their leaf pages put the length of every value in front of it. A reader can then find one record without decoding the ones before it. Entry sizes count the length, so leaves never overflow a page. Storage format goes to 8. The history compaction test sat right at the 50% live ratio, and one byte per record pushed its early segment over, so it now compacts below 75%.
Loading a catalog or posting-list leaf used to decode every record on the page, with every inline member list, to answer a lookup for one of them. A leaf whose records are framed by length now keeps the page bytes and the offset of each record, and decodes a record on first read. The decoded record is kept in the leaf, published with a compare-and-set so readers on other threads see it whole, and charged to the node cache entry that holds the leaf, so the cache still evicts to its budget as records unpack. Charges for an entry that was evicted or cleared are dropped. A packed leaf costs its page image plus its keys and offsets until records are read, so far more of the catalog fits in a given node cache.
Decoding a leaf compared the header's refers-to-pages flag with its values, which a packed leaf can't do without decoding every record. The packed leaf now keeps the flag, and TreeVerifier, which decodes every record anyway, compares the two, so hstore check still catches a header that disagrees with its contents.
A writer copying a packed leaf, or measuring it, read every record through value(), which kept each decoded record in the frozen leaf and charged the node cache for it, one shard lock per record, for a leaf the write was about to replace. Both now decode records straight into the copy or the measurement and leave the packed leaf as it was.
The latency tables show the median of each run's maximum. The report also needs the single slowest operation across all runs, so gen.py now defines it for every latency workload.
All three configurations ran again on commit 9291e18, with both engines alternating in the same window. HStore now lists a hyperedge's members 3.2x faster than HyperGraphDB at scale 4, where it had been 0.13x, and two-hop queries are 4.4x faster. The text follows the new runs. HyperGraphDB opens a crashed store faster at scale 1, the mixed writer kept its schedule in both engines under sync, and JE wrote 497 KiB per synced single commit. The discussion explains why decoding one record per lookup keeps the catalog in a 64 MiB cache.
The results tables come from the same runs as the report. The analysis now reflects scale 4, where HStore leads on member enumeration, and scale 1, where HyperGraphDB recovers from a crash faster. A new section compares allocation and bytes read per lookup before and after the change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The README and
docs/benchmarks.mdstill showed results from before the last six PRs, with unequal caches. This adds a full benchmark report as a PDF, refreshes the published results from main, and updates the README.What changes
Sync fix in the HyperGraphDB adapter. HyperGraphDB 1.4 begins every JE transaction with
WRITE_NO_SYNC, which overrides theCOMMIT_SYNCthe adapter set on the environment. Its "sync" commits took about 0.1 ms while one forced write on this machine takes about 3.8 ms, so they were never synced. The adapter now callsEnvironment.flushLog(true)after each commit in sync mode. The sync tables published before this measured HyperGraphDB without any sync.docs/benchmark-report.pdf, built frombenchmarks/report:gen.pybuilds every table, figure and quoted number from the raw JSON inbenchmarks/report/results, so nothing in it is typed by hand.docs/benchmarks.mdhas the new results, a rewritten analysis, and a setup section that explains the sync fix.README describes logged commits and compaction, and has a short performance table that links to the report.
DatasetStatsprints the dataset size and skew used in the report.Results in short
This branch includes #90, and the numbers are from that code (
9291e18), so #90 should merge first. Both caches at 64 MiB, scale 1 with 5 runs per store in each durability mode, and scale 4 with 3 runs, the two engines alternating.The laptop was busy: the load average was 3.4 to 18.3 on 10 cores. The report marks every ratio whose run ranges overlap. The ratios at scale 1 are smaller than in the first version of this PR, mostly because HyperGraphDB ran faster in this quieter window: between the two sets of runs, its scale 1 throughput rose 1.3 to 3.3 times per workload, and HStore's 0.8 to 1.8 times. Earlier alternated runs of main and #90 at scale 1 showed no slowdown in HStore.
Testing
./mvnw package: 225 tests pass, with perf(tree): decode leaf records on demand so the catalog fits in the node cache #90 merged in.-Pbenchmarksbuild passes.python3 gen.py results gen && tectonic report.texinbenchmarks/reportrebuilds the committed PDF.DatasetStatsoutput matches the dataset table.