Repository navigation
feat(db): a compact seforim.db, 2.5x smaller and faster to read on Android - #147
Draft
kdroidFilter wants to merge 2 commits into
Draft
kdroidFilter wants to merge 2 commits into
kdroidFilter wants to merge 2 commits into
Conversation
The shipped database is no longer the generator's: :packaging:compactDb rewrites it (8,022 MB to 3,382 MB on the current corpus), every line reading back byte for byte. - dao: line.content is BLOB AS String through LineTextCodec: raw deflate with a dictionary of the corpus' own text (3.1x on lines of 750 bytes, where deflate alone gets 2.5x), behind a two-byte header; plain UTF-8 otherwise (what the generator and older databases store). java.util.zip on the JVM and Android, the system's zlib on iOS. - The dictionary lives in line_codec_dict and in the packaging's resources, the same for every release, so deltas carry texts that read with the client's; the logical hash covers it, LuceneUpdater decodes the texts it reindexes. - Indexes: idx_line_book_index covers charCount (a book's scrollbar no longer reads its texts), idx_line_toc is partial, and the unused ones go (idx_line_heref, idx_link_source_line, idx_link_type, idx_toc_text). - Pages written once and full, 4 KiB; SQLite's cache and map sized per platform (Android: 16 MB, no map). - PatchDbSchema.CURRENT_VERSION 2: clients with a version 1 database take the full bundle. - The release workflow compacts the database before packaging it and producing the patch fan. - DbReadWorkload, run on the desktop (DbReadBenchmark) and on Android (AndroidDbReadBenchmark): 2 to 12x fewer bytes read, opening a book 1.5 to 2x faster cold, a line about 5 us more to decode on ART (COMPACT_DB.md). Signed-off-by: Elie Gambache <elyahou.hadass@gmail.com>
…t database reads faster than the old one Deflate cost about 5 us a line on ART (JNI, the dictionary's install, the inflate, the UTF-8): scrolling read slower than plain text. The texts are now one byte per Hebrew letter, nikud or taam, compressed by LZ77 against 60 KiB of the corpus (LZ4's block format): 3.1x, as much as deflate, decoded by copies only, in common Kotlin (no zlib, no platform code; dao compiles for iOS on Linux). - 16 KiB pages (the pageSize option was lost before: PRAGMA page_size now holds through a VACUUM): a page of the reader is one read on flash. 8,022 MB to 3,253 MB (2.47x), compacted in 70 s instead of 250. - Android 10 (emulator), cold: opening books 2x faster, jumps 1.3x, scrolling 1.1x, random lines 1.2x, commentaries 1.5x; warm as fast or faster. A simulated entry-level phone (128 MB page cache, eMMC latency): opening books 10x faster, scrolling 2.4x. - CompactDbVerification checks every line against the generator's (7,019,636, no difference); a delta between two compact releases carries only the changed lines and verifies. - The benchmark adds a sequential scroll; the old phase is a jump to random books. Signed-off-by: Elie Gambache <elyahou.hadass@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The shipped
seforim.dbis no longer the generator's::packaging:compactDbrewrites it in about 70 s, 8,022 MB → 3,253 MB (2.47x), and it reads faster on Android. Details and measurements inCOMPACT_DB.md.What changes
LineTextCodec, dao, common Kotlin). Each Hebrew letter, nikud or taam takes one byte instead of two, then LZ77 (LZ4 block format) compresses against a 60 KiB dictionary of the corpus: 3.1x. Decoding is copies only, with no JNI, no zlib and no platform code.contentisBLOB AS Stringbehind SQLDelight's adapter, so the queries don't change and plain-text databases (the generator's, older ones) still read.line_codec_dictand in the packaging's resources, so deltas between compact releases stay valid. The logical hash covers it, andLuceneUpdaterdecodes the texts it reindexes.idx_line_book_indexcoverscharCount, so a book's scrollbar no longer reads all its texts.idx_line_tocis partial. The unusedidx_line_heref,idx_link_source_line,idx_link_typeandidx_toc_textare dropped.ANALYZE.SqliteTuning): Android gets a 16 MB cache and no mmap; the desktop is unchanged.PatchDbSchema.CURRENT_VERSION= 2: a client with a v1 database takes the full bundle.Results (cold / warm, ms)
What got slower:
Verified
CompactDbVerification, plus an independent decoder on a sample).daocompiles for iOS (iosArm64,iosSimulatorArm64). Not run on iOS.Before shipping
LineTextCodeccan't read a compact database, so the release needs a minimum app version.