Repository navigation
odb upgrades - #2853
Draft
Sebastian Thiel (Byron) wants to merge 34 commits into
Draft
odb upgrades#2853Sebastian Thiel (Byron) wants to merge 34 commits into
Sebastian Thiel (Byron) wants to merge 34 commits into
Conversation
Sebastian Thiel (Byron)
changed the title
Sebastian Thiel (Byron)
force-pushed
the
odb-parallelism
branch
8 times, most recently
from
July 31, 2026 11:15
d106dfd to
af4b335
Compare
Sebastian Thiel (Byron)
force-pushed
the
odb-parallelism
branch
from
August 12, 2026 06:35
af4b335 to
924d9de
Compare
Merged
3 tasks done
Sebastian Thiel (Byron)
force-pushed
the
odb-parallelism
branch
8 times, most recently
from
August 19, 2026 07:46
39938c4 to
6d74a08
Compare
Sebastian Thiel (Byron)
changed the title
Sebastian Thiel (Byron)
force-pushed
the
odb-parallelism
branch
5 times, most recently
from
August 27, 2026 12:35
feb9527 to
c990697
Compare
Sebastian Thiel (Byron)
force-pushed
the
odb-parallelism
branch
from
August 31, 2026 07:14
c990697 to
3e23dc5
Compare
Sebastian Thiel (Byron)
added a commit
that referenced
this pull request
Sep 6, 2026
<!-- agent --> `core.autocrlf=input` or `text=auto` can make worktree diff filtering look up a null resource ID to inspect indexed line endings. These guaranteed misses can trigger repeated pack-directory refreshes until #2853 is merged. Treat null IDs as a missing index object before consulting the database. Clarify the worktree resource contract and add a regression test covering both Git conversion modes and preservation of indexed CRLF content for real IDs. Assisted-by: GPT 6.0 Co-authored-by: GPT 6.0 <codex@openai.com>
Sebastian Thiel (Byron)
force-pushed
the
odb-parallelism
branch
from
September 7, 2026 03:46
3e23dc5 to
92ef2d8
Compare
A pack that was temporarily inaccessible stayed unreadable after access recovered because chmod does not change its length or modification time. Only content-dependent errors can safely be cached using that metadata. Retry native I/O failures while preserving cached malformed-file errors. Validation: a deterministic loader regression failed before the change and now passes for raw and wrapped permission, interruption, would-block, and other I/O failures. All dynamic types unit tests pass.
The fixed-size slot mode is a hard upper bound: once all slots are occupied, new disk state yields InsufficientSlots. Calling it Given obscures that behavior, especially now that a separate mode will be able to grow. Rename the public variant and update all workspace callers and documentation. This commit contains only the breaking vocabulary change so downstream adaptation is easy to identify.
Opening a store should not require listing every pack merely to size a fixed slot map, and later maintenance should not fail only because the repository outgrew its opening-time estimate. Add a Growable slot mode whose initial allocation performs no disk scan. Grow by publishing copy-on-write snapshots of stable slot pointers under the existing structural lock, so readers remain lock-free and all prior slot IDs stay valid. AsNeededByDiskState also gains growth after its initial estimate, while Limit retains the former hard-cap behavior. Ordinary tests cover growth past the initial allocation with shared and stable handles. Adding `Slots::Growable` breaks downstream exhaustive matches on `Slots`.
Publish the slot index and its backing resource nodes through one catalog so handles cannot combine refresh generations or lazily load a pack through a reassigned slot. Snapshots and stable locations retain the exact index and pack resources they observed, without Store-local PackId encoding or artificial slot/member bit limits. Recover republished indices and packs only when their on-disk state actually changed, preserve loaded MIDX packs across redundant loads, and replace rewritten MIDX resources at their logical slot even at a configured limit. Add test-only synchronization hooks and make contending index loads, pack loads, and refreshes share their success or error outcome. This prevents duplicate I/O and keeps failed maintenance transitions from partially replacing usable state. CI's `test-fast (ubuntu-24.04-arm)` and `test-32bit (arm32v7)` jobs also compile `gix` with the ODB's `test-support` feature through workspace feature unification. After the path-reopening API arrived on `main`, its explicit `gix_odb::store::init::Options` initializer failed with `E0063` because it omitted the feature-gated `debug` field. Use the same default-options tail as repository opening, with a targeted Clippy allowance when the field is absent, so both feature combinations compile without enabling test hooks. Validation: the three `repository::location` tests pass with `gix-odb/test-support` enabled, and default-feature `gix` Clippy passes with warnings denied.
Parallel pack copying calls `entry_by_location()` once per packed object. Locking and scanning clone-family retention first serialized workers even when their local snapshots already held the requested pack. Search the local snapshot first and consult shared retention only when the pack is absent there. The selected `Arc` continues to pin the mapped bytes. Validation: the lock-held regression timed out before this change and now passes for standalone packs, multi-pack indices, and cloned handles. Existing stable-clone coverage still passes after pack/index removal.
A larger malformed index could stop object lookup before a later healthy index was searched. Replaying its cached error also prevented recovery of other failed indices and discovery of newly published packs or alternates. Defer index errors while untried or repaired indices can produce another snapshot. Recheck publication after contention, and allow a strict refresh before returning an error. Misses and full enumeration still report the failure when no usable change remains. Validation: the new regressions failed before the fix and now pass for object, header, location, and containment lookups; cloned handles; repaired indices; and newly added packs and alternates. The fixture also confirms Git reads the healthy blob. Existing restoration and failed-loader contention coverage passes.
A present but unreadable pack reported a retry without changing its failed state. Strict missing-object lookups then published a new catalog on every scan and never finished. Only a missing file that reappears transitions to unloaded and requires catalog publication. Failed files retain the loader's own content-change and transient-I/O recovery rules. Validation: the failed-state regression failed before this change and now passes with all dynamic file-state unit tests.
Pack creation enables stable handles before cloning workers. Every packed hit then locked the same clone-family retention collection, serializing otherwise independent warm lookups. Remember retention beside each snapshot's pack and skip both the shared mutex and `Arc` clone on repeated hits. New snapshots retain on first access, and clone-family ownership continues to keep deleted packs available. Keep these flags inside index snapshots so `Handle` and `Repository` sizes do not change. Validation: a regression holding the retention mutex failed before this change and now completes warm object and location lookups for standalone packs, multi-pack indices, and cloned handles. Both retention unit tests pass.
Different pack paths can have the same `CRC32`. Reusing that path-derived ID made stable locations select another pack's bytes or enumerate its objects. Assign monotonically increasing IDs at successful pack open, shared by all handles and retained across catalog generations. Refuse to wrap at exhaustion. Validation: two valid one-blob packs with colliding path CRCs reproduced the failure. Their locations now remain distinct and return the correct packed bytes and object IDs for both `SHA-1` and `SHA-256`. The exhaustion regression also passes and verifies repeated attempts cannot reuse an ID.
Without parallelism, `Cache::into_arc()` rebuilds its `Rc`-backed store. Store-local pack IDs restart at zero, so retaining the old decoded-pack cache can return bytes from another pack at the same delta offset. Recreate that cache through its existing constructor, as cloning already does. Preserve the cache policy and the object cache, which is keyed by object ID. Conversion of an already `Arc`-backed store remains a no-op. Validation: two real delta packs with matching offsets returned the first blob's bytes for the second blob before the fix. The regression now passes with parallelism both disabled and enabled.
An automatically sized store accepts a `usize` minimum. An unallocatable minimum reached `Vec` collection and panicked with capacity overflow despite `Store::at_opts()` returning `io::Result`. Reserve the slot vector fallibly before constructing its entries, returning the allocation error to the caller. Validation: the public-API regression with `minimum: usize::MAX` reproduced the panic before the fix and now receives an error using a read-only fixture.
After maintenance removes a stable handle's index, `pack_offsets_and_oid()` falls back to the clone-family retention collection. Its iterator previously kept that shared mutex locked while collecting every object in the index, blocking workers that needed to retain another pack. Clone the selected lookup under the lock, then enumerate after the guard has dropped. The clone pins the index and pack throughout collection. Validation: the existing stable-clone regression passes for standalone and multi-pack indices, including lookup and enumeration after maintenance.
The Ubuntu `test-fast` job in PR #2853 failed `session_is_private_to_its_worktree_hidden_from_history_and_retained_by_gc` with “could not read the saved rebase”: the session commit was reported missing after `git gc --prune=now` moved loose objects into a pack. The fixture retained its pre-GC object database across that external mutation. The new one-second object refresh window can suppress the immediate scan for the newly created pack. Drop and reopen the isolated repository around GC, matching neighboring GC tests and Tix's command lifecycle. Keep all assertions that the saved session and undo objects survive GC and that the continuation completes. Validation: reproduced the failure locally, then verified the repaired regression and all eight session tests with `GIX_TEST_IGNORE_ARCHIVES=1 cargo test -p gix-tix --features sha1 --lib edit::rebase::session::tests`. `cargo fmt --all -- --check` passed. The ARM32 CI job failed on the same test and missing object.
Add RefreshMode::AfterDuration as a shared gix-odb freshness window. Repeated misses avoid rescanning packs and alternates until the last successful post-initialization refresh reaches the configured age, while failed refreshes, known missing packs, and explicit stale notification remain immediately retryable. Configure gix object handles with a one-second default and expose gitoxide.objects.refreshAfter in milliseconds. Fetch invalidates freshness after publishing a pack, configuration reloads update active handles, and zero restores strict refresh-on-every-miss behavior. Use the test-support clock for deterministic deadline and contention coverage, keep feature-unified workspace builds and Repository size guards valid with the added configuration, and make the minimal CI dependency assertion portable to images without ripgrep. Adding `RefreshMode::AfterDuration` breaks downstream exhaustive matches on `RefreshMode`.
A strict snapshot commit that changes `gitoxide.objects.cacheLimit` and supplies an invalid `gitoxide.objects.refreshAfter` must leave the live repository and its cached settings unchanged. Previously the rejected snapshot updated the cached limit, so retrying the same limit with a valid refresh duration skipped object-cache setup. Parse all fallible values before applying settings or clearing caches. The public regression checks rejected settings on a clone and successful cache setup when the corrected snapshot is committed. Validation: the regression failed before the fix, and both object-refresh integration tests pass with `max-performance` enabled.
Repairing a malformed index without changing its length or modification time kept replaying the saved error even after `mark_disk_state_stale()`. Clear failed-file metadata under the current resource locks so the next lookup retries those files, while preserving loaded mappings. Handles may retain an older MIDX resource node after maintenance replaces it. Direct unloaded-pack lookups to the current catalog before consulting failures cached in a retired node. Regression tests repair standalone indices and packs, MIDX packs, and a failed pack referenced by a retired MIDX. They preserve file metadata, verify the repair with Git, and verify the returned object's hash. The tests fail without the corresponding fixes and pass afterward.
Add one deterministic single-threaded opcode VM for multiple handles, component-wise pack publication and removal, MIDX replacement, alternates, malformed resources, loose-object lifecycle, slot pressure, stable locations, iteration, prefix operations, and refresh policies. Give every opcode an explicit reproducible outcome and validate queries, shared-store checkpoints, retained locations, refresh counts, and fixture convergence against the generated disk model. Deterministic time covers strict, duration-based, and never-refresh behavior without wall-clock races. Keep readable corpus seeds for interrupted maintenance, load recovery, stale snapshots, cloned handles, MIDX transitions, growable stores, and minimized regressions. Use `gix_error::Error` for query and write error classification, matching the public ODB APIs, and raise native iterator errors into the same type. The old `Exn` classifier signatures caused 13 `E0308` mismatches in both the `test` job's standalone fuzz-workspace check and CIFuzz's `Build Fuzzers` step. Validation: `cargo check --manifest-path gix-odb/fuzz/Cargo.toml --all-targets` and `cargo fmt --manifest-path gix-odb/fuzz/Cargo.toml --all -- --check`.
Extend the pre-change dynamic-store baseline with scan-free growable opening, first-growth cost, fresh and expired duration policies, and post-publication discovery through an expired freshness window. Retain the common strict and never-refresh benchmark groups unchanged so Criterion results remain directly comparable across the implementation boundary, while the additional cases measure behavior that did not exist in the baseline API.
Retain object IDs during the serial first pass so extra-header-lookup provides a one-thread baseline instead of rejecting the invocation. Process the cold second pass in 1,000-ID chunks, matching the first pass. This replaces per-object sequentially-consistent work claims and progress updates with one claim and update per chunk while preserving independent ODB handles per worker.
The extra-header-lookup smoke test used one loose blob, so it did not cover cold packed-header reads or distributing several chunks among workers. Move this public-API test into `gitoxide-core/tests/` and use a read-only fixture with 2,001 packed blobs: two full 1,000-object chunks and a partial tail. Verify packed and blob counts and completion of the cold pass with one and two threads. Open an isolated repository separately for each case. Validation: `GIX_TEST_IGNORE_ARCHIVES=1 cargo test -p gitoxide-core --test odb` passes for both thread limits.
An index loader previously claimed the next slot before incrementing the active-load count. A contender exhausting the slots in that window could conclude that nothing was loading and return before the index was published. Register first, retain the guard for a successful claimant, and drop a failed claimant's own registration before waiting. Extend the existing test hook with the active count and coordinate two cold handles to enforce this ordering without timing assumptions.
A handle could verify its snapshot before blocking behind another index loader, then let load_next_index establish its change baseline only after that loader published. This made the loader report no change even though the caller's snapshot was stale. Recheck the caller's marker after an exhausted load attempt and collect current state before applying miss-refresh policy. Exercise the lock handoff deterministically with two indices and two handles.
Allow the dynamic ODB scenario suite to fan each shared object or miss assertion out across synchronized handle clones. This reuses existing state-change coverage while exercising lazy index, pack, and refresh transitions under shared-store contention. Keep serial behavior as the default and select the worker count through GIX_ODB_TEST_THREADS so CI or local stress runs can control resource use.
A stale handle could enter refresh while another handle was publishing a new catalog. Refresh coalescing compared shared catalog snapshots and could return no update even though the caller's own snapshot predated the published state. Recheck the caller marker whenever a coalesced refresh returns no snapshot. Add a deterministic two-handle regression that holds a waiter between observing the old completion count and snapshotting the catalog, then lets the competing refresh publish before the waiter proceeds.
Cloning workers immediately before an assertion gives them snapshots of the current shared Store, leaving only the original handle genuinely stale. Keep reusable cohorts created before disk changes so every synchronized worker carries its own pre-mutation snapshot through later lookups. Apply persistent cohorts to pack publication and removal, multi-index rewrites, alternate changes, and dynamic slot growth. The same tests remain serial by default and use GIX_ODB_TEST_THREADS to select the cohort size.
Use deterministic Store synchronization points to remove a pack while handles contend for its slot, rewrite a multi-pack index while a refresh is paused, and resolve an old stable location while a grown catalog is being published. These schedules model maintenance transitions without relying on scheduler luck. Bound synchronization waits and assert that pack loads and refreshes complete once, publish a coherent catalog, and preserve stable locations.
Drive find, header, exact-prefix, iteration, miss, and stable-location operations from distinct pre-mutation handles after publishing a new pack. This exercises shared refresh and loading through different public paths instead of making every worker repeat one lookup. Use a bounded result channel for liveness and Store metrics to require one coalesced refresh for the known disk mutation. Iteration asserts its documented current-catalog snapshot semantics while lookup operations discover the new pack.
Provide a dedicated Just recipe that exercises the dynamic ODB scenario suite with eight persistent or synchronized handles. Keep the test harness itself single-threaded so each scenario controls its filesystem mutations and Store interleavings while applying real intra-scenario contention. Have CI call the reproducible recipe and use the workflow-native five-minute step timeout so retry loops and deadlocks fail without adding a platform-specific command to the Justfile.
A lazy-initialization waiter could observe the old refresh completion count and then snapshot the catalog published by the active initializer. Refresh coalescing consequently returned no update because its two shared catalog pointers matched, even though the caller still held the uninitialized handle snapshot. Reconcile a missing consolidation result against the caller marker for initialization as well as later refreshes. Add a deterministic two-handle regression that holds the waiter between reading completion and sampling the newly published catalog.
`contains()` discarded `load_one_index()` failures and returned `false`, allowing callers to mistake incomplete index coverage for an absent object. This defeats the dynamic store's deferred-error handling and hides broken indices during duplicate checks, fetch, and connectivity checks. Return `Result<bool>` from `gix_pack::Find::contains()`, `gix_object::Exists::exists()`, `exists_without_refresh()`, and `gix::Repository::has_object()`. Propagate errors through their wrappers, callers, and the fuzz target while retaining in-memory hits and the porcelain's synthetic empty tree. Clear incomplete `gix-fsck` traversal state after a failed presence check so retrying cannot silently succeed. Extend regression coverage for healthy indices alongside malformed ones, true absence, disabled refresh, full-length prefixes, in-memory objects, write checks, and repeated connectivity checks. Validation: - `cargo check --workspace --all-targets` passed. - `gix-odb`, `gix-pack`, and `gix-fsck` tests passed with parallelism enabled, including the threaded ODB scenarios; SHA-256 ODB error tests also passed. - Focused `gix` object and fetch tests passed, as did both blocking and async `gix-protocol` suites and the async-only `gix` build check. - The dynamic-store fuzz target compiled; formatting and diff checks passed. - The broader `gix-object` run passed 171 tests, but four signing tests failed during GPG-agent setup. Full repository QA was not run. CI follow-up: native tests on Linux, macOS and Windows, and the Linux feature-test job failed in `object_checks_and_writes_preserve_index_load_errors` during fixture setup with permission denied. Git-generated pack indices can be read-only, so writing corruption directly into the existing `.idx` file never reached the error-propagation assertions. Write a replacement file and use `gix_testtools::rename` to replace the read-only index before any lookup maps it. This also handles Windows read-only destination flags. Follow-up validation: - The focused regression passes with SHA-1 and SHA-256 fixtures. - `cargo fmt --all --check` and `git diff --check` pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tasks
Performance
Multi-threaded it's a tad slower wallclock wise, 1-3% in a parallel lookup test, but that's not going to be a problem IRL where object-decoding dominates.
Single-threaded, there seems to be a 5% regression, but I think that's Ok for now. It's unclear where it's coming from as well.
Scaling report by Codex
Dynamic ODB thread-scaling experiment
Measured on 2026-08-13 with commit
9d8ae4941ae4a420c87829ee08403c79b2acf686(
gix v0.56.0-242-gea600f8c04) on a machine exposing 16 logical CPUs.The repository was the bare Linux kernel repository at
~/dev/git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux, with a 2.3 GiBobject database, two pack files, and about 10.9 million objects.
Method
The current release binary was built first. A 16-thread warm-up populated the
filesystem cache, followed by one separate process for each measured thread
count:
--extra-header-lookupperforms the normal statistics pass, reopens therepository to obtain a cold ODB, and looks up all known object headers again.
Speedup and efficiency use the end-to-end wall time of the one-thread run as
their baseline.
Results
Each result is one separate warmed run over 10.9M objects on a 16-logical-CPU machine.
¹ Not a valid performance improvement because the PR run failed.
Interpretation
Both implementations scale nearly identically through eight threads. The PR ranges from effectively unchanged to 5% faster, which is too small to treat as significant from single samples. Eight threads is optimal for both:
Beyond eight threads, both suffer a substantial contention or scheduling cliff in the cold second pass. The first pass continues scaling to 16 threads, while cold lookup slows from roughly 1.6 seconds to 3 seconds. User CPU also rises sharply:
The critical difference is correctness. Main completed successfully at every thread count. The PR failed at both 16 and 32 threads, reporting IDs produced by the first pass as missing from the freshly opened ODB. A separate 16-thread PR warm-up also failed, with 14 missing objects instead of 11, demonstrating nondeterminism.
Therefore:
Purpose
This PR establishes the correctness and performance baseline for upcoming work on the multi-handle, multi-threaded dynamic object database. It intentionally changes test infrastructure rather than production ODB behavior: the aim is to make filesystem transitions, refresh behavior, slot pressure, and handle interactions reproducible before changing the implementation.
The infrastructure is intended to support this sequence of follow-up work:
Refresh::Never.What this adds
Shared ODB scenarios
A
gix-testtoolsfixture script builds a deterministic catalog of valid SHA-1 and SHA-256 packs and indices. A utility module undergix-odb/tests/copies and mutates that catalog for tests, benchmarks, and fuzzing without introducing another support crate.The shared vocabulary can:
This keeps all three kinds of coverage grounded in the same well-known repositories and object IDs.
Deterministic correctness coverage
The integration tests exercise multiple live handles in deliberately interleaved, single-threaded sequences while:
The existing threaded packed-object smoke test also now always uses at least two workers and verifies that every enumerated object is actually found.
Benchmarks
Criterion benchmarks provide baselines for:
Refresh::Never;These should make regressions and improvements in refresh suppression, prefix lookup, startup behavior, and slot over-commitment visible.
Deterministic fuzz VM
A single-threaded bytecode VM interleaves operations across several handles with disk mutations. It covers lookup, header access, iteration, prefix queries, refresh modes, pack-component transitions, MIDX transitions, alternates, malformed indices, and slot pressure.
The seed corpus includes standalone packs, MIDX use, alternates, malformed recovery, slot pressure, pack replacement, MIDX expiration, and interrupted maintenance. Maintenance states can expose markers or new components before old packs are removed, remove packs while a MIDX still references them, and abandon or reverse transitions.
The VM checks returned object hashes, header/data agreement, fresh-store equivalence at valid checkpoints, and existing handle metrics. Store statistics can be used or extended further where they help characterize later correctness or performance changes.
Verification
cargo test -p gix-odb --all-features --lib --test odbcargo clippy -p gix-odb --all-features --tests --benches -- -D warnings