Skip to content

ingest_reservoir: re-read unsettled days, keep the venue's minutes, --keep-going, atomic writes - #27

Merged
Thykof merged 5 commits into
mainfrom
fix/reservoir-unsettled
Nov 9, 2026
Merged

Thykof merged 5 commits into
mainfrom
fix/reservoir-unsettled

Conversation

@Thykof

@Thykof Thykof commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Stacked on #26.

Why

scripts/ingest_reservoir.py is how a store gets Hyperliquid history beyond the venue's 5,000
minutes: the campaign's store, and from campaign-7 a live miner's store before its first start (see
docs/deploy-a-champion.md). Several things get in the way.

  • It skips any day already on disk, settled or not. A partition the venue top-up wrote while its
    day was still running carries is_final false. If no later top-up completes it within the
    venue's ~3.5-day reach, it stays partial for good: the venue no longer serves the day, and the
    ingest never replaces it. MinutePriceStore already re-reads such a partition; the ingest did not.
  • Re-reading it must not lose the venue's minutes. When no previous day is stored, the archive
    frame has no close to carry into the minutes before the coin's first archive trade. Overwriting
    the venue's partition would turn those minutes NaN, and the day, now settled, would never be read
    again.
  • One refused day stops the whole run. The dex-pause check refuses a day file with no trade on
    the whole dex for more than 15 minutes, and raises. That is right for the campaign's store, which
    must be complete. Seeding a live miner wants the other days anyway, with the refused one logged.
  • A stalled route holds a run for minutes per read. The S3 client had botocore's 60 s connect
    timeout and 5 attempts, so a seed run per day could hang for about 5 minutes before failing.
  • A killed run can leave a truncated partition. The partition was written in place, and an
    unreadable partition broke every reader of that day, the ingest's own skip check included.

Stacked on docs/context-days.

Merge order. This PR merges after the synth-lib stack, once a pin-bump commit to the merged
synth-lib lands on build/kimi-code-2. The import test below fails on that bump until the same
commit adapts scripts/measure_issuance_lead.py, which imports CONTEXT_MINUTES, a name the new
engine drops.

What

Change Detail
Settled skip Without --force-refresh, a stored day is skipped only when its partition's is_final is true; otherwise it is read again
Stored minutes kept write_day fills the minutes the archive leaves NaN from the partition already stored for the day
Atomic write write_day writes date=D.parquet.tmp beside the partition and os.replaces it into place
Unreadable partition A stored partition that cannot be read counts as not settled: the default run reads the day again and replaces it
Bounded S3 waits connect_timeout 10 s, read_timeout 60 s, 3 attempts (was botocore's 60 s connect and 5 attempts)
IncompleteDayError The dex-pause check raises it, a RuntimeError subclass, so the caller can tell a refused day from any other failure
--keep-going Logs refused: <file>: ... for each refused day file, goes on with the next days, and ends with the list of refused files
Default Unchanged: a refused day stops the run
--force-refresh help "re-read settled days already on disk too"
tests/test_scripts.py Imports every script under scripts/ against the pinned engine, so a name the engine drops fails CI
docs/deploy-a-champion.md The seed command passes --keep-going; what a refused day means live; the ingest skips settled days
docs/operate-a-campaign.md The Hyperliquid retention warning, corrected (below)
docs/install-and-first-run.md The same correction
CHANGELOG.md Under "Merged": the settled skip, the kept minutes, the atomic write, the bounded waits and --keep-going

A refused day writes nothing. The next day then starts from the stored previous day's close, or
NaN before its first trade when there is none, as for a day missing from the archive.

Both docs said the venue's empty answer for a day past its retention is stored settled and skipped
by every later ingest. They now say:

  • A Hyperliquid day the venue holds only in part is stored with NaN before the retention edge and
    settled, so no later ingest reads it again unless ingest_reservoir.py --force-refresh replaces
    it from the archive.
  • A day the venue no longer holds at all is stored unsettled, and ingest_reservoir.py re-reads it
    from the archive.

The .tmp file does not match the readers' date=*.parquet globs, and a later write of the same
day replaces it.

Tests

New or changed, in tests/test_ingest_reservoir.py, with the S3 reads mocked as the existing tests
do:

Test Checks
test_a_settled_day_on_disk_is_skipped a stored settled day is left as it is
test_an_unsettled_day_on_disk_is_read_again archive minutes replace stored ones; stored closes before the first archive trade stay; settled
test_a_failed_write_leaves_the_stored_partition_whole a write that fails midway leaves the stored partition readable and unchanged
test_an_unreadable_partition_is_read_again_and_replaced a truncated stored partition is read again from the archive and replaced, settled
test_a_refused_day_stops_the_run_by_default IncompleteDayError propagates, and the next day is not written
test_keep_going_logs_a_refused_day_and_ingests_the_next the refusal and the summary are printed, and the next day is written

test_a_long_pause_across_the_whole_dex_is_refused now expects IncompleteDayError and shares the
paused day file with the new tests. tests/test_scripts.py adds one import test per script, six.

Check Result
uv run black --check . 12 files unchanged
uv run pytest -q 54 passed
the ingest tests against the old skip rule (.exists()) the unsettled test fails, 23 pass
the ingest tests without the stored-minutes fill the unsettled test fails, 23 pass
the ingest tests with the in-place write the failed-write test fails, 23 pass
tests/test_scripts.py against the synth-lib PR's engine measure_issuance_lead.py fails, 5 pass
scripts/ingest_reservoir.py --help lists --keep-going

Not run: anything against the archive.

🤖 Generated with Claude Code

https://claude.ai/code/session_016kmUsyYSEMPGiwbSuvR9hh

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

Tip: disable this comment in your organization's Code Review settings.

@Thykof
Thykof added this pull request to stack #28 November 9, 2026 10:54
@Thykof
Thykof force-pushed the fix/reservoir-unsettled branch 2 times, most recently from b250aa9 to 78e364b Compare November 9, 2026 10:56
Base automatically changed from docs/context-days to main November 9, 2026 10:58
Thykof and others added 5 commits November 9, 2026 11:58
…used day

A stored day is skipped only when its partition is settled (is_final
true), as MinutePriceStore treats it, so a partition a venue top-up wrote
while its day was still running is read again from the archive instead of
staying partial. --keep-going logs a day file the dex-pause guard refuses
and goes on with the next days, for seeding a live miner's store; without
it the refusal still stops the run, now as IncompleteDayError.

Tests cover the settled skip, the unsettled re-read, the default stop and
--keep-going. deploy-a-champion.md seeds with --keep-going, and the
CHANGELOG lists both changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016kmUsyYSEMPGiwbSuvR9hh
…omically, bound S3 waits

When a stored partition is read again, the minutes the archive leaves
without a price, before the day's first trade when no previous day is
stored, keep the stored partition's candles instead of turning NaN. Each
partition is written to a .partial file beside it and renamed into place,
so a killed run never leaves a truncated one. The S3 client connects
within 10 s, reads within 60 s and tries 3 times, so a stalled route fails
the day's read quickly.

The unsettled re-read test now expects the stored closes before the first
archive trade to survive, and a new test fails a write midway. The docs
say a Hyperliquid day the venue holds only in part is stored with NaN
before the retention edge and settled, while a day it no longer holds is
stored unsettled and re-read from the archive.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016kmUsyYSEMPGiwbSuvR9hh
A parametrized test loads each script as `python <script>` would, so a
name the engine drops fails CI instead of the next run of the script.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016kmUsyYSEMPGiwbSuvR9hh
…placed

A truncated partition used to stop the run in stored_settled and in write_day.
It now counts as not settled, so the default run re-reads the day and replaces
it. The temporary file of an atomic write ends in .tmp.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016kmUsyYSEMPGiwbSuvR9hh
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016kmUsyYSEMPGiwbSuvR9hh
@Thykof
Thykof force-pushed the fix/reservoir-unsettled branch from 78e364b to 5feefe7 Compare November 9, 2026 10:58
@Thykof
Thykof merged commit d50c2ed into main Nov 9, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant