Repository navigation
ingest_reservoir: re-read unsettled days, keep the venue's minutes, --keep-going, atomic writes - #27
Merged
Merged
Conversation
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.
Tip: disable this comment in your organization's Code Review settings.
Thykof
force-pushed
the
fix/reservoir-unsettled
branch
2 times, most recently
from
November 9, 2026 10:56
b250aa9 to
78e364b
Compare
…used day A stored day is skipped only when its partition is settled (is_final true), as MinutePriceStore treats it, so a partition a venue top-up wrote while its day was still running is read again from the archive instead of staying partial. --keep-going logs a day file the dex-pause guard refuses and goes on with the next days, for seeding a live miner's store; without it the refusal still stops the run, now as IncompleteDayError. Tests cover the settled skip, the unsettled re-read, the default stop and --keep-going. deploy-a-champion.md seeds with --keep-going, and the CHANGELOG lists both changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016kmUsyYSEMPGiwbSuvR9hh
…omically, bound S3 waits When a stored partition is read again, the minutes the archive leaves without a price, before the day's first trade when no previous day is stored, keep the stored partition's candles instead of turning NaN. Each partition is written to a .partial file beside it and renamed into place, so a killed run never leaves a truncated one. The S3 client connects within 10 s, reads within 60 s and tries 3 times, so a stalled route fails the day's read quickly. The unsettled re-read test now expects the stored closes before the first archive trade to survive, and a new test fails a write midway. The docs say a Hyperliquid day the venue holds only in part is stored with NaN before the retention edge and settled, while a day it no longer holds is stored unsettled and re-read from the archive. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016kmUsyYSEMPGiwbSuvR9hh
A parametrized test loads each script as `python <script>` would, so a name the engine drops fails CI instead of the next run of the script. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016kmUsyYSEMPGiwbSuvR9hh
…placed A truncated partition used to stop the run in stored_settled and in write_day. It now counts as not settled, so the default run re-reads the day and replaces it. The temporary file of an atomic write ends in .tmp. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016kmUsyYSEMPGiwbSuvR9hh
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016kmUsyYSEMPGiwbSuvR9hh
Thykof
force-pushed
the
fix/reservoir-unsettled
branch
from
November 9, 2026 10:58
78e364b to
5feefe7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #26.
Why
scripts/ingest_reservoir.pyis how a store gets Hyperliquid history beyond the venue's 5,000minutes: the campaign's store, and from campaign-7 a live miner's store before its first start (see
docs/deploy-a-champion.md). Several things get in the way.day was still running carries
is_finalfalse. If no later top-up completes it within thevenue's ~3.5-day reach, it stays partial for good: the venue no longer serves the day, and the
ingest never replaces it.
MinutePriceStorealready re-reads such a partition; the ingest did not.frame has no close to carry into the minutes before the coin's first archive trade. Overwriting
the venue's partition would turn those minutes NaN, and the day, now settled, would never be read
again.
the whole dex for more than 15 minutes, and raises. That is right for the campaign's store, which
must be complete. Seeding a live miner wants the other days anyway, with the refused one logged.
timeout and 5 attempts, so a seed run per day could hang for about 5 minutes before failing.
unreadable partition broke every reader of that day, the ingest's own skip check included.
Stacked on
docs/context-days.Merge order. This PR merges after the synth-lib stack, once a pin-bump commit to the merged
synth-lib lands on
build/kimi-code-2. The import test below fails on that bump until the samecommit adapts
scripts/measure_issuance_lead.py, which importsCONTEXT_MINUTES, a name the newengine drops.
What
--force-refresh, a stored day is skipped only when its partition'sis_finalis true; otherwise it is read againwrite_dayfills the minutes the archive leaves NaN from the partition already stored for the daywrite_daywritesdate=D.parquet.tmpbeside the partition andos.replaces it into placeconnect_timeout10 s,read_timeout60 s, 3 attempts (was botocore's 60 s connect and 5 attempts)IncompleteDayErrorRuntimeErrorsubclass, so the caller can tell a refused day from any other failure--keep-goingrefused: <file>: ...for each refused day file, goes on with the next days, and ends with the list of refused files--force-refreshhelptests/test_scripts.pyscripts/against the pinned engine, so a name the engine drops fails CIdocs/deploy-a-champion.md--keep-going; what a refused day means live; the ingest skips settled daysdocs/operate-a-campaign.mddocs/install-and-first-run.mdCHANGELOG.md--keep-goingA refused day writes nothing. The next day then starts from the stored previous day's close, or
NaN before its first trade when there is none, as for a day missing from the archive.
Both docs said the venue's empty answer for a day past its retention is stored settled and skipped
by every later ingest. They now say:
settled, so no later ingest reads it again unless
ingest_reservoir.py --force-refreshreplacesit from the archive.
ingest_reservoir.pyre-reads itfrom the archive.
The
.tmpfile does not match the readers'date=*.parquetglobs, and a later write of the sameday replaces it.
Tests
New or changed, in
tests/test_ingest_reservoir.py, with the S3 reads mocked as the existing testsdo:
test_a_settled_day_on_disk_is_skippedtest_an_unsettled_day_on_disk_is_read_againtest_a_failed_write_leaves_the_stored_partition_wholetest_an_unreadable_partition_is_read_again_and_replacedtest_a_refused_day_stops_the_run_by_defaultIncompleteDayErrorpropagates, and the next day is not writtentest_keep_going_logs_a_refused_day_and_ingests_the_nexttest_a_long_pause_across_the_whole_dex_is_refusednow expectsIncompleteDayErrorand shares thepaused day file with the new tests.
tests/test_scripts.pyadds one import test per script, six.uv run black --check .uv run pytest -q.exists())tests/test_scripts.pyagainst the synth-lib PR's enginemeasure_issuance_lead.pyfails, 5 passscripts/ingest_reservoir.py --help--keep-goingNot run: anything against the archive.
🤖 Generated with Claude Code
https://claude.ai/code/session_016kmUsyYSEMPGiwbSuvR9hh