Enhancement request, based on field experience pulling 15 years of Schema.STATISTICS for 8 continuous symbols on GLBX.MDP3 (v0.2.0).
Motivation
Each get_range request for continuous-symbol queries carries ~25-30s of fixed server-side latency regardless of payload size. That interacts badly with both obvious client strategies:
- One big request: exceeds the 100s default read timeout (separate issue) and is all-or-nothing on failure.
- Sequential chunking: 8 symbols x 16 calendar years = 128 requests x ~28s = ~1 hour of accumulated fixed latency for ~66MB of data.
What actually worked: calendar-year chunks fetched concurrently (8 @async tasks gated by a Base.Semaphore) — the same 128 requests complete in a few minutes, and per-chunk failures are retryable without losing the rest. The client is already documented as safe to share across tasks, and the built-in 429 retry-with-backoff handled rate-limit brushes.
Proposal
A get_range_chunked (or kwargs on get_range: chunk = Year(1), concurrency = 8) that:
- splits
[start_dt, end_dt) into chunks,
- fetches them concurrently under a semaphore,
- concatenates records in time order,
- warns-and-continues on per-chunk failure (collecting failed chunk ranges in the return for retry).
Also worth considering: an encoding = "csv" escape hatch on get_range returning a raw table — it is a useful fallback when a binary decode path has problems (it is how I worked around the StatMsg layout bug reported in DatabentoBinaryEncoding.jl), and trivially supports map_symbols/pretty_px for interactive use.
A reference implementation of the chunk+concurrency pattern is in my creditspreadsresearch repo (part1/download_statistics.jl).
Enhancement request, based on field experience pulling 15 years of
Schema.STATISTICSfor 8 continuous symbols onGLBX.MDP3(v0.2.0).Motivation
Each
get_rangerequest for continuous-symbol queries carries ~25-30s of fixed server-side latency regardless of payload size. That interacts badly with both obvious client strategies:What actually worked: calendar-year chunks fetched concurrently (8
@asynctasks gated by aBase.Semaphore) — the same 128 requests complete in a few minutes, and per-chunk failures are retryable without losing the rest. The client is already documented as safe to share across tasks, and the built-in 429 retry-with-backoff handled rate-limit brushes.Proposal
A
get_range_chunked(or kwargs onget_range:chunk = Year(1),concurrency = 8) that:[start_dt, end_dt)into chunks,Also worth considering: an
encoding = "csv"escape hatch onget_rangereturning a raw table — it is a useful fallback when a binary decode path has problems (it is how I worked around the StatMsg layout bug reported in DatabentoBinaryEncoding.jl), and trivially supportsmap_symbols/pretty_pxfor interactive use.A reference implementation of the chunk+concurrency pattern is in my creditspreadsresearch repo (
part1/download_statistics.jl).