CryptoTrader

CryptoTrader is a config-driven Python workspace for durable market-data capture and trading research. Its implemented path records provider payloads as the source of truth and derives reproducible normalized datasets from sealed raw files.

Implemented System

• Captures live market data from Pyth and Aster.
• Captures TradingView webhook alerts as raw events.
• Runs an installed recorder as a systemd-managed service with a service-owned Unix control socket.
• Stores append-only compressed raw records in a route-based filesystem layout.
• Emits runtime health manifests and route-aware raw-data quality reports.
• Implements phase-1 normalization from sealed raw recorder data into deterministic partitioned Parquet datasets for seven tables:
  • pyth_prices
  • trades
  • book_ticker
  • mark_price
  • l2_partial_depth
  • l2_diff_depth
  • l2_depth_snapshots

CryptoTrader is a market-data and research system, not a live-trading system: it does not place orders, manage positions, or use private trading endpoints.

Sources and Responsibilities

Source or component	Current responsibility
Pyth	Records Hermes SSE reference-price updates.
Aster	Records non-depth market streams, depth streams, and periodic REST depth snapshots.
TradingView	Receives webhook requests and records their raw alert events. Alert analysis and downstream signal workflows are not implemented.
Raw recorder	Wraps, timestamps, writes, rotates, and seals provider records without changing their source payloads.
Normalizer	Reads sealed raw files, writes provisional chunks and canonical Parquet partitions, and records ledger and normalization-manifest state.

Architecture and Data Boundaries

external sources
  -> raw/source capture
  -> normalized datasets

The implemented path covers raw capture and normalization. Raw/source data remains reconstructable; normalized data is derived and reproducible from sealed raw files. Feature generation, replay, backtesting, model workflows, alert analysis, and downstream signal workflows are not implemented runtime pipelines.

Planned (Not Implemented Yet)

• local order-book reconstruction;
• production feature generation;
• shared live/replay state reducers;
• replay/backtest runtime;
• primitives/episodes/cases layers;
• operational Copilot review workflow;
• alert notifications;
• app API, PWA, Web Push, social-news enrichment, and trade execution.

Raw records use route-local files:

<data-root>/raw/<source>/<transport>/<source_symbol>/<stream>/date=YYYY-MM-DD/hour=HH/
  part-<segment_start>-<run_id>.jsonl.zst.open
  part-<segment_start>-<segment_end>-<run_id>.jsonl.zst

Active .jsonl.zst.open files are writer-owned and incomplete. Sealed .jsonl.zst files are the validation and read target. The complete filesystem reference, including normalized outputs, chunks, ledger, manifests, and staging directories, is in the data-layout reference.

runtime.data_root selects storage for checkout commands and direct config-consuming CLI invocations. The committed local configuration uses ./data. The installed worker is separate: systemd StateDirectory=market-recorder/%i passes its effective --data-root, /var/lib/market-recorder/<instance>, to service-worker. The installed production configuration and worker state normally both use /var/lib/market-recorder/production, but the unit—not a YAML edit and restart—owns the worker's storage path. The installer provisions derived normalization directories there before the unit uses them.

Repository Map

config/                 Example runtime and source configuration
docs/                   Architecture, provider, and operations documentation
ops/                    Deployment and operational assets
scripts/                Utility entrypoints
src/market_recorder/    Recorder package and provider adapters
tests/                  Unit tests and integration-test scaffolding

Key package areas:

• src/market_recorder/config.py loads and validates configuration.
• src/market_recorder/runtime.py owns the shared runtime lifecycle.
• src/market_recorder/storage/ owns raw paths, writing, and validation.
• src/market_recorder/sources/ contains market-data provider adapters.
• src/market_recorder/alerts/ contains raw alert ingestion.
• src/market_recorder/service.py supervises unattended recording and writes runtime health manifests.
• src/market_recorder/normalize/ contains the sealed-raw-to-Parquet normalizer.
• ops/install/ and ops/systemd/ contain installed-service assets.

Focused References

• Agent guidebook: implementation map and subsystem roles.
• Development guide: Dev Core setup and truthful local validation baseline.
• Schema reference: raw-envelope and normalized-table contracts.
• Data-layout reference: current data-root artifacts and their ownership.
• Pyth provider reference, Aster provider reference, and TradingView provider reference: source integration details.
• Operator guide: canonical internal installation, lifecycle, diagnosis, data-handling, and normalization procedures.
• Deployment notes and Monitoring notes: compatibility landing pages for the operator guide.
• Implementation status: confirmed subsystem status.