Skip to content

feat(embeddings): batch chunks across files and run API embedding requests in parallel - #175

Merged
spelech merged 3 commits into
mainfrom
feat/parallel-embedding-171
Nov 18, 2026
Merged

spelech merged 3 commits into
mainfrom
feat/parallel-embedding-171

Conversation

@spelech

@spelech spelech commented Nov 18, 2026

Copy link
Copy Markdown
Owner

Closes #171

Summary

  • Collects unembedded chunks across files per sync batch (process_file_content(..., embed=False) + embed_points_across_files) in git_syncer.py and local_syncer.py so API requests batch chunks across files rather than making one sequential HTTP round-trip per file.
  • Dispatches API embedding slices in parallel via ThreadPoolExecutor controlled by EMBEDDING_API_CONCURRENCY (default 4), slicing requests by both EMBEDDING_BATCH_SIZE (default 128 for API, up to 256) and estimated token count (EMBEDDING_MAX_TOKENS_PER_REQUEST, default 100000).
  • Handles HTTP 429 rate limits (Retry-After / retry-after-ms) with AIMD concurrency scaling, a clear rate-limit warning log, optional EMBEDDING_TPM_LIMIT pacing, and sync status telemetry (embedding_chunks_per_sec, rate_limit_wait_seconds, rate_limit_hits).
  • Swaps _openai_client atomically in init_embeddings and stops startup seeding from overriding .env EMBEDDING_BATCH_SIZE / EMBEDDING_NUM_THREADS.

@spelech
spelech merged commit 9a37f4a into main Nov 18, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Embedding API calls run one at a time; large repos index slowly

1 participant