Repository navigation
fix(indexing): per-input token limit & 400 bisection, pipeline stage cancellation, and persisted sync progress (#178, #179, #180) - #181
Merged
Conversation
added 2 commits
November 20, 2026 09:24
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Resolves three indexing and embedding pipeline issues found during large-repository deployment:
Per-Input Token Cap & HTTP 400 Batch Bisection (
#178):EMBEDDING_MAX_TOKENS_PER_INPUT/max_tokens_per_inputsetting (default8000tokens) acrossconfig_store.py,embeddings.py,EmbeddingSettingsRequest,.env.example, and configuration docs.max_tokens_per_inputbefore batch slicing (truncate_text_to_token_limit) and logs the source file path.400error during batch embedding (execute_api_embeddings_batch), recursively bisects the batch slice to isolate the single bad input, logs its file path, skips only that chunk (None), filters unembedded points before vector store upsert, and continues repository synchronization without aborting the sync.Pipeline Producer Cancellation & Clean Stage Thread Shutdown (
#179):channel.is_closed(_cancelledor_error is not None) and the return value ofchannel.write(...)in_produce_parsed_files,_produce_local_files, and_stage2_coalescerso upstream stages stop immediately when any downstream stage fails or the sync is cancelled.t1) and Stage 2 (t2) threads inChannelIngestionPipeline.run()'sfinallyblock before shutting downembed_executor(cancel_futures=True) and beforesync_single_git_repocleans up the temporary clone directory, eliminating spurious[Errno 2] No such file or directoryandcannot schedule new futures after shutdownerrors.Persist-Stage Progress & Error Status Retention (
#180):processed_filesupdates and[n/total] Ingested <path>log lines from Stage 1 (_produce_parsed_files) to Stage 3/4 (_persist_parsed_batch) after vector store upsert and relational DB persistence complete.parsed_filestracking (Parsed n/total) for Stage 1 parsing progress without advancingprocessed_files.GitProgressTracker.update_stepfrom overwriting a job'sstatusback to"syncing"once the job has failed (status == "error") or been cancelled.Closes #178
Closes #179
Closes #180
Verification
tests/test_embedding_parallel_batch.py,tests/backend/test_git_syncer_progress.py, andtests/backend/test_git_progress.pycovering pre-batch token truncation, HTTP 400 batch bisection and bad-chunk skipping, producer/coalescer shutdown on persist stage failure, persist-stageprocessed_files/Ingestedcounting, anderrorstatus preservation.