Skip to content

feat: add endpoints to wipe the vector db efficiently - #374

Open
kyteinsky wants to merge 9 commits into
masterfrom
feat/vector-db-wipe
Open

feat: add endpoints to wipe the vector db efficiently#374
kyteinsky wants to merge 9 commits into
masterfrom
feat/vector-db-wipe

Conversation

@kyteinsky

@kyteinsky kyteinsky commented Oct 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

and some performance improvements.
it needs context chat changes to pass the tests.

related: #359 and nextcloud/context_chat#175

🤖 AI (if applicable)

  • The content of this PR was partly or fully generated using AI (N/A)

@kyteinsky
feat: /deleteAll endpoint to delete all the indexed documents from …
cb8f1e3 
…the vector DB

used by `occ context_chat:wipe`

Signed-off-by: kyteinsky <kyteinsky@gmail.com>
Assisted-by: Github Copilot:claude-opus-5-5
@kyteinsky
fix: stop bg threads earlier by waiting on THREAD_STOP_EVENT
82e5893 
Signed-off-by: kyteinsky <kyteinsky@gmail.com>
Assisted-by: Github Copilot:claude-opus-5-5
@kyteinsky
feat: stop bg threads while db wipe takes place
4cb2c5f 
Signed-off-by: kyteinsky <kyteinsky@gmail.com>
Assisted-by: Github Copilot:claude-opus-5-5
@kyteinsky
fix: do source and provider deletes within the db
0af09b7 
don't fetch the chunk ids into the python app

Signed-off-by: kyteinsky <kyteinsky@gmail.com>
Assisted-by: Github Copilot:claude-opus-5-5
@kyteinsky
feat: add /deleteProvider to wipe one provider's documents
31e5ca8 
Synchronous counterpart of the delete_provider_id action, so
context_chat:wipe --provider can report the result and reindex right
after. The bg threads are stopped while it runs.

Signed-off-by: kyteinsky <kyteinsky@gmail.com>
Assisted-by: Github Copilot:claude-opus-5-5
@nextcloud-command nextcloud-command added the AI assisted This PR contains AI-assisted commits label Oct 27, 2026
marcelklehr
marcelklehr reviewed Oct 27, 2026
View reviewed changes
Comment thread context_chat_backend/task_fetcher.py Outdated
Any batch the threads are working on is finished first, so nothing they fetched
before can be written to the vector db while the caller runs.
'''
with BG_THREADS_PAUSE_LOCK:

@marcelklehr marcelklehr Oct 27, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

worky in k8s?

@kyteinsky kyteinsky Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice catch!
yeah this will just be received in one pod, we don't know about others.
see the context_chat part of the change: nextcloud/context_chat#290, we can turn the tap off there and wait for the locked docs to be done before starting a wipe, that might work well.

there is one issue that some docs are locked for 24h due to failures, they won't clear up in the timeout of 30 mins. that we wait for the batch the complete.

this code can probably be removed from CCB then I suppose.

@marcelklehr

marcelklehr commented Oct 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Claude says:

enabled_handler doesn't take BG_THREADS_PAUSE_LOCK (controller.py:205-215):

  • Enable during a pause: THREAD_STOP_EVENT is set, so the handler calls start_bg_threads(), and indexing starts again in the middle of the wipe.
  • Disable during a pause: the finally restarts the threads anyway, so they run while the app is disabled. They then just poll app_enabled, so the effect is small.

Taking the same lock in the handler, or checking a "paused" flag there, would fix both.

The pause also waits for the request-processing thread. wait_for_bg_threads() joins every thread, including the one answering chat/LLM queries. That thread only reads, so it doesn't need to stop for a wipe. Waiting for a long LLM task, or for an indexing batch still being embedded on CPU, can hold the HTTP request open for minutes. The occ context_chat:wipe call on the PHP side may time out before then. Two suggestions:

  • Pause only the indexing and updates threads.
  • Check the HTTP client timeout in context_chat for these two endpoints.

marcelklehr
marcelklehr reviewed Oct 27, 2026
View reviewed changes
Comment thread context_chat_backend/vectordb/pgvector.py Resolved
@kyteinsky kyteinsky mentioned this pull request Oct 29, 2026
Open
1 task done
@kyteinsky
removeme: use the php app's branch for testing ci
401aa3d 
Signed-off-by: kyteinsky <kyteinsky@gmail.com>
@kyteinsky
kyteinsky force-pushed the feat/vector-db-wipe branch from bbbca5c to 401aa3d Compare October 29, 2026 13:57
@kyteinsky
fix(db): lock all three tables before a wipe + lock order fixes in ot…
eb694ac 
…her deletes

Signed-off-by: kyteinsky <kyteinsky@gmail.com>
@kyteinsky
revert: don't pause bg threads, pause is handled in the php app
43366b6 
Signed-off-by: kyteinsky <kyteinsky@gmail.com>
@kyteinsky

kyteinsky commented Nov 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

BG_THREADS_PAUSE_LOCK

removed it since it was only effective in docker based installs, the tap is closed in the php side now before calling out to CCB

Waiting for a long LLM task, or for an indexing batch still being embedded on CPU, can hold the HTTP request open for minutes. The occ context_chat:wipe call on the PHP side may time out before then.

which if fine, we have locks to prevent prompts running in parallel, and we wait for indexing of a batch to complete before doing a wipe for 30 mins and show an error to the admin if that is crossed.

also modified k8s integration test to have multiple pods per type and some work in parallel to test the lock and php side indexing stop changes.

@kyteinsky
kyteinsky force-pushed the feat/vector-db-wipe branch 3 times, most recently from 2a7967f to 66dd527 Compare November 4, 2026 05:37
@kyteinsky
ci(k8s): run multiple pods + run tasks in parallel to db wipe
1f39695 
Signed-off-by: kyteinsky <kyteinsky@gmail.com>
@kyteinsky
kyteinsky force-pushed the feat/vector-db-wipe branch from 66dd527 to 1f39695 Compare November 4, 2026 06:22
marcelklehr
marcelklehr reviewed Nov 4, 2026
View reviewed changes
Comment thread context_chat_backend/types.py


class DeleteProviderRequest(BaseModel):
providerKey: Annotated[str, AfterValidator(_validate_provider_id)]

@marcelklehr marcelklehr Nov 4, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this providerKey and not providerId?

@kyteinsky kyteinsky Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

providerKey is appId__providerId

didn't notice that the two apps refer to provider id differently, ccb thinks it's both appId + providerId, php app thinks it's the different providers by a single app
maybe we can do the change in another PR consistently?

@kyteinsky

kyteinsky commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

also modified k8s integration test to have multiple pods per type and some work in parallel to test the lock and php side indexing stop changes.

this passes now 🎉

still doing the changes suggested in the context_chat PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@marcelklehr marcelklehr marcelklehr approved these changes

Assignees

No one assigned

Labels

AI assisted This PR contains AI-assisted commits

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@kyteinsky @marcelklehr @nextcloud-command