Skip to content

fix(indexer): close PageIndex's local SQLite connection so rollback doesn't hit WinError 32 #249

Description

@sebastianbraun25

Problem

index_long_document() in openkb/indexer.py (local PageIndex indexing for
long PDFs) never closes the local PageIndex SQLite connection it opens.
client = PageIndexClient(...) / col = client.collection() opens
.openkb/pageindex.db (WAL mode) and keeps it open for the lifetime of the
process — there is no close()/finally anywhere in the function.

If a later step in the same add (indexing itself, or the subsequent
compile_long_doc()) raises, the mutation rollback in openkb/mutation.py
tries to unlink()/rename pageindex.db/-wal/-shm as part of discarding
the failed add. On Windows this fails with WinError 32 ("the process cannot
access the file because it is being used by another process"), because the
SQLite connection opened earlier in the same process is still holding the
file open:

openkb.mutation WARNING: Rolled back interrupted add journal ....json.
...
openkb.mutation WARNING: Mutation rollback failed: [WinError 32] Der Prozess
kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess
verwendet wird: '...\.openkb\pageindex.db'
[ERROR] Rollback failed; mutation journal retained for recovery: ...

The rollback failure leaves a dirty mutation journal on disk and (per
add_coordinator.DirtyRollbackError) stops the rest of the batch instead of
continuing to the next file — a single failed long-document add can abort an
entire multi-file openkb add run.

Reproduction

Context

  • Windows, openkb installed from VectifyAI/OpenKB (fork-based dev
    install), Python 3.12.
  • The pinned pageindex version exposes close() only on the low-level
    SQLiteStorage its local backend holds internally (client._backend ._storage) — there is no public close()/context-manager API on
    PageIndexClient/Collection itself.
  • Related: [Bug] PageIndex long document indexing fails with custom OPENAI_API_BASE / 401 Unauthorized #219 (PageIndex's own LLM calls not receiving the resolved
    credential bundle) touches the same function and can be the trigger that
    causes indexing to fail and hit this rollback bug — but the two are
    independent problems (this one reproduces with any failure after PageIndex
    has opened its local DB, not just an auth failure).

Vorschlag

Always close PageIndex's local storage connection(s) in a finally around
index_long_document()'s body, in both the success and the failure path, so
a subsequent mutation rollback can always remove pageindex.db on Windows —
best-effort (never raises), since cleanup must not mask the real indexing
error.


This issue was drafted with the assistance of an AI assistant.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions