You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
index_long_document() in openkb/indexer.py (local PageIndex indexing for
long PDFs) never closes the local PageIndex SQLite connection it opens. client = PageIndexClient(...) / col = client.collection() opens .openkb/pageindex.db (WAL mode) and keeps it open for the lifetime of the
process — there is no close()/finally anywhere in the function.
If a later step in the same add (indexing itself, or the subsequent compile_long_doc()) raises, the mutation rollback in openkb/mutation.py
tries to unlink()/rename pageindex.db/-wal/-shm as part of discarding
the failed add. On Windows this fails with WinError 32 ("the process cannot
access the file because it is being used by another process"), because the
SQLite connection opened earlier in the same process is still holding the
file open:
openkb.mutation WARNING: Rolled back interrupted add journal ....json.
...
openkb.mutation WARNING: Mutation rollback failed: [WinError 32] Der Prozess
kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess
verwendet wird: '...\.openkb\pageindex.db'
[ERROR] Rollback failed; mutation journal retained for recovery: ...
The rollback failure leaves a dirty mutation journal on disk and (per add_coordinator.DirtyRollbackError) stops the rest of the batch instead of
continuing to the next file — a single failed long-document add can abort an
entire multi-file openkb add run.
The mutation rollback then hits the WinError 32 above and the batch stops.
Context
Windows, openkb installed from VectifyAI/OpenKB (fork-based dev
install), Python 3.12.
The pinned pageindex version exposes close() only on the low-level SQLiteStorage its local backend holds internally (client._backend ._storage) — there is no public close()/context-manager API on PageIndexClient/Collection itself.
Related: [Bug] PageIndex long document indexing fails with custom OPENAI_API_BASE / 401 Unauthorized #219 (PageIndex's own LLM calls not receiving the resolved
credential bundle) touches the same function and can be the trigger that
causes indexing to fail and hit this rollback bug — but the two are
independent problems (this one reproduces with any failure after PageIndex
has opened its local DB, not just an auth failure).
Vorschlag
Always close PageIndex's local storage connection(s) in a finally around index_long_document()'s body, in both the success and the failure path, so
a subsequent mutation rollback can always remove pageindex.db on Windows —
best-effort (never raises), since cleanup must not mask the real indexing
error.
This issue was drafted with the assistance of an AI assistant.
Problem
index_long_document()inopenkb/indexer.py(local PageIndex indexing forlong PDFs) never closes the local PageIndex SQLite connection it opens.
client = PageIndexClient(...)/col = client.collection()opens.openkb/pageindex.db(WAL mode) and keeps it open for the lifetime of theprocess — there is no
close()/finallyanywhere in the function.If a later step in the same
add(indexing itself, or the subsequentcompile_long_doc()) raises, the mutation rollback inopenkb/mutation.pytries to
unlink()/renamepageindex.db/-wal/-shmas part of discardingthe failed add. On Windows this fails with
WinError 32("the process cannotaccess the file because it is being used by another process"), because the
SQLite connection opened earlier in the same process is still holding the
file open:
The rollback failure leaves a dirty mutation journal on disk and (per
add_coordinator.DirtyRollbackError) stops the rest of the batch instead ofcontinuing to the next file — a single failed long-document add can abort an
entire multi-file
openkb addrun.Reproduction
openkb add <folder-with-a-long-pdf>on Windows.credential-forwarding bug in [Bug] PageIndex long document indexing fails with custom OPENAI_API_BASE / 401 Unauthorized #219, or any other transient error during
index_long_document()/compile_long_doc().WinError 32above and the batch stops.Context
openkbinstalled fromVectifyAI/OpenKB(fork-based devinstall), Python 3.12.
pageindexversion exposesclose()only on the low-levelSQLiteStorageits local backend holds internally (client._backend ._storage) — there is no publicclose()/context-manager API onPageIndexClient/Collectionitself.credential bundle) touches the same function and can be the trigger that
causes indexing to fail and hit this rollback bug — but the two are
independent problems (this one reproduces with any failure after PageIndex
has opened its local DB, not just an auth failure).
Vorschlag
Always close PageIndex's local storage connection(s) in a
finallyaroundindex_long_document()'s body, in both the success and the failure path, soa subsequent mutation rollback can always remove
pageindex.dbon Windows —best-effort (never raises), since cleanup must not mask the real indexing
error.
This issue was drafted with the assistance of an AI assistant.