Problem
atomic_write_bytes() in openkb/locks.py (the shared helper behind atomic_write_text/atomic_write_json, used by the compiler, indexer, converter, config, and page_ops) performs a single, unretried os.replace(tmp_path, path) to publish a write. On Windows, a freshly-written temp file can be briefly locked by an external process (real-time antivirus scanning, the search indexer, backup/sync agents) right before the rename — os.replace() then raises PermissionError (WinError 5, Access is denied) even though the lock typically clears within milliseconds to a couple of seconds.
Because this exception surfaces all the way up, _run_compile_with_retry() in cli.py (added in #229) treats it like any other failure: it discards the fully-generated summary/concept/entity content already held in memory and re-runs the entire LLM compile pipeline from scratch (summary, concepts-plan, every concept/entity page, summary-rewrite) just to retry a single file rename — wasting the LLM cost and time of a full document compile for what is, at the OS level, a transient and typically self-clearing lock.
Reproduktion
Observed during a long-running batch openkb add (single KB, debug: true, thousands of documents processed sequentially over several hours):
[ERROR] Compilation failed: [WinError 5] Zugriff verweigert: '<kb>\wiki\summaries\.<doc>-xxxxxxxx.md.xxxxxxxx.tmp' -> '<kb>\wiki\summaries\<doc>-xxxxxxxx.md'
The retry (_run_compile_with_retry, 2 attempts) re-ran the whole compile and hit the same error again on the same file before finally failing the add. A later automatic retry (the calling batch script re-scans and re-adds failed documents) succeeded on the same file without any code change, confirming the lock was transient.
Kontext
- OS: Windows 11, NTFS local disk (not a network/cloud-sync path)
- Python 3.12/3.13,
openkb installed editable from a recent main-based branch
- Affects any long-running, high-file-count
add session, not a specific document type
Vorschlag
Add a small, bounded retry-with-backoff around the os.replace() call inside atomic_write_bytes() itself (e.g. a handful of attempts with short exponential backoff, catching PermissionError specifically), so the transient lock is absorbed at the exact point it occurs. This fixes the issue for every caller of atomic_write_text/atomic_write_json, not just the compiler, and — because it resolves before the exception ever reaches _run_compile_with_retry — avoids the wasteful full-pipeline regeneration in the common case without needing to restructure the compiler's generate/persist flow.
This issue was drafted with the assistance of an AI assistant.
Problem
atomic_write_bytes()inopenkb/locks.py(the shared helper behindatomic_write_text/atomic_write_json, used by the compiler, indexer, converter, config, and page_ops) performs a single, unretriedos.replace(tmp_path, path)to publish a write. On Windows, a freshly-written temp file can be briefly locked by an external process (real-time antivirus scanning, the search indexer, backup/sync agents) right before the rename —os.replace()then raisesPermissionError(WinError 5, Access is denied) even though the lock typically clears within milliseconds to a couple of seconds.Because this exception surfaces all the way up,
_run_compile_with_retry()incli.py(added in #229) treats it like any other failure: it discards the fully-generated summary/concept/entity content already held in memory and re-runs the entire LLM compile pipeline from scratch (summary, concepts-plan, every concept/entity page, summary-rewrite) just to retry a single file rename — wasting the LLM cost and time of a full document compile for what is, at the OS level, a transient and typically self-clearing lock.Reproduktion
Observed during a long-running batch
openkb add(single KB,debug: true, thousands of documents processed sequentially over several hours):The retry (
_run_compile_with_retry, 2 attempts) re-ran the whole compile and hit the same error again on the same file before finally failing theadd. A later automatic retry (the calling batch script re-scans and re-adds failed documents) succeeded on the same file without any code change, confirming the lock was transient.Kontext
openkbinstalled editable from a recentmain-based branchaddsession, not a specific document typeVorschlag
Add a small, bounded retry-with-backoff around the
os.replace()call insideatomic_write_bytes()itself (e.g. a handful of attempts with short exponential backoff, catchingPermissionErrorspecifically), so the transient lock is absorbed at the exact point it occurs. This fixes the issue for every caller ofatomic_write_text/atomic_write_json, not just the compiler, and — because it resolves before the exception ever reaches_run_compile_with_retry— avoids the wasteful full-pipeline regeneration in the common case without needing to restructure the compiler's generate/persist flow.This issue was drafted with the assistance of an AI assistant.