Skip to content

fix(locks): retry atomic_write_bytes on transient PermissionError (WinError 5) - #255

Open
sebastianbraun25 wants to merge 1 commit into
VectifyAI:mainfrom
sebastianbraun25:fix/issue-254-atomic-write-retry
Open

fix(locks): retry atomic_write_bytes on transient PermissionError (WinError 5)#255
sebastianbraun25 wants to merge 1 commit into
VectifyAI:mainfrom
sebastianbraun25:fix/issue-254-atomic-write-retry

Conversation

@sebastianbraun25

Copy link
Copy Markdown

Problem

atomic_write_bytes() in openkb/locks.py (the shared helper behind atomic_write_text/atomic_write_json, used by the compiler, indexer, converter, config, and page_ops) performed a single, unretried os.replace(tmp_path, path) to publish a write. On Windows, a freshly-written temp file can be briefly locked by an external process (real-time antivirus scanning, the search indexer, backup/sync agents) right before the rename — os.replace() then raises PermissionError (WinError 5, Access is denied) even though the lock typically clears within milliseconds to a couple of seconds.

Because this exception surfaced all the way up, _run_compile_with_retry() in cli.py (added in #229) treated it like any other failure: it discarded the fully-generated summary/concept/entity content already held in memory and re-ran the entire LLM compile pipeline from scratch just to retry a single file rename — wasting the LLM cost and time of a full document compile for a transient, typically self-clearing lock.

Root Cause

_compile_concepts() already holds all LLM-generated content (summary, concept/entity pages) in memory and only writes it to disk at the very end of the pipeline — so a write failure at that point has nothing to do with generation. The actual gap was one level lower: atomic_write_bytes()'s os.replace() call had no retry of its own, so any transient OS-level lock immediately propagated up to the much more expensive full-pipeline retry instead of being absorbed where it occurred.

Solution / Changes

  • openkb/locks.py: new _replace_with_retry() wraps the os.replace() call in atomic_write_bytes() with a bounded retry (5 attempts, exponential backoff starting at 50ms). Only PermissionError is retried — any other OSError (a real permissions problem) is raised immediately, unchanged from before. Each retry logs at DEBUG level (surfaces in the per-file debug log when debug: true/-v is enabled).
  • tests/test_locks.py: three new tests — succeeds after transient PermissionErrors, re-raises after exhausting all attempts (and still cleans up the temp file), and does not retry an unrelated OSError.
  • No config/CLI surface changes; behavior is unchanged on the happy path and for genuine (non-transient) permission errors.

Issues

Resolves #254. Related to #229 (origin of _run_compile_with_retry, whose full-pipeline retry this change makes largely unnecessary for this failure class).

…nError 5)

A freshly written temp file can be briefly locked by antivirus/indexing/backup software right before os.replace(), causing a transient PermissionError. Retry with short exponential backoff so this is absorbed before it ever reaches the compiler's full-pipeline retry, which previously had to re-run the entire LLM compile just to redo a file rename.\n\nResolves VectifyAI#254
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(locks): retry atomic_write_bytes on transient PermissionError (WinError 5)

1 participant