Skip to content

Add concurrency control to LocalFileDescriptorStore #686

Open

Description

@s-heppner

Currently, LocalFileDescriptorStore in server/app/backend/local_file.py has the same concurrency problems that #591 fixes for LocalFileIdentifiableStore (see #554):

  • Multiple instances (e.g. one per uWSGI worker) can operate on the same directory. Each instance keeps its own _object_cache, which is not invalidated when another instance modifies the files.
  • add() and commit() check for the file's existence before writing it, without holding a lock. A concurrent add() or discard() can create or delete the file in between, so writes are silently overwritten or a discarded descriptor is written back.
  • add() and commit() write directly into the target file with open(..., "w"). A crash during the write leaves a truncated file, and a concurrent read can see a half-written one. LocalFileIdentifiableStore already avoids this by writing to a temp file and swapping it in with os.replace().

LocalFileDescriptorStore should get the same mechanism as LocalFileIdentifiableStore after #591:

  • Acquire a DirectoryLock on the storage directory, including the disable_strict_locking opt-out for systems without fcntl
  • Perform file and cache operations under a single _write_lock
  • Write atomically via a temp file and os.replace()
  • Add a close() method that releases the directory lock

Ideally, both stores share this logic, so that it does not have to be maintained twice.

This was deferred from the review of #591.

Activity

  1. added
    bugSomething isn't working
    serverSomething to do with the `server` package
    on Nov 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingserverSomething to do with the `server` package

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions