Skip to content

Every segment file stays open for the life of the engine #68

Closed

Description

@venkat1701

PageStore opens every segment file at startup and keeps a FileChannel for each until the engine closes. So the number of open files grows with the size of the database: about 400 for 100 GB at today's 256 MiB segments, and thousands if segments get smaller (#52).

The engine should manage this itself:

  • keep at most N segment files open, as a new setting
  • open a file when a page in it is needed, and close the least recently used one when the pool is full
  • never close a file another thread is using
  • never close a file with unsynced writes before forcing it, since sync commits rely on those pages being on disk
  • sync() should only force files written since the last sync, not every file

Also try memory-mapping sealed segments (read-only, with the file handle closed after mapping) next to the pool. Keep it only if it can be made reliable: a mapped file that is truncated or deleted underneath can crash the process instead of raising an error, and mapped memory is harder to bound.

Needed before #52 can make segments smaller.

Activity

  1. venkat1701 commented on Oct 11, 2026

    @venkat1701
    CollaboratorAuthor

    Partly done: #77 maps sealed segments read-only, and the maps are now closed as soon as a segment is replaced or deleted. Each segment still keeps its FileChannel open for the life of the engine, so the open-file count still grows with the database, and #52 still waits on this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions