fs: read small files in one thread pool round trip - #65327
Open
codebytere wants to merge 1 commit into
Open
Conversation
codebytere
force-pushed
the
perf/fs-readfile-one-roundtrip
branch
from
August 16, 2026 16:54
0eaf058 to
21db0ab
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #65327 +/- ##
==========================================
- Coverage 90.13% 90.10% -0.04%
==========================================
Files 752 752
Lines 251568 251915 +347
Branches 47270 47353 +83
==========================================
+ Hits 226759 226976 +217
- Misses 16168 16266 +98
- Partials 8641 8673 +32
🚀 New features to boost your workflow:
|
anonrig
approved these changes
Aug 16, 2026
Collaborator
jasnell
reviewed
Aug 17, 2026
jasnell
requested changes
Aug 17, 2026
jasnell
left a comment
Member
There was a problem hiding this comment.
Test failure on AIX needs to be looked at. Also, are these reads actually abortable in any way?
fs.readFile(path) took four libuv thread pool round trips for a typical small file -- open, fstat, read and close, each its own uv_fs request with its own queue wait, completion callback and JS/C++ crossing -- and fs.promises.readFile(path) did the same through a FileHandle. For the small files applications read most, the round trips are the cost, and each occupies a slot in the pool that concurrent dns.lookup(), fs and crypto work is also queueing for. Add ReadFileJob (an AsyncWrap + ThreadPoolWork) that performs open + fstat + read-to-EOF + close as one pool task and reports the whole content, or, when the file turns out to be larger than one chunk (kReadFileBufferLength, 512 KiB), stops after fstat() and hands the fd and size back so that the existing chunked reader continues unchanged (large reads stay interleaved and abortable between chunks, and still save the fstat round trip). fs.readFile() and fs.promises.readFile() use it for path arguments without a user buffer; file descriptors, FileHandles, options.buffer and an active VFS keep their paths. Behavior is otherwise kept: same bytes for every size and encoding; open failures report syscall 'open' with the path, read failures 'read'; permission errors are delivered through the callback/promise as before; an abort that arrives while the read is in flight still wins; the job is an FSREQCALLBACK resource for async_hooks; a handed back fd is tracked exactly like one from a plain open(). Tests that asserted the internal open/fstat/read/close request chain, used readFile() as a proxy for an fstat trace event, or injected faults through FileHandle.prototype for path-based reads are adjusted to keep testing what they test (a file just over one chunk where the chain shape matters, fs.fstat() for the fstat trace, a larger file so the FileHandle path is taken). fs.readFile() of 4 KiB files at concurrency 64 goes from ~51k to ~306k files per second, and a mixed stat/readFile/dns.lookup burst from ~66k to ~312k operations per second. Signed-off-by: Shelley Vohr <shelley.vohr@gmail.com>
codebytere
force-pushed
the
perf/fs-readfile-one-roundtrip
branch
from
August 17, 2026 05:17
21db0ab to
b626e93
Compare
Member
Author
|
@jasnell re aborts: the single round trip itself isn't interruptible once it's on the pool, same as an individual read req now; an already-aborted signal never schedules it, an abort that lands while it's in flight wins when it completes, and anything over one chunk hands the fd back to the existing chunked reader, so those stay abortable between chunks exactly as before. |
jasnell
approved these changes
Aug 17, 2026
Collaborator
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fs.readFile()/fs.promises.readFile()of small files get 3–6× faster, and use one libuv thread-pool task instead offour, by doing open + fstat + read + close in a single round trip.
(Linux x64,
--set duration=2, 30 runs.)Today a path-based
readFileissuesopen,fstat,readandcloseas four separateuv_fs_*requests, each withits own queue wait, completion callback and JS↔C++ crossing; the promise API does the same through a
FileHandle. Forsmall files those round trips are the whole cost, and each one takes a pool slot away from concurrent dns/zlib/crypto/fs
work.
ReadFileJob(anAsyncWrap+ThreadPoolWork, providerFSREQCALLBACK) runs open + fstat + read-to-EOF + close as onetask and returns the content. If the file is larger than one chunk (
kReadFileBufferLength, 512 KiB) it stops afterfstatand hands back the fd and size, and the existing chunked reader continues exactly as today (interleaved,abortable between chunks). Both
readFiles use it for path arguments without a user buffer; fds andFileHandles areunchanged.
Preserved on purpose: identical results for every size/encoding;
openerrors reportsyscall: 'open'+path, readerrors
'read'; permission errors arrive through the callback/promise; an abort that lands while the round trip is inflight still wins; the handed-back fd is tracked and closed like any other; size-0 files (procfs) are read to EOF.
One open point: 16–32 MiB reads via
fs.promises.readFile(…, 'utf-8')at concurrency 10 measure −2…3 % (***),reproducibly; the same sizes as Buffers, via the callback API, or at concurrency 1 are flat. They take the hand-back path
with identical syscalls, and direct timing shows ≤2 %, so I haven't pinned it down. If preferred, the promise API can keep
its current path and only the callback API changes.
Tests:
test-fs-readfile-one-roundtrip.js(new; also passes on currentmain): sizes across the 512 KiB threshold,encodings, flags, error shapes, abort before/during, fd/
FileHandleinputs, no fd leak on hand-back, procfs/sysfs,async_hooks lifecycle. Adjusted to keep testing what they test:
test/async-hooks/test-fsreqcallback-readFile.js(acceptsone request instead of exactly four),
test-graph.fsreq-readFile.js(reads a 512 KiB+1 file so the four-requestchain keeps its shape),
test-async-exec-resource-match.js(resource + ≥1 fs request),test-trace-events-fs-async.js(uses
fs.fstat()for the fstat trace instead ofreadFileas a proxy),test-fs-promises-readfile.js(zero-size-liarcase goes through a
FileHandle),test-fs-promises-file-handle-{op,aggregate,close}-errors.js(use a >512 KiB fileso the patched
FileHandlepath is taken). fs, async-hooks, permission, worker, process and child_process suites pass.Disclosure: the code, test, measurements and this description were written by Claude Code, directed and reviewed by @codebytere.