Repository navigation
Open a folder of a million files in seconds, with progress on the panel - #18
Merged
Merged
Conversation
Each file was read in turn, and the disk sat idle through every round trip between one request and the next, which on a folder of a few thousand documents accounted for most of the time spent in the read. Up to 32 reads are now in flight at once, both for the frontmatter when a folder opens and for the full text the first time the body links are asked for, and each document still takes its place in the list whatever order the reads finish in, so that the same folder always gives the same map. | Frontmatter of 3,844 files | Time | |---|---| | One at a time | 1.74 s | | 32 at a time | 0.63 s |
The slow part of opening a large folder was never the map's own work but the folder input, which builds a File object for every file below the chosen folder before the page hears anything at all. On a source tree of 1,302,052 files, Chrome spent 131 seconds on that list with no progress on the panel and then froze the tab, while about half of those files sat in `.git`, `node_modules` and build output that the map would have discarded. No change to the page could shorten a wait that happens inside the browser, and the fix therefore had to take the listing away from the input. In Chrome, the folder button now asks for the folder through the File System Access API, and the map walks the tree itself, one folder at a time. It never opens a dot folder, `node_modules` or `__pycache__`, it skips any folder that carries `CACHEDIR.TAG` or `pyvenv.cfg`, the tags that cargo, pytest and Python virtual environments leave behind, and it keeps only the markdown, while the panel counts folders and files as the walk goes and the page no longer looks frozen. A folder dropped on the page takes the same walk in every browser through the older entries API, and counterintuitively it is the faster of the two ways in, by about a third, because Chrome builds an entry far more cheaply than a handle. Firefox and Safari lack the File System Access API, and their button still waits on the input, which now at least skips the same cache folders. | Same tree | Time | |---|---| | Folder input | unfinished after 14 minutes | | Folder button | 20.3 s | | Drop | 13.0 s |
A dropped folder is now the fastest way into the map in every browser, and in Firefox and Safari it is the only way that avoids the full listing of the input, yet nothing on the page said that a drop was possible at all. The heading of the opening panel now reads "Choose a folder, or drop one on the page", and the first note of the help panel opens on the same sentence.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A large folder made the map look frozen, and on a source tree of 1,302,052 files it never opened at all: the folder input spent 131 seconds building a File object for every file before the page heard anything, and the tab then froze, although about half of those files sat in
.git,node_modulesand build output that the map would have discarded. The wait happens inside the browser, and no change to the page could shorten it while the input owned the listing.In Chrome the folder button now asks for the folder through the File System Access API, and the map walks the tree itself, one folder at a time, skipping dot folders,
node_modules,__pycache__and any folder tagged withCACHEDIR.TAGorpyvenv.cfg, while the panel counts folders and files as it goes. A folder dropped on the page takes the same walk in every browser through the entries API, and it is the faster of the two ways in, which the opening panel now says. Up to 32 files are read at once, for the frontmatter and for the body links.Notes
Need help on this PR? Tag
@codesmith-botwith what you need.