Repository navigation
GalleryFilesystem: follow paginated folder listings - #67
Open
tilorspace wants to merge 1 commit into
Open
tilorspace wants to merge 1 commit into
tilorspace wants to merge 1 commit into
Conversation
GalleryFilesystem looked for the Gallery folder on the first page of the Home folder listing only, and listdir returned only the first page of a folder. With RSpace's default page size of 20 and newest-first ordering, the filesystem failed to open (StopIteration) once a user had about 14 items of their own in Home, and listings of Gallery folders with more than 20 entries were silently truncated. Galaxy's shipped RSpace file source is built on this class, so both showed up there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
looks good to me. could you take a brief look as well @nebay-abraha ? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
rspace_client.eln.fs.GalleryFilesystem(the PyFilesystem2 class Galaxy's shippedrspacefile source is built on) read only the first page of/folders/tree:__init__looks for the folder namedGalleryin the Home listing. RSpace returns 20 items per page, newest first, and Gallery dates from account creation, so once a user has about 14 items of their own in Home the lookup raisesStopIterationand Galaxy reports "Problem listing file source path".listdirreturned only the first page, so a Gallery folder with more than 20 entries showed only the newest 20 in Galaxy's picker. Galaxy'sPyFilesystem2FilesSource._listgoesfilterdir→scandir→listdir.Found while retesting against RSpace 2.27.0 and Galaxy 26.1: the integration tests had filled Home with test folders and the shipped plugin stopped working.
Fix
A small
_all_recordsgenerator follows the listing'snextlinks, used by both the Gallery lookup andlistdir. The new fsspec filesystem onfeature/fsspec-filesystemalready pages this way.Testing
ElnFilesystemPagingTest) fail without the fix and pass with it; the full unit suite passes.listdiron Miscellaneous and Images returns 31 and 25 entries, matching the server'stotalHits.rspaceplugin lists the Gallery again with thiseln/fs.py.🤖 Generated with Claude Code