Repository navigation
Add rspace_client.fs: the Gallery, Inventory and ELN Workspace as one fsspec filesystem - #68
Open
tilorspace wants to merge 29 commits into
Open
tilorspace wants to merge 29 commits into
tilorspace wants to merge 29 commits into
Conversation
…s the filesystems need URLs the server advertises in `_links` are rebased onto the configured host so paging and download links work behind a proxy or port-forward. The session never follows a redirect, because `requests` would carry the `apiKey` header to the new host. `doDelete` tolerates a leading slash. `upload_file` and `upload_attachment_by_global_id` take `filename=`, `list_folder_tree` takes `page_size=`, and the Inventory client gains `get_workbench_by_id` and `attach_gallery_file_by_global_id`. Inventory CSV import goes through the shared session. The existing filesystem tests now expect the redirect flag on every request. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… suite A standard-library HTTP server covering the endpoints the PyFilesystem classes use, backed by an in-memory dataset with the awkward cases a filesystem has to handle (duplicate names, system folders, paginated sections, documents with mixed field types). It counts requests so tests can assert what an operation costs. One server runs for the whole pytest session and `MockServerTestCase` resets its fixtures before each test. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…s of the PyFilesystem work RSDEV-1519: the unified filesystem is being built on fsspec rather than PyFilesystem2. This branch starts from the mock-server stage (fs-stack/2) and takes from review/unified-pyfilesystem only what does not depend on PyFilesystem: - docs/unified-pyfilesystem-design.md, the browsing-model record, with a status note saying which sections describe what to build and which glue is replaced - rspace_client/fs/paths.py, the segment and path-style helpers, now importing from posixpath instead of fs.path - tools/galaxy/check_plugin.py and its README, to be re-pointed at the fsspec plugin - tools/fsspec_spike/, the 2026-09-21 prototype that proved the Gallery branch, the mount dispatcher, transactions and Galaxy's FsspecFilesSource end to end; it documents the fsspec traps found and is to be deleted once the real package exists Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…e plugin Galaxy 25.1 already ships _fsspec.py (fsspec 2025.9.0) but its path hooks take no config argument and _open_fs no cache_options, so the prototype plugin accepts both signatures. rspace-client 2.7.4 in that venv has no page_size on list_folder_tree, so the prototype falls back. check_in_galaxy_container.py drives the shipped PyFilesystem plugin and the fsspec one through ConfiguredFileSources against the mock server: same listings, fsspec adds name-based URIs, glob search and disambiguated duplicates. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ce files Live RSpace 2.26 file records carry created but no lastModified, and Galaxy's fsspec base reads only mtime, modified or LastModified for the time column. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…override for bracketed names Exploratory testing in Galaxy 25.1 on 2026-10-06 showed that a clicked row is re-resolved from its stored URI string in a later request, so the appended-ID rule is load-bearing in a pure UI consumer too, and that Galaxy's bracket escaping defeats fsspec's glob, which an override of glob in our class fixes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
rspace_client.fs presents RSpace as an fsspec filesystem (RSDEV-1519). RSpaceFilesystem mounts gallery/, inventory/ and workspace/; the three branches keep the browsing model of the PyFilesystem design record unchanged (directory = contains files, documents as folders of text fields, Inventory records as folders of records and attachments, attachment fields as one-file folders, link instead of copy, read-only by default, signed-document lock). Framework glue, new for fsspec: - base.py: entries are dicts (name = full path, type, size, mtime, created, globalId, rspace, writable); client exceptions become OSError subclasses at every public method (FileNotFoundError, PermissionError, ConnectionError, RemoteApiError; a ReadOnlyError for the posture); the listing keeps segments unique and self-addressing (duplicates and names that read as addresses get their real ID, as the design requires, because a consumer such as Galaxy stores the segment and re-resolves it later); reads buffer the file, writes buffer until close or transaction commit; glob answers the "<dir>/*<text>*" pattern Galaxy's search sends with the brackets it escapes. - rspace.py: the mount dispatcher fsspec does not ship, plus rspace:// URL handling (host and options from the URL, the key only from RSPACE_API_KEY, http for loopback hosts only, credentials in the URL refused). Registered as the "rspace" protocol via fsspec.specs; UPath works. - tree.py: the tree view over entries. The deprecated PyFilesystem classes stay as they are for Galaxy's shipped plugin and now warn; they and the fs dependency leave in 3.0. tools/galaxy/rspace_fsspec_source.py is the reference Galaxy file source for the new filesystem. The prototype under tools/fsspec_spike is gone. Tests: the PyFilesystem suites ported to the fsspec API, 139 tests over the mock server, plus the live integration module. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- Resolving a bare name listed the parent with the Gallery's per-file size fetches, so
info() of one file by name cost 6 requests; lookups and emptiness checks now read the
listing in a names-only mode (4 requests, sizes still fetched for real listings).
- glob("<dir>/**") was answered as a search of <dir> rather than a recursive glob.
- invalidate_cache() left fsspec's listings cache untouched (the base does nothing), so a
write with use_listings_cache=True left the old listing in place; the cache is dropped.
- mv() out of the Gallery copied the file and only then failed on rm_file, leaving a
duplicate the API cannot delete; a branch that cannot delete files refuses before copying,
both directly and through the mount.
- _strip_protocol() rejected the list form fsspec passes to cat()/rm()/expand_path().
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
fsspec's put() passes directories through put_file and expects the backend to make them before their files arrive; returning early left 'Documents/sub/b.txt' with no parent. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… a RemoteApiError fsspec.core.url_to_fs, get_mapper and UPath hand the mount paths such as 'host:port/gallery/Images?scheme=http'; only the fixed mount names may appear at the top level, so a first segment that looks like a host is dropped with its query string, and find()/keys now line up with the root path those consumers use. RSpace answers 500 for a request it understood but could not carry out (an invalid image), with the reason in the message; only 502/503/504 are mapped to ConnectionError now. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Reuse, simplification, efficiency and altitude pass over rspace_client.fs: - the deprecated eln.fs Gallery module imports the section policy, the classification table, GallerySectionMismatch, Placement and the placement helpers from fs.gallery instead of carrying a copy - one translate_class helper replaces four hand-written translation loops - RSpaceFilesystem delegates its simple methods through a factory, strips host and query once in _strip_protocol by splitting the URL, and derives mount_names from the branch dict - shared branch code moves into the base: file download, file-bearing field helpers, a default link() behind CAN_LINK, _require_movable - the _names_only instance flag becomes a names_only argument - a listing records every child in the name cache, so browsing a directory and then its children no longer re-lists the parent per child - Inventory field lookups fetch the record without container contents; get_file and rm resolve the path instead of an extra info request - the Galaxy search glob regex states what it accepts; the mismatch message is built in one function; dead helpers and duplicate checks go - tests share one names()/segments() helper and one lazily started mock server; fixture builders call the mutation methods with a chosen id - universal-pathlib is declared as a dev dependency (the mount tests import it) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…in contract requests 2.34 checks that a multipart file part defines read on its class, which on Python 3.12+ skips __getattr__. Galaxy's shipped rspace file source passes exactly such a wrapper (FakedNameIO) to ELNClient.upload_file, so exports from Galaxy 26.1, which pins requests 2.34.2, fail with a TypeError on any released rspace-client. _post_multipart now wraps such objects in one that defines read and keeps the name. Ports galaxy_plugin_contract_test.py from the closed PyFilesystem stack, adapted to the shim as it is here (the 2.7 class, not a subclass of the new one), and restores MISCELLANEOUS_SECTION, the one 2.7.4 shim export the move to rspace_client.fs.gallery dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fsspec.filesystem("rspace") and rspace:// URLs now resolve without importing
rspace_client.fs first, as the package docstring already said they did.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Renamed to docs/fsspec-filesystem-design.md. The sections on mounting, write posture, listings, errors and the opener described PyFilesystem glue that no longer exists; the Galaxy section now lists the five things the shipped plugin relies on, the requests 2.34 breakage, and what the upstream file source pull request has to carry. DEVELOPING points at the contract test, CHANGELOG records the fix and the entry point. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
That module never shipped; #57 was a docs error, fixed by pointing the usage guide at rspace_client.inv.attachment_fs. Say which shim Galaxy imports and why the other stays. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_rebase_url swapped scheme and host but kept the server's path, so a client configured with a path prefix (https://proxy/rspace) followed next-page and download links to https://proxy/api/v1/... and lost /rspace. Re-anchor the part from /api/ on under our own base path, the same base relative requests use. Links without /api/ keep their path. Raised in review of PR #60. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Covers the review's test gaps: doDelete single-slash URLs (regression for delete_document), upload_file and upload_attachment_by_global_id filename=, list_folder_tree page_size=, get_workbench_by_id id forms and includeContent, attach_gallery_file_by_global_id body, and CSV parse through the shared session with redirects off and the client timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tilorspace
added a commit
that referenced
this pull request
Oct 12, 2026
…e filesystems need (#60) Client layer: rebase advertised links, refuse redirects, add calls the filesystems need (#60) Changes to the ELN and Inventory REST clients that the filesystem work needs, plus fixes that stand on their own: - Links the server advertises in `_links` are rebased onto the configured base URL, path prefix included, so next-page and download links work behind a port-forward, in a container, or behind a proxy that serves RSpace under a sub-path such as /rspace. - The shared session never follows an HTTP redirect, because requests would carry the apiKey header to the new host. - doDelete tolerates a leading slash, which fixes delete_document. - ELNClient.upload_file and InventoryClient.upload_attachment_by_global_id take an optional filename=; list_folder_tree takes page_size=. - New InventoryClient.get_workbench_by_id (/containers answers 422 for a bench) and attach_gallery_file_by_global_id (link an existing Gallery file to an Inventory record). - Inventory CSV import goes through the shared session, so it gets the same timeout, retries and error handling as every other request. - A stray debug print is removed from upload_attachment_by_global_id. Review follow-up: the path-prefix handling in link rebasing, and unit tests for each changed or new client method (client_methods_test.py). Stack: this was part 1 of five PRs towards a unified PyFilesystem (#58). On 2026-10-06 the filesystem was moved to fsspec instead, so parts 3 to 5 (#62, #63, #64) were closed without merging. This client layer and the offline mock server (#61) are independent of the filesystem framework and are kept. The fsspec filesystem itself is #68, which builds on #61. Merge #61 next, then retarget #68 to main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… suite A standard-library HTTP server covering the endpoints the PyFilesystem classes use, backed by an in-memory dataset with the awkward cases a filesystem has to handle (duplicate names, system folders, paginated sections, documents with mixed field types). It counts requests so tests can assert what an operation costs. One server runs for the whole pytest session and `MockServerTestCase` resets its fixtures before each test. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
tilorspace
force-pushed
the
fs-stack/2-mock-server
branch
from
October 12, 2026 07:43
638a685 to
631051a
Compare
This was referenced Oct 12, 2026
fs-stack/2 was rebased onto main after #60 was squash-merged; its content is unchanged, so the feature branch keeps its own tree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tilorspace
added a commit
that referenced
this pull request
Oct 12, 2026
… suite (#61) A standard-library mock of the ELN (/api/v1) and Inventory (/api/inventory/v1) endpoints the filesystem classes use, under rspace_client/tests/mock_rspace. Nothing here is installed with the package. - server.py serves an in-memory dataset from fixtures.py and counts requests, so a test can assert what a listing costs. - The dataset carries the cases a filesystem has to get right: duplicate names, names that look like global IDs, system folders in Home, sections large enough to paginate, a document with text and non-text fields, a signed document, a bench with attachments. - conftest.py starts one server for the whole pytest session; MockServerTestCase resets its fixtures before every test. No test uses it yet; #68 adds the fsspec filesystem tests, the Galaxy contract test and the plugin check tool that run against it. Stack: part 2 of the original PyFilesystem stack (#58). Parts 3 to 5 (#62, #63, #64) were closed when the filesystem moved to fsspec; the client layer (#60) and this mock are framework-neutral and were kept. #68, the fsspec filesystem, is now based on main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#61 was squash-merged into main; main now holds the same content this branch already had, so the branch keeps its own tree (conftest.py stays deleted, the feature's mock fixtures stay). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The tests were gated on RSPACE_SUPPORTS_LINK_FIELDS while server PR rspace-os/rspace-web#803 was unreleased. It has since shipped, and CI runs against the latest rspace-web release (2.27.0), so the gate only made CI skip them. All five pass against RSpace 2.27. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RSpace dates end in Z for UTC. Galaxy's shipped rspace file source parses the shim's rspace.created with datetime.fromisoformat, which rejects Z before Python 3.11, so on Python 3.10 (still supported by Galaxy) listing any Gallery folder that holds files failed with ValueError. That is true of every released rspace-client; the contract test surfaced it when CI ran it on 3.9 and 3.10. GalleryFilesystem now spells the same instant as +00:00 in the created field and passes the rest of the record through. A new contract test pins the form on every Python version. Checked with Galaxy's real plugin (galaxy-files 26.1.1) on Python 3.10 against the mock: the listing fails before and succeeds after. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Inventory attachment tests borrowed the first listed sample with an attachment field. CI runs as a sysadmin, who sees samples it may not edit, so the first match (SA4) answered 422 "cannot be edited by current User". Each test now creates a sample from a template with one attachment field and one subsample, and deletes both afterwards. The tests no longer skip on an account without such a sample either. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked PR. Merge after #60 (client layer) and #61 (offline mock server); this one targets
fs-stack/2-mock-server.Replaces the PyFilesystem2 version of the unified filesystem (#62 to #64, closed). PyFilesystem2 has had no release since 2022, and Galaxy, the main consumer, is moving every file source to fsspec (galaxyproject/galaxy#21832, where
rspace.pyis on the list). Tracked in RSDEV-1519.What it adds
rspace_client.fs, an fsspec filesystem.RSpaceFilesystemmountsgallery/,inventory/andworkspace/, sharing one ELN and one Inventory client. Each branch is also usable on its own (GalleryFilesystem,InventoryFilesystem,WorkspaceFilesystem).gallery/Images/microscope.png). Duplicates, and names that read as addresses, carry their global ID (data [GL112].csv).path_style="labelled"or"id"give paths that survive a rename.writable=True,allow_delete=True). Uploads into Inventory records and ELN fields go through the Gallery and are linked.cp_filefrom a Gallery-backed file links instead of copying bytes. Transactions defer uploads.rspace://host/...URLs viafsspec.open,fsspec.filesystem("rspace")andUPath, registered as anfsspec.specsentry point. The key comes fromRSPACE_API_KEYonly, never from the URL.OSErrorsubclasses, soexists,walk,copyand Galaxy behave.rspace_client.eln.fs.GalleryFilesystemandrspace_client.inv.attachment_fs.InventoryAttachmentFilesystem. Their behaviour is unchanged apart from the two Galaxy fixes below, they warn on construction, and they're removed in 3.0 together withfs, after Galaxy's shipped plugin has moved.readthrough__getattr__. That's how Galaxy'sFakedNameIOwrapper works, so exports through Galaxy's shippedrspacesource fail on every released rspace-client._post_multipartnow wraps such objects. Verified inside a Galaxy 26.1.2 container: 2.7.4 raises theTypeError, this branch uploads.rspacesource parses a file'screateddate withdatetime.fromisoformat, which rejects RSpace's trailingZbefore Python 3.11, so listing any Gallery folder with files failed on Python 3.10 with every released rspace-client.GalleryFilesystemnow gives the same UTC instant as+00:00. Verified with Galaxy's real plugin (galaxy-files 26.1.1) on Python 3.10.galaxy_plugin_contract_test.py, which pins what Galaxy's shipped plugin relies on. A failure there is a release blocker.tools/galaxy/rspace_fsspec_source.py, the reference Galaxy file source (typerspace_unified) to propose upstream once this is released.docs/usage-guide.mdand the design recorddocs/fsspec-filesystem-design.md, which explains why the package is shaped this way and what must not move.Review order
docs/fsspec-filesystem-design.md(the decisions)rspace_client/fs/paths.py,rspace_client/fs/base.py(shared behaviour)gallery.py,inventory.py,workspace.py, thenrspace.py(the mount)rspace_client/eln/fs.py,rspace_client/inv/attachment_fs.py, and the contract testfs_*_test.py),tools/galaxy/Testing
poetry run pytest -m "not integration"), using the mock server from Add an offline mock of the RSpace ELN and Inventory APIs for the test suite #61.fs_integration_test.pyagainst a docker RSpace 2.27, plus a record-type matrix covering Gallery, documents, samples, subsamples, containers, instruments, and the bench and template refusals.%), pagination, import from Gallery, Inventory and Workspace, and export into a sample, a field and the Gallery.New dependency:
fsspec>=2024.3.0.fsand thesetuptools<82pin stay until 3.0.🤖 Generated with Claude Code