diff --git a/CHANGELOG.md b/CHANGELOG.md index 81fa09a..a27e800 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,36 @@ All notable changes to this project will be documented in this file ## 2.8.0 (unreleased) +- New `rspace_client.fs`: RSpace as an [fsspec](https://filesystem-spec.readthedocs.io) + filesystem. `RSpaceFilesystem` mounts `gallery/`, `inventory/` and `workspace/` (Gallery + folders and files; Inventory benches, containers, samples, subsamples, templates and + instruments as folders of their records and attachments, attachment fields as one-file + folders; ELN folders, notebooks and documents, a document as a folder of its text fields + and a field as a folder of its linked files). Paths are record names; duplicate names + and names that read as addresses carry their global ID (`data [GL112].csv`); + `path_style="labelled"` / `"id"` for stable paths. Read-only by default (`writable=True`, + `allow_delete=True`). Uploads into Inventory and the Workspace go through the Gallery and + link; `cp_file` from any Gallery-backed file links instead of copying bytes. Transactions + defer uploads. `rspace://host` URLs via `fsspec.open`, `fsspec.filesystem("rspace")` and + `UPath` (registered as an `fsspec.specs` entry point), key from `RSPACE_API_KEY` only. `glob("/**")` answers Galaxy's search + including bracketed names. New dependency `fsspec`. +- Deprecated `rspace_client.eln.fs.GalleryFilesystem` and + `rspace_client.inv.attachment_fs.InventoryAttachmentFilesystem` (PyFilesystem2, unmaintained + upstream). Unchanged in behaviour apart from the fixes below, now emit a `DeprecationWarning`; removal together with + the `fs` dependency in 3.0, after Galaxy's shipped plugin has moved to `rspace_client.fs`. +- Fix: uploads accept a file object that forwards `read` through `__getattr__`. requests 2.34 + refuses such objects on Python 3.12+, which broke exports from Galaxy 26.1 (it pins requests + 2.34.2) to RSpace through Galaxy's shipped `rspace` file source. +- Fix: `GalleryFilesystem` reports a file's `created` date as `+00:00` instead of RSpace's + trailing `Z` (the same UTC instant). Galaxy's shipped `rspace` file source parses it with + `datetime.fromisoformat`, which rejects `Z` before Python 3.11, so on Python 3.10 Galaxy + failed to list any Gallery folder holding files. +- `rspace_client/tests/galaxy_plugin_contract_test.py` pins what Galaxy's shipped plugin relies + on; a failure there is a release blocker. +- Tests: the Link-field integration tests always run instead of needing + `RSPACE_SUPPORTS_LINK_FIELDS=1`; current RSpace releases support Link fields. +- `tools/galaxy/rspace_fsspec_source.py`: the reference Galaxy file source for the new + filesystem, to be proposed upstream. - Security: the client never follows an HTTP redirect, since `requests` would carry the `apiKey` header to the new host. - Client-level changes: `_links` URLs are rebased onto the configured base URL, path prefix diff --git a/DEVELOPING.md b/DEVELOPING.md index bef4d7a..cc26d84 100644 --- a/DEVELOPING.md +++ b/DEVELOPING.md @@ -58,6 +58,10 @@ All top-level methods for use by client code should be unit-tested. RSpace can be run on Docker on a developer machine, providing access to a sandbox environment. +#### Galaxy's RSpace file source + +Galaxy ships an `rspace` file source built on this library's deprecated `rspace_client.eln.fs.GalleryFilesystem`, so changes here can break Galaxy. `rspace_client/tests/galaxy_plugin_contract_test.py` pins what it relies on and runs with the normal suite; a failure there is a release blocker. [tools/galaxy/README.md](tools/galaxy/README.md) shows how to drive Galaxy's real plugin without a Galaxy server, and how to try the fsspec file source proposed for Galaxy. Why the contract looks the way it does is in [docs/fsspec-filesystem-design.md](docs/fsspec-filesystem-design.md), section 8. + ### Making a release diff --git a/README.md b/README.md index 28a4076..4b7cf06 100644 --- a/README.md +++ b/README.md @@ -45,7 +45,7 @@ Full REST API reference is served by your own RSpace instance at `https:///**` shape: one listing of the directory, an + unescaped case-insensitive substring match on the last segment, results in listing order. + +Inventory adds fixed section names at its root, so each branch has a real resolver rather than +the old "strip the two-letter prefix" helper. Nothing in a path is a page token: `page-N` +pseudo-folders were built and removed, see section 5. + +## 3. Write posture + +Read-only by default. `writable=True` allows uploads, folder creation and links; +`allow_delete=True` allows removals. The two are separate on purpose, so `read_only` is false +if either is set. Every mutating method checks the posture before any request and raises +`ReadOnlyError` (a `PermissionError`), and the fixed top level of `RSpaceFilesystem` refuses +writes the same way, so a write there cannot silently succeed. A `move` checks the delete permission before copying anything. + +`remove` deliberately differs per branch: in the Workspace it unlinks the file from its field +and leaves the Gallery file alone; in the Inventory it deletes the attachment record; in the +Gallery it is unsupported, because the API cannot delete Gallery files. Each says which it did. + +A signed ELN document is locked. Every field write (upload, link, remove) refuses before any +request is made, and a document's fields are listed with a `(signed)` suffix under the `name` +style and report the lock as `writable: false` in their entry. The folder-tree endpoint does not carry +the flag, so the document's own entry in its parent listing is not marked. + +Uploading into a text field is a read-modify-write of the field's HTML; concurrent edits are +last-writer-wins, as they are for the web client. + +## 4. What each branch contains + +**Gallery.** Root resolved lazily on first use from the Home tree (no network in a +constructor). The folder-tree endpoint returns no file sizes, and Galaxy requires an integer +`size`, so with `fetch_sizes=True` (the default) a listing fetches each file's record to fill +it in; `fetch_sizes=False` keeps a listing at one call. Under PyFilesystem this was decided per +call by the requested namespace; fsspec's `ls` carries no such hint, so it is per instance. +Upload routing: the Gallery is split into media-type sections and a file may only go into +a folder of the matching section; `upload_fileobj` returns a `Placement`, and a mismatch either raises `GallerySectionMismatch` or, with +`on_mismatch="reroute"`, lets RSpace file it and reports where it landed. + +**Inventory.** Root lists benches (read via `/workbenches/{id}`, since `/containers/{id}` +answers 422 for a bench) and the sections. A container lists its child containers and +subsamples plus attachments; a sample lists its subsamples plus attachments; a subsample lists +its attachments plus a `sample: ` shortcut folder holding the sample's attachments only, +so the tree has no cycles. Samples are not stored in containers, their subsamples are. + +Records carry files in two disjoint places: their own `attachments`, and template-defined +attachment fields (`SF`) on samples, templates and instruments, each holding at most one file. +Only `attachment`-type fields are shown, as folders of 0 or 1 file. Uploading into an occupied +field is refused with `FileExistsError` rather than replacing, because a folder reads as +something you add to. Subsamples have attachments but no fields (84 live ones checked). + +Where a file can go, verified live against RSpace 2.27 and refused client-side before any +upload where the server would refuse: + +| Record | Own attachments | Attachment fields | +| --- | --- | --- | +| Sample | read and write | read and write where the template defines one | +| Subsample, container | read and write | none exist | +| Instrument | read and write | where the template defines one | +| Sample template | server refuses | browsable | +| Bench | server refuses | none exist | + +An uploaded attachment becomes a **Gallery file** linked to the record, which is what the web +interface's "link from Gallery" produces. The alternative the API offers creates a file that +exists only inside Inventory and is reusable nowhere; `via_gallery=False` keeps it, and the +deprecated `InventoryAttachmentFilesystem` is pinned to it. Linking is also what a copy within +RSpace means: `RSpaceFilesystem.copy` from any Gallery-backed source into an Inventory record +or an ELN field calls the link endpoint instead of downloading and re-uploading, which used to +leave a permanent duplicate behind. + +**Workspace.** The Home tree with the Gallery root and the Templates system +folder hidden. A document exposes its fields as sub-folders, using the field's own name; the +server's `ApiDocumentField` already supplies each field's linked `files`, so no HTML parsing is +needed to list them. Only `text` fields are shown: the form editor offers no other type that +holds files, every caller of `Field.addMediaFileLink` in rspace-web passes a text field, and the +legacy `Attachment` type is uncreatable and always empty. The document's entry counts the +omitted fields in `rspace.hiddenFields`. Uploading puts the bytes in the Gallery and appends a +`` token to that field; the server rewrites the token into an `attachmentDiv` or an +`` on save, and unlinking strips exactly those forms with BeautifulSoup so a field id is +never mistaken for a file id. `makedir` creates folders only. Only the user's own Home tree is +reachable; shared and group documents are not in it. + +The same Gallery file can appear at several paths: under `/gallery` and under every +field it is linked into. The namespace is a set of overlapping views, not a tree of uniquely +located files. + +## 5. Listings and pagination + +Every RSpace listing call already returns `name`, `globalId` and timestamps per child, so each +branch serves `ls` and `scandir` from one listing call rather than one `info` per child, and +fills `size`, `created` and `mtime` as epoch seconds. Entries follow fsspec's convention: +`name` is the full path without a leading slash. The base class implements `ls` itself rather +than relying on fsspec's `_ls_from_cache` and default `info`, which return a directory's own +entry among its children and fall back to listing a path to describe it. fsspec's listings +cache is off by default and cleared on every write. + +Section and folder listings are lazy streams that follow the API's `next` links only as far +as the consumer reads. `page_size` is the fetch batch size, default 100, invisible in the +namespace. This replaced `page-N` pseudo-folders after seeing them in Galaxy's file browser, +where they read as data rather than as an implementation detail, and Galaxy paginates +listings itself (after a full `ls`), so page folders had it paginating over our pagination. + +## 6. Errors + +RSpace client exceptions are translated once, at every public method of the base class (and +of every subclass, which `__init_subclass__` wraps), into builtin `OSError` subclasses, which +is what fsspec's own `exists`, `walk` and `copy`, and Galaxy, understand: a 404 becomes +`FileNotFoundError`, 401 and 403 `PermissionError`, a lost connection or a 502 to 504 +`ConnectionError`. Every other API refusal, a 500 with a message included, becomes +`RemoteApiError`, an `OSError` that keeps the server's explanation in its message and the +original exception as `api_error`. `GallerySectionMismatch` passes through as it is, since its +message already says what to do. + +## 7. Opener + +`fsspec.filesystem("rspace", server=...)` and `rspace://host/...` URLs (`fsspec.open`, +`UPath`) return an `RSpaceFilesystem`, through an `fsspec.specs` entry point, so nothing has to +be imported first. The URL carries the host and options only. The key is read from +`RSPACE_API_KEY` and nothing else: a URL is often supplied by someone other than the person +whose environment it is opened in, so it must not be able to name the variable that leaves the +machine. User info in the URL is refused. Plain http is accepted for loopback hosts only. The +client never follows a redirect, because requests keeps the `apiKey` header across a change of +host. + +## 8. Galaxy, the load-bearing consumer + +Galaxy pins `rspace-client>=2.6.1,<3` and ships a file source (`galaxy/files/sources/rspace.py`, +since Galaxy 25.1) that **subclasses the deprecated** `rspace_client.eln.fs.GalleryFilesystem`, +monkeypatches `ELNClient.upload_file` with a function taking only `(file, folder_id, caption)`, +and passes it a `FakedNameIO` wrapper that forwards `read` through `__getattr__`. It relies on +five things no documented API promises: + +| Galaxy relies on | Kept by | +| --- | --- | +| `upload_file` accepting no extra keyword | the shim sends no `filename=`; only `rspace_client.fs` does | +| `upload(folder_path, file)` naming the file after the file object | the shim's `upload`, unchanged | +| bare global-ID segments, from which it builds URIs | the shim, unchanged | +| `basic.name`, `rspace.name`, ISO `rspace.created`, integer `size` | the shim's `getinfo`, unchanged | +| a file part that only forwards `read` being accepted | `_post_multipart` wraps it, see below | + +The last row is new with requests 2.34 (pinned by Galaxy 26.1): requests now checks that a +multipart file part defines `read` on its class, which `FakedNameIO` does not, so Galaxy's +export to RSpace fails with a `TypeError` whatever rspace-client is installed. The client now +wraps such an object in one that defines `read`, which fixes the shipped plugin from the +client side. + +All five are pinned by `rspace_client/tests/galaxy_plugin_contract_test.py`, which copies +Galaxy's wrapper classes so the suite runs without Galaxy installed. **A failure there is a +release blocker, not a test to update.** The shims (`rspace_client.eln.fs`, which Galaxy +imports, and `rspace_client.inv.attachment_fs`, which it does not but 2.x users may) are the +2.7 PyFilesystem classes, with their section-routing helpers now imported +from `rspace_client.fs.gallery` and a `DeprecationWarning` on construction. They cannot be +removed on a schedule: removal follows a Galaxy release that no longer imports them, no +earlier than 3.0, together with the `fs` dependency and the `setuptools` pin. + +Galaxy discovers file sources by walking its own package, so exposing `/inventory` and +`/workspace` to Galaxy users needs a pull request to `galaxyproject/galaxy`: +`tools/galaxy/rspace_fsspec_source.py` is the proposed file source, built on Galaxy's +`FsspecFilesSource`. It also needs a file source template type, since Galaxy's template catalog +rejects unknown types, and a raised `rspace-client` pin. Pointing the existing `rspace` type at +it with `mounts=gallery` and `path_style="id"` would keep saved URIs working and retire the +deprecated import. That pull request depends on a released `rspace-client` that contains this +package. diff --git a/docs/usage-guide.md b/docs/usage-guide.md index 4f2b9a4..671c05d 100644 --- a/docs/usage-guide.md +++ b/docs/usage-guide.md @@ -9,7 +9,7 @@ bash> export RSPACE_URL=https://myrspace.com bash> export RSPACE_API_KEY=abcdefgh... ``` -- [Using the library as a PyFilesystem implementation](#using-the-rspace_client-library-as-pyfilesystem-implementation) +- [Using RSpace as a filesystem (fsspec)](#using-rspace-as-a-filesystem-fsspec) - [A basic query to list documents](#a-basic-query-to-list-documents) - [Iterating over pages of results](#iterating-over-pages-of-results) - [Searching](#searching) @@ -24,93 +24,124 @@ bash> export RSPACE_API_KEY=abcdefgh... - [Forms](#forms) - [Export](#export) -## Using the rspace_client library as PyFilesystem implementation +## Using RSpace as a filesystem (fsspec) -Starting with rspace-client 2.6.0 rspace-client the library partially implements -[PyFilesystem](https://docs.pyfilesystem.org/en/latest/index.html) API for accessing -RSpace Gallery files and RSpace Inventory attachments. +`rspace_client.fs` presents RSpace as an [fsspec](https://filesystem-spec.readthedocs.io) +filesystem, the same interface pandas, pyarrow, dask, zarr, `universal_pathlib` and Galaxy +use for remote storage. One mount holds three branches: -To start, export the URL and API key into environment variables (as explained before). +``` +gallery/ Gallery folders and files +inventory/ benches and the sections Containers, Samples, Templates; every record is a + folder holding its child records and its attachments +workspace/ ELN folders, notebooks and documents; a document is a folder of its text + fields, a field is a folder of the files linked into it +``` + +A directory is anything that contains files; a file is a downloadable blob. Paths are the +records' own names, without a leading slash (`gallery/Images/microscope.png`). Every entry +returned by `ls()` or `info()` is a dict with `name`, `type`, `size`, `mtime`, `created`, +`globalId` and `rspace` (the raw RSpace record). -To access the Gallery files construct a `GalleryFilesystem` object. ```python -from rspace_client.eln import fs +from rspace_client.fs import RSpaceFilesystem, print_tree -# create rspace_fs object which supports generic fs methods (listdir, getinfo etc.) -rspace_gallery_fs = fs.GalleryFilesystem(os.getenv("RSPACE_URL"), os.getenv("RSPACE_API_KEY")) +fs = RSpaceFilesystem(os.getenv("RSPACE_URL"), os.getenv("RSPACE_API_KEY")) -content = rspace_gallery_fs.listdir("/") -print(content) -for globalId in content: - print(rspace_gallery_fs.getinfo(globalId).raw) +fs.ls("", detail=False) # ['gallery', 'inventory', 'workspace'] +fs.ls("gallery/Images", detail=False) # ['gallery/Images/microscope.png', ...] +fs.info("gallery/Images/microscope.png") # {'name': ..., 'type': 'file', 'size': 1096, 'globalId': 'GL100', ...} +data = fs.cat_file("gallery/Images/microscope.png") +fs.get_file("inventory/Containers/Freezer -80/manual.pdf", "manual.pdf") +print_tree(fs, "workspace/Project Alpha", max_depth=3) ``` -#### Uploading into Gallery folders and the media-type section rule - -The RSpace Gallery is split into media-type sections (Images, Documents, -Chemistry, ...) and a file can only be placed in a folder whose section matches -the file's media type. Uploading, say, a PDF into a folder that lives in the -Images section is rejected by the server. - -By default `upload` turns that rejection into a clear `GallerySectionMismatch` -that names the folder's section, rather than an opaque API error: +With `RSPACE_API_KEY` in the environment the filesystem also opens from a URL. The URL +carries the host and options only, never the key: ```python -from rspace_client.eln.fs import GallerySectionMismatch +import fsspec +from upath import UPath -try: - rspace_gallery_fs.upload("/GF123", file_obj) # GF123 is in the Images section -except GallerySectionMismatch as e: - print(e) # explains the section clash - print(e.folder_section) # e.g. "Images" +fs = fsspec.filesystem("rspace", server="https://my.rspace.host") +with fsspec.open("rspace://my.rspace.host/gallery/Documents/data.csv") as f: + ... +for entry in UPath("rspace://my.rspace.host/inventory/Samples").iterdir(): + print(entry.name) ``` -Alternatively, opt in to automatic rerouting. On a mismatch the file is placed -in the correct section's inbox instead of failing. Set the policy once when -constructing the filesystem (it then applies to every write, including generic -PyFilesystem operations), or override it per call: +Plain http is accepted for loopback hosts only (`rspace://localhost:8080/...?scheme=http`). + +### Writing + +Everything is read-only by default. `writable=True` allows uploads and folder creation, +`allow_delete=True` allows removals; the two are separate on purpose. ```python -# filesystem-wide default -rspace_gallery_fs = fs.GalleryFilesystem(url, api_key, on_mismatch="reroute") +fs = RSpaceFilesystem(url, api_key, writable=True, allow_delete=True) -# or per upload -placement = rspace_gallery_fs.upload("/GF123", file_obj, on_mismatch="reroute") +with fs.open("gallery/Documents/notes.txt", "wb") as f: # uploads when closed + f.write(b"...") +fs.put_file("results.csv", "workspace/Project Alpha/Alpha protocol/Results/results.csv") +fs.mkdir("gallery/Images/2026-10 run") +fs.rm_file("inventory/Samples/Plasmid pUC19/old_gel.png") # deletes the attachment +fs.rm_file("workspace/Project Alpha/Alpha protocol/Results/old.csv") # unlinks; the Gallery file stays ``` -`upload` returns a `Placement` telling you where the file actually landed: +What a write means depends on the branch: a file put into a Gallery folder is a Gallery +upload; into an Inventory record it is uploaded to the Gallery and attached to the record; +into a document field it is uploaded to the Gallery and linked into that field. A copy from +any Gallery-backed file into Inventory or a document field (`fs.cp_file`) links the existing +file instead of moving bytes. Transactions defer uploads until the block ends: ```python -placement.rerouted # True if it did not go in the requested folder -placement.section # e.g. "Documents" -placement.path # e.g. "Gallery/Documents/Api Inbox" -placement.file_global_id # e.g. "GL999" +with fs.transaction: + with fs.open("gallery/Documents/a.txt", "wb") as f: f.write(b"a") + with fs.open("gallery/Documents/b.txt", "wb") as f: f.write(b"b") +# both uploaded here; neither if the block raised ``` -Note that rerouting places the file in the section's inbox, not a subfolder -matching the one you requested. +A signed ELN document is locked and its fields are listed with a `(signed)` suffix. An +Inventory attachment field holds one file and refuses an upload while occupied. A bench or +sample template takes no attachments, and the Gallery API cannot delete Gallery files; each +refusal says so before anything is sent. -To access Inventory attachments construct a `InventoryAttachmentFilesystem` object. -```python -from rspace_client.inv.attachment_fs import InventoryAttachmentFilesystem - -rspace_inv_fs = InventoryAttachmentFilesystem(os.getenv("RSPACE_URL"), os.getenv("RSPACE_API_KEY")) +#### Names, duplicates and global IDs -# list the attachments that an existing record has, in this case a subsample -rspace_inv_fs.listdir('/SS123') +Where two records in a folder share a name, the second and later ones are listed with their +global ID before the extension (`data [GL112].csv`), and a name that would itself read as an +address gets its real ID the same way. A segment carrying a global ID always resolves to that +record, whatever the style, and survives a rename. `path_style="labelled"` lists every +segment as `Name [GID]`; `path_style="id"` lists bare global IDs. -# get the metadata for a particular attachment -rspace_inv_fs.getinfo("IF123") +#### Gallery sections -# or download it -file_obj = BytesIO() -rspace_inv_fs.download('/IF123', file_obj) +The Gallery is split into media-type sections (Images, Documents, Chemistry, ...) and a file +can only go into a folder of the matching section. A mismatch raises +`GallerySectionMismatch`, which names the folder's section; `on_mismatch="reroute"` (on the +filesystem or per `upload_fileobj` call) lets RSpace file it in the right section instead +and the returned `Placement` says where it landed. -# or attach a new file -file_obj_2 = BytesIO(b'test file content') -rspace_inv_fs.upload('/SS123', file_obj_2) +```python +placement = fs.upload_fileobj("gallery/Images/report.pdf", file_obj, on_mismatch="reroute") +placement.rerouted, placement.section, placement.path, placement.file_global_id ``` +#### Galaxy + +Galaxy's file sources are built on fsspec. `tools/galaxy/rspace_fsspec_source.py` is the +reference plugin for `RSpaceFilesystem`; the shipped Galaxy plugin still uses the +deprecated classes below. + +### Deprecated: the PyFilesystem classes + +`rspace_client.eln.fs.GalleryFilesystem` and +`rspace_client.inv.attachment_fs.InventoryAttachmentFilesystem` implement the unmaintained +[PyFilesystem2](https://github.com/PyFilesystem/pyfilesystem2) API with global IDs as path +segments (`/GF123`, `/IF123`). They are kept unchanged for the Galaxy plugin that imports +them, emit a `DeprecationWarning`, and will be removed in rspace-client 3.0 together with +the `fs` dependency. New code should use `rspace_client.fs`. + ## A basic query to list documents First of all we'll get our URL and key from a command-line parameters. diff --git a/pyproject.toml b/pyproject.toml index 65b7b84..c790eff 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -24,14 +24,20 @@ exclude = ["examples/*", "rspace_client/tests/*"] python = "^3.9" requests = "^2.25.1" beautifulsoup4 = "^4.9.3" -fs = "^2.4.16" -setuptools = "<82" +fs = "^2.4.16" # deprecated PyFilesystem2 classes only (rspace_client.eln.fs, rspace_client.inv.attachment_fs); removed in 3.0 +setuptools = "<82" # fs 2.4.16 imports pkg_resources; goes with fs +fsspec = ">=2024.3.0" # rspace_client.fs + +# lets fsspec.filesystem("rspace") and rspace:// URLs work before rspace_client.fs is imported +[tool.poetry.plugins."fsspec.specs"] +rspace = "rspace_client.fs:RSpaceFilesystem" [tool.poetry.group.dev.dependencies] python-dotenv = "^1.1.1" black = "^21.6b0" pytest = "^8.0.0" responses = "^0.26.2" +universal-pathlib = ">=0.2" [build-system] requires = ["poetry-core>=1.0.0"] diff --git a/rspace_client/client_base.py b/rspace_client/client_base.py index 45e4e03..bb4a639 100644 --- a/rspace_client/client_base.py +++ b/rspace_client/client_base.py @@ -1,3 +1,4 @@ +import inspect import logging import re import requests @@ -25,6 +26,35 @@ REDIRECT_STATUSES = (301, 302, 303, 307, 308) +class _ReadForwarder: + """ + Gives requests a file object it recognises. Since requests 2.34, a multipart file part + must define ``read`` on its class: on Python 3.12+ the check skips ``__getattr__``, so + a wrapper that forwards ``read`` dynamically (Galaxy's ``FakedNameIO``, which Galaxy's + RSpace file source passes to ``ELNClient.upload_file``) is refused with a TypeError. + """ + + def __init__(self, file): + self._file = file + self.name = getattr(file, "name", None) # requests names the part after it + + def read(self, *args): + return self._file.read(*args) + + +def _readable_part(part): + """A multipart file part, with a file object that only reads dynamically wrapped.""" + if isinstance(part, (tuple, list)) and len(part) >= 2: + return (part[0], _readable_part(part[1])) + tuple(part[2:]) + if ( + not isinstance(part, (str, bytes, bytearray)) + and inspect.getattr_static(part, "read", None) is None + and callable(getattr(part, "read", None)) + ): + return _ReadForwarder(part) + return part + + class _RSpaceRetry(Retry): """ Retry policy that additionally retries POST requests, but only on @@ -296,6 +326,8 @@ def _post_multipart(self, endpoint, files=None, data=None): url = endpoint if not self._is_absolute_url(endpoint): url = self._get_api_url() + endpoint + if files: + files = {key: _readable_part(part) for key, part in files.items()} try: response = self._session.post( url, diff --git a/rspace_client/eln/fs.py b/rspace_client/eln/fs.py index 060c098..0f8cbca 100644 --- a/rspace_client/eln/fs.py +++ b/rspace_client/eln/fs.py @@ -1,5 +1,5 @@ import logging -from dataclasses import dataclass +import warnings from fs.base import FS from rspace_client.eln import eln from rspace_client.client_base import ClientBase @@ -14,61 +14,11 @@ logger = logging.getLogger(__name__) -# Accepted values for the GalleryFilesystem ``on_mismatch`` policy. -ON_MISMATCH_RAISE = "raise" -ON_MISMATCH_REROUTE = "reroute" -_ON_MISMATCH_VALUES = (ON_MISMATCH_RAISE, ON_MISMATCH_REROUTE) - -# The RSpace Gallery's media-type sections. "Miscellaneous" is the catch-all -# that accepts any file not matching a more specific section; the others accept -# only their listed extensions (including "Documents", which is a fixed set, not -# a catch-all). -MISCELLANEOUS_SECTION = "Miscellaneous" - -# Best-effort mapping of file extension to the Gallery section RSpace classifies -# it into, taken from the Gallery documentation. This mirrors the server's own -# classification but is NOT authoritative: it is used only to phrase error -# messages, never to decide whether an upload is allowed. Any extension not -# listed falls through to "Miscellaneous", matching the server default. -_SECTION_BY_EXTENSION = { - # Images - "png": "Images", "jpg": "Images", "jpeg": "Images", "gif": "Images", - "bmp": "Images", "tif": "Images", "tiff": "Images", - # Audios - "mp3": "Audios", "wav": "Audios", "wma": "Audios", "aac": "Audios", - "ogg": "Audios", - # Videos - "mp4": "Videos", "mov": "Videos", "hdmov": "Videos", "m4v": "Videos", - "wmv": "Videos", "avi": "Videos", "mpg": "Videos", "mpeg": "Videos", - "flv": "Videos", "3gp": "Videos", - # Documents (a fixed set, NOT a catch-all) - "doc": "Documents", "docx": "Documents", "rtf": "Documents", - "pdf": "Documents", "odt": "Documents", "ods": "Documents", - "odp": "Documents", "txt": "Documents", "ppt": "Documents", - "pptx": "Documents", "xls": "Documents", "xlsx": "Documents", - "csv": "Documents", "pps": "Documents", "md": "Documents", - # Chemistry (documented subset; the server accepts more) - "skc": "Chemistry", "mrv": "Chemistry", "cxsmiles": "Chemistry", - "cxsmarts": "Chemistry", "cdx": "Chemistry", "cdxml": "Chemistry", - "csrdf": "Chemistry", "cml": "Chemistry", -} - - -def classify_media_section(filename: Optional[Text]) -> Optional[Text]: - """ - Best-effort guess of the Gallery section for a filename, mirroring RSpace's - server-side classification. Returns a section name (e.g. "Images"), - "Miscellaneous" as the catch-all when the extension is not one of a specific - section's types, or None when no filename/extension is available to guess - from. - - The server remains the authority on placement; this is only used to make - error messages and log lines more helpful. - """ - if not filename or "." not in filename: - return None - ext = filename.rsplit(".", 1)[-1].lower() - return _SECTION_BY_EXTENSION.get(ext, MISCELLANEOUS_SECTION) +# The section policy, the classification table, the exception and the Placement record +# are shared with the fsspec filesystem that replaces this module. +from rspace_client.fs.gallery import (MISCELLANEOUS_SECTION, ON_MISMATCH_RAISE, ON_MISMATCH_REROUTE, # noqa: E402,F401 + GallerySectionMismatch, Placement, check_policy, classify_media_section, folder_section, + mismatch_message, placement) def _filename_for(file: BinaryIO, options: Mapping[Text, Any]) -> Optional[Text]: @@ -81,37 +31,15 @@ def _filename_for(file: BinaryIO, options: Mapping[Text, Any]) -> Optional[Text] return name if isinstance(name, str) else None -class GallerySectionMismatch(ClientBase.ApiError): - """ - Raised when a file could not be uploaded into the requested Gallery folder - because that folder belongs to a media-type section that does not accept the - file. Carries the folder's section and, when it could be guessed, the file's - media type, so callers can react programmatically as well as read the message. - """ - - def __init__(self, message, *, folder_section=None, folder_global_id=None, - file_media_type=None, response_status_code=None): - super().__init__(message, response_status_code=response_status_code) - self.folder_section = folder_section - self.folder_global_id = folder_global_id - self.file_media_type = file_media_type - - -@dataclass -class Placement: - """Where an uploaded file actually ended up in the Gallery. - - Returned by :meth:`GalleryFilesystem.upload`. When ``rerouted`` is True the - file did not land in the folder named by ``requested_path`` (its section did - not accept the file) and was placed in the correct section instead; - ``path`` reports the human-readable location it ended up in. - """ - file_global_id: Optional[Text] - folder_global_id: Optional[Text] - section: Optional[Text] - path: Text - rerouted: bool - requested_path: Optional[Text] = None +def _with_parsable_created(info: Mapping[Text, Any]) -> Mapping[Text, Any]: + """RSpace dates end in ``Z`` for UTC. Galaxy's ``rspace`` file source parses ``created`` + with ``datetime.fromisoformat``, which rejects ``Z`` before Python 3.11, so on 3.10 every + file listing failed. Spell the same instant as ``+00:00``; the rest of the record is + passed through untouched.""" + created = info.get("created") + if isinstance(created, str) and created.endswith("Z"): + return {**info, "created": created[:-1] + "+00:00"} + return info def is_folder(path): @@ -127,6 +55,10 @@ def __init__(self, obj, *args, **kwargs) -> None: class GalleryFilesystem(FS): def __init__(self, server: str, api_key: str, on_mismatch: str = ON_MISMATCH_RAISE) -> None: + warnings.warn( + "rspace_client.eln.fs.GalleryFilesystem (PyFilesystem2) is deprecated and will be removed " + "in rspace-client 3.0; use rspace_client.fs.GalleryFilesystem or RSpaceFilesystem (fsspec)", + DeprecationWarning, stacklevel=2) """ :param server: RSpace server URL :param api_key: RSpace API key @@ -139,11 +71,7 @@ def __init__(self, server: str, api_key: str, on_mismatch: str = ON_MISMATCH_RAI :meth:`upload` calls may override it. """ super(GalleryFilesystem, self).__init__() - if on_mismatch not in _ON_MISMATCH_VALUES: - raise ValueError( - "on_mismatch must be one of {}".format(_ON_MISMATCH_VALUES) - ) - self.on_mismatch = on_mismatch + self.on_mismatch = check_policy(on_mismatch) self.eln_client = eln.ELNClient(server, api_key) self.gallery_id = next(file['id'] for file in self.eln_client.list_folder_tree()['records'] if file['name'] == 'Gallery') @@ -165,7 +93,7 @@ def getinfo(self, path, namespaces=None) -> Info: "size": info.get('size', 0), "type": is_folder(path) and "1" or "2", }, - "rspace": info, + "rspace": _with_parsable_created(info), }) def listdir(self, path: Text) -> List[Text]: @@ -232,63 +160,6 @@ def download(self, path: Text, file: BinaryIO, chunk_size: Optional[int] = None, else: self.eln_client.download_file(path_to_id(path), file) - def _folder_section(self, folder_id: Text) -> Optional[Text]: - """The Gallery section (mediaType) a folder belongs to, or None if it - cannot be determined. Best-effort: never raises, so it cannot mask the - real outcome of an upload.""" - try: - return self.eln_client.get_folder(folder_id).get("mediaType") - except Exception: - return None - - def _human_path(self, folder: Mapping[Text, Any]) -> Text: - """Best-effort readable path for a folder, e.g. 'Gallery/Documents/Api - Inbox'. Uses the API's pathToRootFolder when present, otherwise falls - back to the section and folder name.""" - trail = folder.get("pathToRootFolder") - if isinstance(trail, list) and trail: - names = [f.get("name") for f in trail if f.get("name")] - if names: - return "/".join(names) - parts = ["Gallery"] - if folder.get("mediaType"): - parts.append(folder["mediaType"]) - if folder.get("name"): - parts.append(folder["name"]) - return "/".join(parts) - - def _placement(self, response: Any, requested_path: Optional[Text], - rerouted: bool) -> Placement: - """Build a Placement from an upload response, resolving the parent - folder for section/path feedback where possible. - - Tolerates a response that is not a dict (e.g. None): some callers wrap - or replace ``eln_client.upload_file`` and discard its return value, so - Placement construction must never crash a successful upload. - """ - if not isinstance(response, dict): - response = {} - parent_id = response.get("parentFolderId") - section = None - folder_global_id = None - path = "Gallery" - if parent_id is not None: - try: - folder = self.eln_client.get_folder(parent_id) - section = folder.get("mediaType") - folder_global_id = folder.get("globalId") - path = self._human_path(folder) - except Exception: - pass - return Placement( - file_global_id=response.get("globalId"), - folder_global_id=folder_global_id, - section=section, - path=path, - rerouted=rerouted, - requested_path=requested_path, - ) - def upload(self, path: Text, file: BinaryIO, chunk_size: Optional[int] = None, on_mismatch: Optional[str] = None, **options: Any) -> Placement: """ @@ -309,19 +180,13 @@ def upload(self, path: Text, file: BinaryIO, chunk_size: Optional[int] = None, section (``"raise"``) or places the file in the correct section's inbox and returns a Placement with ``rerouted=True`` (``"reroute"``). """ - policy = on_mismatch if on_mismatch is not None else self.on_mismatch - if policy not in _ON_MISMATCH_VALUES: - raise ValueError( - "on_mismatch must be one of {}".format(_ON_MISMATCH_VALUES) - ) + policy = check_policy(self.on_mismatch if on_mismatch is None else on_mismatch) folder_id = path_to_id(path) if path else None try: response = self.eln_client.upload_file(file, folder_id) - return self._placement(response, requested_path=path or None, rerouted=False) + return placement(self.eln_client, response, requested_path=path or None, rerouted=False) except ClientBase.ApiError as err: - if folder_id is None: - raise - section = self._folder_section(folder_id) + section = None if folder_id is None else folder_section(self.eln_client, folder_id) if section is None: raise filename = _filename_for(file, options) @@ -333,39 +198,14 @@ def upload(self, path: Text, file: BinaryIO, chunk_size: Optional[int] = None, except (AttributeError, OSError, ValueError): pass response = self.eln_client.upload_file(file, None) - placement = self._placement(response, requested_path=path, rerouted=True) + placed = placement(self.eln_client, response, requested_path=path, rerouted=True) logger.info( - "RSpace Gallery: %s could not go in %s (section '%s'); " - "placed in %s instead", - "'{}'".format(filename) if filename else "file", - path, section, placement.path, - ) - return placement + "RSpace Gallery: %s could not go in %s (section '%s'); placed in %s instead", + "'{}'".format(filename) if filename else "file", path, section, placed.path) + return placed - named = "'{}'".format(filename) if filename else "the file" - message = ( - "Could not upload {named} to Gallery folder {path}. That folder " - "is in the '{section}' section, which only accepts {section} " - "files".format(named=named, path=path, section=section) - ) - if guessed and guessed != section: - if guessed == MISCELLANEOUS_SECTION: - message += ( - ", but {named} does not match a specialised section and " - "belongs in 'Miscellaneous'".format(named=named) - ) - else: - message += ", but {named} looks like a '{guessed}' file".format( - named=named, guessed=guessed - ) - message += ( - ". Upload it to a folder in the matching section, omit the " - "folder path to let RSpace place it in the correct section " - "automatically, or construct the filesystem with " - "on_mismatch='reroute'. Original API error: {err}".format(err=err) - ) raise GallerySectionMismatch( - message, + mismatch_message("'{}'".format(filename) if filename else "the file", path, section, guessed, err), folder_section=section, folder_global_id="GF" + str(folder_id), file_media_type=guessed, diff --git a/rspace_client/fs/__init__.py b/rspace_client/fs/__init__.py new file mode 100644 index 0000000..7453013 --- /dev/null +++ b/rspace_client/fs/__init__.py @@ -0,0 +1,48 @@ +""" +fsspec (https://filesystem-spec.readthedocs.io) implementations for RSpace. + + from rspace_client.fs import GalleryFilesystem, InventoryFilesystem, WorkspaceFilesystem + from rspace_client.fs import RSpaceFilesystem # one mount: gallery/, inventory/, workspace/ + + import fsspec # key from RSPACE_API_KEY: + fsspec.filesystem("rspace", server="https://my.rspace.host") + fsspec.open("rspace://my.rspace.host/gallery/Images/microscope.png").open().read() + +All filesystems are read-only by default. Pass ``writable=True`` to allow uploads and +folder creation, and ``allow_delete=True`` to allow removals. ``mounts=`` narrows what +``RSpaceFilesystem`` exposes, for a caller that only wants part of RSpace. + +A path segment is the record's own name (``gallery/Images/microscope.png``). Where two +records in a folder share a name, the later ones carry their global ID before the extension +(``data [GL112].csv``). A segment written as a global ID always resolves, whatever style is +in force, and ``path_style="labelled"`` or ``"id"`` put one on every segment, which is what +you want if a stored path has to survive a rename. The full RSpace record for any entry is +under the ``rspace`` key of its info dict. +""" +import fsspec + +from .base import RSpaceFSBase, ReadOnlyError, RemoteApiError, make_entry +from .gallery import GalleryFilesystem, GallerySectionMismatch, Placement +from .inventory import InventoryFilesystem +from .workspace import WorkspaceFilesystem +from .rspace import RSpaceFilesystem +from .tree import format_tree, print_tree + +# also declared as an ``fsspec.specs`` entry point in pyproject.toml, so that +# ``fsspec.filesystem("rspace")`` works before this package was imported +fsspec.register_implementation("rspace", RSpaceFilesystem, clobber=True) + +__all__ = [ + "RSpaceFSBase", + "ReadOnlyError", + "RemoteApiError", + "make_entry", + "GalleryFilesystem", + "GallerySectionMismatch", + "Placement", + "InventoryFilesystem", + "WorkspaceFilesystem", + "RSpaceFilesystem", + "format_tree", + "print_tree", +] diff --git a/rspace_client/fs/base.py b/rspace_client/fs/base.py new file mode 100644 index 0000000..84482a8 --- /dev/null +++ b/rspace_client/fs/base.py @@ -0,0 +1,749 @@ +""" +Shared base class for the RSpace fsspec filesystems. + +Provides, so that every branch (Gallery, Inventory, Workspace) behaves the same: + +* a path style: ``name`` shows a record's own name, ``labelled`` shows ``Name [GID]`` + and ``id`` shows bare global IDs. A segment carrying a global ID resolves under + every style, in one call; a *bare name* resolves only under ``name``, through + ``_lookup``, which matches it against the parent's listing; +* a write posture: read-only by default, ``writable=True`` enables uploads and + folder creation, ``allow_delete=True`` enables removals; +* ``entry`` dicts in fsspec's shape (``name`` is the full path without a leading + slash, ``type`` is ``file`` or ``directory``, ``size``, ``mtime`` and ``created`` are + epoch seconds) plus ``globalId``, ``rspace`` (the raw RSpace record) and, where RSpace + tells us, ``writable``; +* ``ls``/``scandir``, which subclasses feed from a single listing call via ``_children`` + instead of one ``info`` request per child, and which keep the segments in a listing + unique and self-addressing even when two records share a name (see ``_listing``); +* ``stream_pages``, which turns any paginated RSpace response into a lazy stream of + records by following its ``next`` links only as far as the consumer reads; +* file I/O: reads buffer the whole file (RSpace serves no byte ranges), writes buffer + until the file is closed or committed and upload in one request, which is what makes + fsspec transactions work; +* error mapping: RSpace client exceptions become builtin ``OSError`` subclasses at every + public method, so ``exists``, ``walk``, ``copy`` and Galaxy behave; +* a ``glob`` that understands the ``/**`` pattern Galaxy's search sends, with + the escaping Galaxy applies, so that a segment carrying ``[GL83]`` can be found. +""" +from __future__ import annotations + +import errno +import functools +import io +import os +import re +from dataclasses import dataclass +from datetime import datetime, timezone +from typing import (Any, BinaryIO, Callable, Dict, Iterable, Iterator, List, Mapping, Optional, + Tuple) + +from fsspec.spec import AbstractBufferedFile, AbstractFileSystem + +from rspace_client.client_base import ClientBase +from rspace_client.exceptions import ApiError, AuthenticationError, RSpaceConnectionError + +from . import paths + + +# ------------------------------------------------------------------ errors +# +# fsspec callers (``exists``, ``walk``, ``copy``, Galaxy's file source) expect ``OSError`` +# subclasses: ``FileNotFoundError`` for a missing path, ``PermissionError`` for a refusal, +# ``FileExistsError`` for a clash. The RSpace clients raise their own exceptions, so every +# public filesystem method translates them at the boundary, once, here. + + +class RemoteApiError(OSError): + """An RSpace API refusal that is neither "not found" nor "not allowed": a wrong Gallery + section (422), a refused link (409), a bad request (400). Carries the server's own + explanation in ``str()`` and the original exception as ``api_error``.""" + + def __init__(self, path: str, exc: Exception) -> None: + super().__init__(errno.EIO, f"{path!r}: {exc}", path) + self.api_error = exc + self.status = getattr(exc, "response_status_code", None) + + +class ReadOnlyError(PermissionError): + """A write was refused because of the filesystem's posture, not by the server.""" + + +def os_error_for(exc: Exception, path: str) -> Optional[OSError]: + """The ``OSError`` that stands for a client exception, or None to re-raise as it is.""" + if getattr(exc, "fs_passthrough", False): # e.g. GallerySectionMismatch: already explained + return None + if isinstance(exc, RSpaceConnectionError): + return ConnectionError(f"{path!r}: {exc}") + if isinstance(exc, AuthenticationError): + return PermissionError(errno.EACCES, f"{path!r}: {exc}", path) + status = getattr(exc, "response_status_code", None) + if status == 404: + return FileNotFoundError(errno.ENOENT, f"{path!r}: {exc}", path) + if status in (401, 403): + return PermissionError(errno.EACCES, f"{path!r}: {exc}", path) + if status in (502, 503, 504): # the server or a proxy in front of it is not answering + return ConnectionError(f"{path!r}: {exc}") + if isinstance(exc, ApiError): + # a 500 included: RSpace answers one for a request it understood but could not + # carry out (an invalid image, say), and the message explains why + return RemoteApiError(path, exc) + return None + + +#: Public methods whose first argument is a path and which may call the server. +_TRANSLATED_METHODS = ("ls", "scandir", "info", "cat_file", "get_file", "put_file", "_open", + "download_fileobj", "upload_fileobj", "mkdir", "makedirs", "rmdir", + "rm_file", "link", "glob", "cp_file", "mv", "created", "modified") + + +_CLIENT_ERRORS = (ApiError, AuthenticationError, RSpaceConnectionError) + + +def _reraise(exc: Exception, path: str): + translated = os_error_for(exc, path) + if translated is None: + raise exc + raise translated from exc + + +def _translating_iterator(iterator: Iterator, path: str) -> Iterator: + try: + yield from iterator + except _CLIENT_ERRORS as exc: + _reraise(exc, path) + + +def translate_api_errors(method): + """Wrap a filesystem method so RSpace client errors surface as ``OSError``.""" + if getattr(method, "_translates_api_errors", False): + return method + + @functools.wraps(method) + def wrapper(self, path, *args, **kwargs): + try: + result = method(self, path, *args, **kwargs) + except _CLIENT_ERRORS as exc: + _reraise(exc, str(path)) + # a lazy listing fetches further pages while it is consumed; a file handle is + # iterable too but must be returned as it is + if isinstance(result, Iterator) and not hasattr(result, "read"): + return _translating_iterator(result, str(path)) + return result + + wrapper._translates_api_errors = True + return wrapper + + +def translate_class(cls) -> None: + """Apply ``translate_api_errors`` to the public methods ``cls`` itself defines.""" + for name in _TRANSLATED_METHODS: + if name in cls.__dict__: + setattr(cls, name, translate_api_errors(cls.__dict__[name])) + + +def writes(method): + """Mark a method that changes the server: it is refused unless ``writable=True``.""" + @functools.wraps(method) + def wrapper(self, path, *args, **kwargs): + self._require_writable(path) + return method(self, path, *args, **kwargs) + return wrapper + + +def deletes(method): + """Mark a method that removes something: it is refused unless ``allow_delete=True``.""" + @functools.wraps(method) + def wrapper(self, path, *args, **kwargs): + self._require_delete(path) + return method(self, path, *args, **kwargs) + return wrapper + + +# ------------------------------------------------------------------ entries + + +def to_epoch(value: Any) -> Optional[float]: + """Convert an RSpace timestamp (ISO-8601 string, or epoch millis/seconds) to epoch seconds.""" + if value is None or value == "": + return None + if isinstance(value, (int, float)): + return float(value) / 1000.0 if value > 1e11 else float(value) + text = str(value).strip() + if text.endswith("Z"): + text = text[:-1] + "+00:00" + try: + parsed = datetime.fromisoformat(text) + except ValueError: + return None + if parsed.tzinfo is None: + parsed = parsed.replace(tzinfo=timezone.utc) + return parsed.timestamp() + + +def make_entry(segment: str, is_dir: bool, raw: Optional[Mapping] = None, size: Optional[int] = None, + created: Any = None, modified: Any = None, writable: Optional[bool] = None) -> dict: + """Build an entry dict. ``segment`` is the entry's own path segment; ``ls`` prefixes the + parent path later. + + ``writable`` is set only where RSpace tells us something specific (a signed ELN document + is locked). Left None, the key is absent, which means "unknown" rather than a guess: a + record shared read-only with you is equally unwritable and looks no different from here. + """ + entry: Dict[str, Any] = { + "name": segment, + "type": "directory" if is_dir else "file", + "size": size, + # Galaxy and most fsspec consumers read mtime; RSpace files carry no lastModified, + # so created stands in for it there + "mtime": to_epoch(modified) if modified is not None else to_epoch(created), + "created": to_epoch(created), + "globalId": (raw or {}).get("globalId"), + "rspace": dict(raw) if raw else {}, + } + if writable is not None: + entry["writable"] = writable + return entry + + +@dataclass +class Target: + """What a path addresses, as worked out by a branch's ``_resolve``. + + ``kind`` is the branch's own vocabulary (``record``, ``field``, ``document``, ``file``...), + ``gid`` the global ID of the addressed record, ``parent_gid`` the record a field or file + belongs to (an Inventory record, an ELN document), ``field_gid`` the field a file was + reached through, and ``section`` the Inventory root section the path went through. + """ + kind: str + gid: Optional[str] = None + parent_gid: Optional[str] = None + field_gid: Optional[str] = None + section: Optional[str] = None + + +def client_or_new(client, factory, server: Optional[str], api_key: Optional[str], what: str): + """The client that was passed in, or a new one from ``server`` and ``api_key``.""" + if client is not None: + return client + if not server or not api_key: + raise ValueError(f"{what} needs server and api_key, or a client") + return factory(server, api_key) + + +def stream_pages(client: ClientBase, first_page: Mapping, key: str) -> Iterator[dict]: + """ + Stream the records of a paginated RSpace response. + + ``first_page`` is the first response, ``key`` the name of the collection inside it + ("records", "containers", "samples", ...). The next API page is fetched only when the + consumer reaches it, so a caller that stops after a handful of entries never pays for + the rest of the listing. + """ + page = first_page + while True: + yield from page.get(key, []) + # not client.link_exists(): that raises when a page carries no _links at all + if not any(link.get("rel") == "next" for link in page.get("_links") or []): + return + page = client.get_link_contents(page, "next") + + +# ------------------------------------------------------------------ files + + +class RSpaceWriteFile(AbstractBufferedFile): + """A file opened for writing: everything is buffered and sent in one request when the + file is closed (autocommit) or when the surrounding transaction commits. RSpace takes + one multipart body per upload, so there is nothing to send chunk by chunk.""" + + def _initiate_upload(self) -> None: + pass + + def _upload_chunk(self, final: bool = False) -> bool: + if final and self.autocommit: + self._send() + return False # keep the buffer; one body per upload + + def commit(self) -> None: + self._send() + + def discard(self) -> None: + self.buffer = io.BytesIO() + + def _send(self) -> None: + self.buffer.seek(0) + self.fs.upload_fileobj(self.path, self.buffer) + + +class RSpaceFSBase(AbstractFileSystem): + """Common behaviour for all RSpace filesystems. Subclasses implement the RSpace calls. + + Paths are fsspec paths: no protocol, no leading slash, the root is ``""``. + """ + + #: Instances hold an API key: never keep them in fsspec's process-wide instance cache. + cachable = False + root_marker = "" + #: How many resolved name lookups to remember before starting over. Deep trees browsed + #: top-down reuse the same handful of entries, so a small bound is plenty. + NAME_CACHE_LIMIT = 4096 + #: ``Target.kind`` values that are files: listing one raises ``NotADirectoryError``. + FILE_KINDS: Tuple[str, ...] = () + #: Field types (lower case) that can hold a file; other fields are not browsable. + FILE_BEARING_FIELD_TYPES: Tuple[str, ...] = () + #: False where the RSpace API cannot delete a file (the Gallery): a move is then refused + #: before anything is copied, since the second half could never happen. + CAN_DELETE_FILES = True + #: True where a Gallery file can be attached by reference (``link``) instead of copied. + CAN_LINK = False + + def __init_subclass__(cls, **kwargs) -> None: + super().__init_subclass__(**kwargs) + # every subclass gets the translation on the methods it defines or overrides, so a new + # branch cannot forget it + translate_class(cls) + + def __init__(self, *, writable: bool = False, allow_delete: bool = False, + path_style: str = "name", **storage_options: Any) -> None: + # fsspec caches directory listings by default and never expires them; this + # filesystem's own name cache is cleared on every write, so leave listings uncached + # unless the caller (Galaxy's cache options, say) asks for them + storage_options.setdefault("use_listings_cache", False) + super().__init__(**storage_options) + if path_style not in paths.PATH_STYLES: + raise ValueError(f"path_style must be one of {paths.PATH_STYLES}, got {path_style!r}") + self.writable = writable + self.allow_delete = allow_delete + #: "name" -> segments look like 'Images'; "labelled" -> 'Images [GF12]'; "id" -> 'GF12'. + #: See paths.py. + self.path_style = path_style + #: (parent path, segment) -> global ID, for segments written as a plain name. A name + #: can only be resolved against its parent's listing, so without this every operation + #: on a deep path would re-list every directory above it. + self._name_cache: Dict[Tuple[str, str], str] = {} + + # ------------------------------------------------------------ paths + + @classmethod + def _strip_protocol(cls, path): + if isinstance(path, list): + return [cls._strip_protocol(p) for p in path] + path = super()._strip_protocol(path) + return path.strip("/") + + @staticmethod + def _join(parent: str, segment: str) -> str: + return f"{parent}/{segment}" if parent else segment + + def _segment(self, name, gid: str) -> str: + """Path segment for a record, in this filesystem's path style.""" + return paths.segment_for(self.path_style, name, gid) + + def _upload_target(self, path: str) -> Tuple[str, str]: + """Split an upload path into the container to put the file in and the name to give + it: the last segment is the name of the file to create and its parent is the + container (a Gallery folder, an Inventory record, a document field).""" + name = paths.last_segment(path) + if not name or paths.is_root(path): + raise IsADirectoryError(errno.EISDIR, f"{path!r}: a file path is needed to upload to", path) + return self._parent(path), name + + @staticmethod + def _upload_kwargs(name: str) -> dict: + """Extra keywords for the client's upload call: the name to store the file under.""" + return {"filename": name} + + # ------------------------------------------------------------ write guard + + def _require_writable(self, path: str) -> None: + if not self.writable: + raise ReadOnlyError( + errno.EROFS, f"{path!r}: this filesystem was opened read-only; pass writable=True " + f"to allow writes", str(path)) + + def _require_delete(self, path: str) -> None: + if not self.allow_delete: + raise ReadOnlyError( + errno.EROFS, f"{path!r}: deletion is disabled; pass allow_delete=True to allow it", str(path)) + + @property + def read_only(self) -> bool: + """Read-only means nothing here can change, so a filesystem that may delete is not + read-only even when it refuses uploads. The two flags are deliberately separate.""" + return not (self.writable or self.allow_delete) + + # ------------------------------------------------------------ listing + + def scandir(self, path: str) -> Iterator[dict]: + """Yield an entry per child from a single RSpace listing call, lazily: a section or + folder with thousands of records is fetched a page at a time as the iterator is + consumed. ``ls`` is this, collected into a list.""" + path = self._strip_protocol(path) + return self._listing(path) + + def ls(self, path: str, detail: bool = True, refresh: bool = False, **kwargs) -> list: + path = self._strip_protocol(path) + out = None if refresh else self.dircache.get(path) + if out is None: + # the directory is resolved before the list is built, so a bad path fails here + out = list(self._listing(path)) + if self.dircache.use_listings_cache: + self.dircache[path] = out + return out if detail else [entry["name"] for entry in out] + + def _listing(self, path: str, names_only: bool = False) -> Iterator[dict]: + """``_children`` with full names and the guarantee that no two children share a + segment and that every segment addresses the record it shows. + + RSpace lets siblings share a name, so under the ``name`` path style a plain listing + could show the same segment twice, which would make one of the two unreachable. The + first record to use a name keeps it and every later one gains its global ID before + the extension (``data [GL112].csv``), which matches how ``_lookup`` resolves a bare + name. A name that itself parses as an address (``report [GL111].txt``) is given its + real ID the same way, in ``paths.segment_for``, so that a consumer which stores the + segment and resolves it later (Galaxy stores URIs) always gets the record it saw. + The rule needs no lookahead, so listings stay lazy. + + ``names_only`` says the listing is read only to resolve a name or to test for + emptiness, so branches may skip per-entry extras (the Gallery's size fetches). + """ + # _children resolves the directory before returning its generator, so that a bad + # path fails on the call rather than on first iteration; keep that by not making + # this a generator + target = self._resolve(path) + if target.kind in self.FILE_KINDS: + raise NotADirectoryError(errno.ENOTDIR, f"{path!r} is a file", path) + return self._named(path, self._children(target, names_only)) + + def _named(self, parent: str, children: Iterator[dict]) -> Iterator[dict]: + """Prefix each child with ``parent``, make segments unique under the ``name`` style + and remember every (parent, segment) -> global ID, so that one listing resolves all + of its children for later calls.""" + seen = set() + for entry in children: + gid = entry.get("globalId") + if self.path_style == "name" and entry["name"] in seen and gid: + entry["name"] = paths.disambiguated_segment(entry["name"], gid) + seen.add(entry["name"]) + if gid: + self._remember(parent, entry["name"], gid) + entry["name"] = self._join(parent, entry["name"]) + yield entry + + def _remember(self, parent: str, segment: str, gid: str) -> None: + if len(self._name_cache) >= self.NAME_CACHE_LIMIT: + self._name_cache.clear() + # both spellings, so a path stored before a document was signed still resolves + self._name_cache[(parent, segment)] = gid + self._name_cache[(parent, paths.unmarked(segment))] = gid + + def _resolve(self, path: str) -> Target: + """What ``path`` addresses. Branches implement it with their own vocabulary.""" + raise NotImplementedError(f"{type(self).__name__} does not implement _resolve") + + def _children(self, target: Target, names_only: bool = False) -> Iterator[dict]: + """The entries inside a directory target, with segment names, from one listing call + where possible.""" + raise NotImplementedError(f"{type(self).__name__} does not implement _children") + + def _record(self, gid: str) -> dict: + """The raw RSpace record that owns fields (an Inventory record, an ELN document).""" + raise NotImplementedError(f"{type(self).__name__} does not implement _record") + + def _file_fields(self, record: Mapping) -> List[dict]: + """The record's fields that can hold a file. Other field types are not browsable.""" + return [f for f in record.get("fields") or [] + if str(f.get("type", "")).lower() in self.FILE_BEARING_FIELD_TYPES and f.get("globalId")] + + def _field(self, owner_gid: str, field_gid: str, record: Optional[Mapping] = None) -> dict: + record = record if record is not None else self._record(owner_gid) + return self._find_field(self._file_fields(record), field_gid, owner_gid) + + def _gids(self, segments: List[str], start: int = 0) -> Iterator[Tuple[int, str]]: + """``(index, global id)`` for each segment from ``start`` on. A segment carrying a + global ID says what it addresses; a bare name is looked up in its parent's listing.""" + for index in range(start, len(segments)): + gid, _ = paths.parse_segment(segments[index]) + if gid is None: + gid = self._lookup("/".join(segments[:index]), segments[index]) + yield index, gid + + def _entry(self, record: Mapping, is_dir: bool, gid: Optional[str] = None, + name: Optional[str] = None, **extra: Any) -> dict: + """The entry for an RSpace record, named after it in this filesystem's path style.""" + gid = gid or record["globalId"] + return make_entry(self._segment(record.get("name") if name is None else name, gid), is_dir, + raw=record, size=record.get("size"), created=record.get("created"), + modified=record.get("lastModified"), **extra) + + @staticmethod + def _find_field(fields: Iterable[Mapping], field_gid: str, owner: str) -> dict: + for field in fields: + if field.get("globalId") == field_gid: + return field + raise FileNotFoundError(errno.ENOENT, f"no field {field_gid} on {owner}", f"{owner}/{field_gid}") + + def _is_empty(self, path: str) -> bool: + """True if the directory has no children, without listing all of them.""" + return next(self._listing(path, names_only=True), None) is None + + # ------------------------------------------------------------ resolving a segment + + def _lookup(self, parent: str, segment: str) -> str: + """Global ID of the child of ``parent`` whose path segment is ``segment``. + + Only needed for a segment written as a plain name: one carrying a global ID says + what it addresses. Costs one listing of the parent, remembered afterwards; the + listing records every child it passes, so siblings resolve for free. + """ + key = (parent, segment) + if key not in self._name_cache: + for _ in self._listing(parent, names_only=True): + if key in self._name_cache: + break + else: + raise FileNotFoundError(errno.ENOENT, f"{self._join(parent, segment)!r} not found", + self._join(parent, segment)) + return self._name_cache[key] + + def _gid_at(self, path: str) -> str: + """Global ID addressed by the last segment of ``path``, whichever style wrote it.""" + segment = paths.last_segment(path) + gid, _ = paths.parse_segment(segment) + return gid if gid is not None else self._lookup(self._parent(path), segment) + + def clear_name_cache(self) -> None: + """Forget resolved name lookups. Called after every change made through this + filesystem; call it yourself if records are renamed elsewhere while it is open.""" + self._name_cache.clear() + + def invalidate_cache(self, path: Optional[str] = None) -> None: + """Drop cached listings (fsspec's) and resolved names (ours). + + fsspec's default does nothing to the listings cache, so a write through this + filesystem would otherwise leave a cached listing showing the old state. Writes are + rare next to reads, so everything is dropped rather than just the paths involved. + """ + super().invalidate_cache(path) + self.dircache.clear() + self.clear_name_cache() + + # ------------------------------------------------------------ info + + def info(self, path: str, **kwargs) -> dict: + path = self._strip_protocol(path) + entry = self._info_of(path, self._resolve(path)) + entry["name"] = path + return entry + + def _info_of(self, path: str, target: Target) -> dict: + """The entry for a resolved path. Branches implement it.""" + raise NotImplementedError(f"{type(self).__name__} does not implement _info_of") + + def _stamp(self, path: str, key: str) -> Optional[datetime]: + stamp = self.info(path).get(key) + return datetime.fromtimestamp(stamp, tz=timezone.utc) if stamp is not None else None + + def created(self, path: str) -> Optional[datetime]: + return self._stamp(path, "created") + + def modified(self, path: str) -> Optional[datetime]: + return self._stamp(path, "mtime") + + # ------------------------------------------------------------ search + + #: ``/**`` with no other wildcard anywhere (a backslash escape inside the text + #: is allowed, which is how Galaxy sends a ``[``). + _GALAXY_SEARCH = re.compile(r"^(?P[^*?]*?)/?\*(?P[^*?/]+)\*$", re.S) + + def glob(self, path: str, maxdepth: Optional[int] = None, **kwargs): + """fsspec glob, with one special case: the ``/**`` pattern a search box + sends (Galaxy builds exactly this) is answered by one listing of ```` and a + case-insensitive substring match on the last segment, in listing order. fsspec's + own glob treats ``[`` as a character class and ignores the backslash Galaxy escapes + it with, so a segment carrying ``[GL83]`` could never be found that way.""" + path = self._strip_protocol(path) + match = self._GALAXY_SEARCH.match(path) + if match and maxdepth in (None, 1): + needle = re.sub(r"\\(.)", r"\1", match.group("text")).lower() + found = {entry["name"]: entry for entry in self._listing(match.group("dir")) + if needle in paths.last_segment(entry["name"]).lower()} + return found if kwargs.get("detail") else list(found) + return super().glob(path, maxdepth=maxdepth, **kwargs) + + # ------------------------------------------------------------ files + + def _file_id(self, path: str) -> str: + """Numeric id of the file at ``path``; ``IsADirectoryError`` for anything else.""" + target = self._resolve(path) + if target.kind not in self.FILE_KINDS: + raise IsADirectoryError(errno.EISDIR, f"{path!r} is a directory", path) + return target.gid[2:] + + def _download(self, file_id: str, file: BinaryIO, chunk_size: int) -> None: + """The client call that streams file ``file_id`` into ``file``. Branches implement it.""" + raise NotImplementedError(f"{type(self).__name__} does not implement _download") + + def download_fileobj(self, path: str, file: BinaryIO, chunk_size: int = 128) -> None: + """Stream the file at ``path`` into the open binary ``file``.""" + self._download(self._file_id(self._strip_protocol(path)), file, chunk_size) + + def upload_fileobj(self, path: str, file: BinaryIO, **options: Any): + """Write the open binary ``file`` to the file at ``path``: the last segment is the name + to give it and its parent is the container to put it in. Branches implement it.""" + raise NotImplementedError(f"{type(self).__name__} cannot upload") + + def cat_file(self, path: str, start: Optional[int] = None, end: Optional[int] = None, **kwargs) -> bytes: + buffer = io.BytesIO() + self.download_fileobj(path, buffer) + data = buffer.getvalue() + return data[start:end] if (start is not None or end is not None) else data + + def get_file(self, rpath: str, lpath, callback=None, outfile=None, **kwargs) -> None: + """Download straight into the local file, without the whole-file buffer ``open`` needs.""" + if isinstance(lpath, (str, bytes)) or hasattr(lpath, "__fspath__"): + rpath = self._strip_protocol(rpath) + if self._resolve(rpath).kind not in self.FILE_KINDS: + os.makedirs(lpath, exist_ok=True) + return + with open(lpath, "wb") as handle: + self.download_fileobj(rpath, handle) + else: + self.download_fileobj(rpath, lpath) + + def put_file(self, lpath, rpath: str, callback=None, mode: str = "overwrite", **kwargs) -> None: + """Upload the local file as the RSpace file at ``rpath``.""" + if os.path.isdir(lpath): + # fsspec's put() hands directories through put_file as well, and relies on the + # backend to create them before their files arrive + self.makedirs(rpath, exist_ok=True) + return + with open(lpath, "rb") as handle: + self.upload_fileobj(rpath, handle, **kwargs) + + def _open(self, path: str, mode: str = "rb", block_size=None, autocommit: bool = True, + cache_options=None, **kwargs): + """Reads buffer the whole file (RSpace serves no byte ranges). Writes buffer until the + file is closed, or until the transaction commits, and upload in one request.""" + path = self._strip_protocol(path) + if mode in ("rb", "r"): + return io.BytesIO(self.cat_file(path)) + if mode in ("wb", "xb", "w", "x"): + self._require_writable(path) + if not paths.last_segment(path): + raise IsADirectoryError(errno.EISDIR, f"{path!r} is a directory", path) + if mode.startswith("x") and self.exists(path): + raise FileExistsError(errno.EEXIST, f"{path!r} exists", path) + return RSpaceWriteFile(self, path, mode="wb", block_size=2 ** 62, autocommit=autocommit) + raise ValueError(f"mode {mode!r} is not supported; use 'rb' or 'wb'") + + # ------------------------------------------------------------ folders (templates) + + @staticmethod + def _new_folder(path: str) -> Tuple[str, str]: + """``(parent path, folder name)`` for mkdir. 'New [GF1]' creates 'New', as does + 'New'; a bare global ID is refused because it names nothing to create.""" + segment = paths.last_segment(path) + gid, label = paths.parse_segment(segment) + if gid is not None and label is None: + raise ValueError(f"{path!r}: mkdir takes a folder name, not a global ID") + return RSpaceFSBase._parent(path), label if gid is not None else segment + + def _create_folder(self, path: str, name: str, siblings: Iterable[Mapping], + create: Callable[[], Mapping], folder_prefixes: Tuple[str, ...], + exist_ok: bool) -> str: + """Create ``name`` unless a folder of that name is already among ``siblings``. RSpace + would happily make a second folder with the same name; the filesystem contract says + to refuse instead (or, with ``exist_ok``, to return the existing one). Returns the + path of the folder, in this filesystem's path style.""" + parent = self._parent(path) + for record in siblings: + if record.get("name") == name and record["globalId"][:2] in folder_prefixes: + if exist_ok: + return self._join(parent, self._segment(name, record["globalId"])) + raise FileExistsError(errno.EEXIST, f"{path!r} exists", path) + created = create() + gid = created.get("globalId") or f"{folder_prefixes[0]}{created['id']}" + self.invalidate_cache() + return self._join(parent, self._segment(name, gid)) + + def _remove_folder(self, path: str, folder_gid: Optional[str], delete: Callable[[str], Any], + not_a_folder: Exception) -> None: + """Delete the empty folder at ``path``. ``folder_gid`` is its global ID, or None when + the path is not a removable folder, in which case ``not_a_folder`` is raised.""" + if paths.is_root(path): + raise PermissionError(errno.EPERM, "the root cannot be removed", path) + if folder_gid is None: + raise not_a_folder + if not self._is_empty(path): + raise OSError(errno.ENOTEMPTY, f"{path!r} is not empty", path) + delete(folder_gid[2:]) + self.invalidate_cache() + + # ------------------------------------------------------------ mutations (defaults) + + def mkdir(self, path: str, create_parents: bool = True, **kwargs) -> None: + raise NotImplementedError(f"{type(self).__name__} cannot create directories") + + def makedirs(self, path: str, exist_ok: bool = False) -> None: + """Only the last segment is created; RSpace folders are not made in bulk here.""" + path = self._strip_protocol(path) + if exist_ok and self.exists(path): + if not self.isdir(path): + raise FileExistsError(errno.EEXIST, f"{path!r} exists and is not a directory", path) + return + self.mkdir(path, create_parents=False, exist_ok=exist_ok) + + def rmdir(self, path: str) -> None: + raise NotImplementedError(f"{type(self).__name__} cannot delete directories") + + def rm_file(self, path: str) -> None: + raise NotImplementedError(f"{type(self).__name__} cannot delete files") + + def rm(self, path, recursive: bool = False, maxdepth: Optional[int] = None) -> None: + """Remove files and empty directories. Recursive removal is refused: RSpace records + are not something a filesystem should delete wholesale.""" + if recursive: + raise NotImplementedError("recursive removal is not supported on RSpace") + for one in ([path] if isinstance(path, str) else path): + one = self._strip_protocol(one) + if self._resolve(one).kind in self.FILE_KINDS: + self.rm_file(one) + else: + self.rmdir(one) + + def link(self, path: str, media_file_gid: str) -> dict: + """Attach a Gallery file by reference. Only the branches with ``CAN_LINK`` can.""" + raise NotImplementedError(f"{path!r}: the Gallery holds the files, it cannot link to them") + + def cp_file(self, path1: str, path2: str, **kwargs) -> None: + """Copy a file within this filesystem by downloading and uploading it.""" + path1, path2 = self._strip_protocol(path1), self._strip_protocol(path2) + self._require_writable(path2) + self.upload_fileobj(path2, io.BytesIO(self.cat_file(path1))) + + def _require_movable(self, path1: str, recursive: bool) -> None: + """A move is a copy followed by a remove, so check that the remove is allowed + *before* uploading: otherwise the copy succeeds and the caller is left with an + error and a duplicate the API may not be able to delete.""" + self._require_delete(path1) + if not self.CAN_DELETE_FILES: + raise NotImplementedError( + f"{path1!r} cannot be moved: the RSpace API cannot delete a Gallery file, so only " + f"a copy is possible") + if recursive: + raise NotImplementedError("recursive move is not supported on RSpace") + + def mv(self, path1: str, path2: str, recursive: bool = False, maxdepth: Optional[int] = None, + **kwargs) -> None: + self._require_writable(path2) + self._require_movable(path1, recursive) + self.cp_file(path1, path2) + self.rm_file(path1) + + +# ``__init_subclass__`` only sees subclasses; the methods the base defines itself (``ls``, +# ``info``, ``cat_file``...) are translated here, once, so a branch that inherits them +# unchanged still raises ``OSError`` rather than a client exception. +translate_class(RSpaceFSBase) diff --git a/rspace_client/fs/gallery.py b/rspace_client/fs/gallery.py new file mode 100644 index 0000000..ec5dc80 --- /dev/null +++ b/rspace_client/fs/gallery.py @@ -0,0 +1,395 @@ +""" +fsspec view of the RSpace Gallery. + +Gallery folders (``GF``) are directories and Gallery files (``GL``) are files. The +filesystem root is the Gallery's top folder, resolved lazily on first use, by +name, from the user's Home folder listing. + + fs = GalleryFilesystem(server, api_key) # read-only + fs = GalleryFilesystem(server, api_key, writable=True) # upload / mkdir allowed + fs = GalleryFilesystem(eln_client=my_client) # share a client (unified mount) + fs = GalleryFilesystem(server, api_key, path_style="id") # bare 'GF12' segments instead of 'Images' + +Listings are one request per directory. The folder-tree endpoint omits file sizes, and +consumers such as Galaxy need an integer size for every file, so by default each file in a +listing is topped up with its own record (``fetch_sizes=True``); pass ``fetch_sizes=False`` +for a listing that stays at one request and reports ``size`` as None. + +Gallery upload routing: the Gallery is split into media-type sections +(Images, Documents, Chemistry, ...) and a file may only be placed in a folder of the +matching section. ``upload_fileobj()`` returns a :class:`Placement`; when the target +folder's section rejects the file it either raises :class:`GallerySectionMismatch` +(``on_mismatch="raise"``, the default) or lets RSpace place the file in the correct +section and reports where it landed (``on_mismatch="reroute"``). +""" +from __future__ import annotations + +import errno +import logging +from dataclasses import dataclass +from typing import Any, BinaryIO, Iterator, Mapping, Optional + +from rspace_client.client_base import ClientBase +from rspace_client.eln import eln +from rspace_client.exceptions import RSpaceError + +from . import paths +from .base import RSpaceFSBase, Target, client_or_new, deletes, make_entry, stream_pages, writes + +logger = logging.getLogger(__name__) + +FOLDER_PREFIXES = ("GF", "FL", "NB") +FILE_PREFIX = "GL" +#: The Gallery root is the folder of this name in the user's Home folder listing. +GALLERY_FOLDER_NAME = "Gallery" + +# Accepted values for the ``on_mismatch`` policy. +ON_MISMATCH_RAISE = "raise" +ON_MISMATCH_REROUTE = "reroute" +_ON_MISMATCH_VALUES = (ON_MISMATCH_RAISE, ON_MISMATCH_REROUTE) + +# The RSpace Gallery's media-type sections. "Miscellaneous" is the catch-all +# that accepts any file not matching a more specific section; the others accept +# only their listed extensions (including "Documents", which is a fixed set, not +# a catch-all). +MISCELLANEOUS_SECTION = "Miscellaneous" + +# Best-effort mapping of file extension to the Gallery section RSpace classifies +# it into, taken from the Gallery documentation. This mirrors the server's own +# classification but is NOT authoritative: it is used only to phrase error +# messages, never to decide whether an upload is allowed. Any extension not +# listed falls through to "Miscellaneous", matching the server default. +_SECTION_BY_EXTENSION = { + # Images + "png": "Images", "jpg": "Images", "jpeg": "Images", "gif": "Images", + "bmp": "Images", "tif": "Images", "tiff": "Images", + # Audios + "mp3": "Audios", "wav": "Audios", "wma": "Audios", "aac": "Audios", + "ogg": "Audios", + # Videos + "mp4": "Videos", "mov": "Videos", "hdmov": "Videos", "m4v": "Videos", + "wmv": "Videos", "avi": "Videos", "mpg": "Videos", "mpeg": "Videos", + "flv": "Videos", "3gp": "Videos", + # Documents (a fixed set, NOT a catch-all) + "doc": "Documents", "docx": "Documents", "rtf": "Documents", + "pdf": "Documents", "odt": "Documents", "ods": "Documents", + "odp": "Documents", "txt": "Documents", "ppt": "Documents", + "pptx": "Documents", "xls": "Documents", "xlsx": "Documents", + "csv": "Documents", "pps": "Documents", "md": "Documents", + # Chemistry (documented subset; the server accepts more) + "skc": "Chemistry", "mrv": "Chemistry", "cxsmiles": "Chemistry", + "cxsmarts": "Chemistry", "cdx": "Chemistry", "cdxml": "Chemistry", + "csrdf": "Chemistry", "cml": "Chemistry", +} + + +def classify_media_section(filename: Optional[str]) -> Optional[str]: + """ + Guess which Gallery section RSpace will file ``filename`` under, mirroring the + server-side classification. Returns a section name (e.g. "Images"), + "Miscellaneous" as the catch-all when the extension is not one of a specific + section's types, or None when no filename/extension is available to guess + from. + + The server remains the authority on placement; this is only used to make + error messages and log lines more helpful. + """ + if not filename or "." not in filename: + return None + ext = filename.rsplit(".", 1)[-1].lower() + return _SECTION_BY_EXTENSION.get(ext, MISCELLANEOUS_SECTION) + + +def _a_or_an(word: str) -> str: + return "an" if word[:1].upper() in "AEIOU" else "a" + + +def check_policy(on_mismatch: str) -> str: + if on_mismatch not in _ON_MISMATCH_VALUES: + raise ValueError("on_mismatch must be one of {}".format(_ON_MISMATCH_VALUES)) + return on_mismatch + + +class GallerySectionMismatch(ClientBase.ApiError): + """ + Raised when a file could not be uploaded into the requested Gallery folder + because that folder belongs to a media-type section that does not accept the + file. Carries the folder's section and, when it could be guessed, the file's + media type, so callers can react programmatically as well as read the message. + + It is deliberately *not* translated into an ``OSError``: the message is written for the + person in front of the screen and the attributes are what a library caller needs. + """ + + fs_passthrough = True + + def __init__(self, message, *, folder_section=None, folder_global_id=None, + file_media_type=None, response_status_code=None): + super().__init__(message, response_status_code=response_status_code) + self.folder_section = folder_section + self.folder_global_id = folder_global_id + self.file_media_type = file_media_type + + +@dataclass +class Placement: + """Where an uploaded file actually ended up in the Gallery. + + Returned by :meth:`GalleryFilesystem.upload_fileobj`. When ``rerouted`` is True the + file could not go into the requested folder and RSpace filed it in the section its + media type belongs to; ``path`` is a readable description of where that is. + """ + file_global_id: Optional[str] + folder_global_id: Optional[str] + section: Optional[str] + path: str + rerouted: bool + requested_path: Optional[str] = None + + +def folder_section(eln_client: eln.ELNClient, folder_id) -> Optional[str]: + """The Gallery section (mediaType) a folder belongs to, or None if it cannot be + determined. Best-effort: never raises, so it cannot mask the real outcome of an upload.""" + try: + return eln_client.get_folder(folder_id).get("mediaType") + except RSpaceError: + return None + + +def human_path(folder: Mapping[str, Any]) -> str: + """Best-effort readable path for a folder, e.g. 'Gallery/Documents/Api Inbox'. Uses the + API's pathToRootFolder when present, otherwise the section and folder name.""" + trail = folder.get("pathToRootFolder") + names = [f.get("name") for f in trail if f.get("name")] if isinstance(trail, list) else [] + if names: + return "/".join(names) + return "/".join(["Gallery"] + [folder[k] for k in ("mediaType", "name") if folder.get(k)]) + + +def placement(eln_client: eln.ELNClient, response: Any, requested_path: Optional[str], + rerouted: bool) -> Placement: + """Build a Placement from an upload response, resolving the parent folder for + section/path feedback where possible. Tolerates a response that is not a dict (e.g. + None): a caller may wrap ``upload_file`` and discard its return value, and Placement + construction must never crash a successful upload.""" + if not isinstance(response, dict): + response = {} + folder: Mapping[str, Any] = {} + parent_id = response.get("parentFolderId") + if parent_id is not None: + try: + folder = eln_client.get_folder(parent_id) + except RSpaceError: + pass + return Placement( + file_global_id=response.get("globalId"), + folder_global_id=folder.get("globalId"), + section=folder.get("mediaType"), + path=human_path(folder) if folder else "Gallery", + rerouted=rerouted, + requested_path=requested_path, + ) + + +def mismatch_message(named: str, container: str, section: str, guessed: Optional[str], err: Exception) -> str: + """The explanation a person reads when a folder's section refuses their file.""" + message = (f"Could not upload {named} to Gallery folder {container or '/'}. That folder " + f"is in the '{section}' section, which only accepts {section} files") + if guessed and guessed != section: + looks = ("does not match a specialised section and belongs in 'Miscellaneous'" + if guessed == MISCELLANEOUS_SECTION else f"looks like {_a_or_an(guessed)} '{guessed}' file") + message += f", but {named} {looks}" + remedy = f"Choose a folder in the '{guessed}' section instead" + else: + remedy = "Choose a folder in the section that matches this file type" + # Lead with what the person in front of the screen can act on; the API-level option is + # a parenthetical for library callers, who are not always the audience (this message + # reaches Galaxy users verbatim). + return message + (f". {remedy}, or upload without choosing a folder and RSpace will file it in the " + f"right section itself (library callers can pass on_mismatch='reroute' to do that " + f"automatically). Original API error: {err}") + + +class GalleryFilesystem(RSpaceFSBase): + """Target kinds: root | folder | file.""" + + protocol = ("rspace-gallery",) + FILE_KINDS = ("file",) + CAN_DELETE_FILES = False # the RSpace API has no call to delete a Gallery file + + def __init__(self, server: Optional[str] = None, api_key: Optional[str] = None, *, + eln_client: Optional[eln.ELNClient] = None, writable: bool = False, + allow_delete: bool = False, path_style: str = "name", + on_mismatch: str = ON_MISMATCH_RAISE, page_size: int = 100, + fetch_sizes: bool = True, **storage_options: Any) -> None: + super().__init__(writable=writable, allow_delete=allow_delete, path_style=path_style, + **storage_options) + #: records fetched per API request while a listing is consumed, as in the other branches + self.page_size = max(1, int(page_size)) + self.eln_client = client_or_new(eln_client, eln.ELNClient, server, api_key, "GalleryFilesystem") + self.on_mismatch = check_policy(on_mismatch) + #: fetch each file's own record while listing, because the folder-tree endpoint + #: carries no size (one extra request per file) + self.fetch_sizes = fetch_sizes + self._gallery_id: Optional[int] = None + + # ------------------------------------------------------------ RSpace plumbing + + @property + def gallery_id(self) -> int: + """Numeric id of the Gallery root folder, looked up once on first use.""" + if self._gallery_id is None: + for record in self._all_records(None): + if record.get("name") == GALLERY_FOLDER_NAME: + self._gallery_id = int(record["id"]) + break + else: + raise FileNotFoundError( + errno.ENOENT, f"no folder named {GALLERY_FOLDER_NAME!r} in the Home folder listing", "") + return self._gallery_id + + def _all_records(self, folder_id) -> Iterator[dict]: + """Stream a folder's contents, following the tree listing's 'next' links only as + far as the consumer reads.""" + first_page = self.eln_client.list_folder_tree(folder_id, page_size=self.page_size) + return stream_pages(self.eln_client, first_page, "records") + + def _resolve(self, path: str) -> Target: + if paths.is_root(path): + return Target("root") + gid = self._gid_at(path) + if gid[:2] in FOLDER_PREFIXES: + return Target("folder", gid=gid) + if gid[:2] == FILE_PREFIX: + return Target("file", gid=gid) + raise FileNotFoundError(errno.ENOENT, f"{path!r} not found", path) + + def _folder_id(self, path: str): + """Numeric folder id for a directory path (root -> Gallery root).""" + target = self._resolve(path) + if target.kind == "root": + return self.gallery_id + if target.kind != "folder": + raise NotADirectoryError(errno.ENOTDIR, f"{path!r} is a file", path) + return target.gid[2:] + + def _record_entry(self, record: dict) -> dict: + return self._entry(record, record["globalId"][:2] in FOLDER_PREFIXES) + + # ------------------------------------------------------------ listing and info + + def _children(self, target: Target, names_only: bool = False) -> Iterator[dict]: + folder_id = self.gallery_id if target.kind == "root" else target.gid[2:] + return self._scan_records(folder_id, self.fetch_sizes and not names_only) + + def _scan_records(self, folder_id, want_sizes: bool) -> Iterator[dict]: + # One listing call per directory, plus one record fetch per file when sizes are + # wanted: folder-tree items carry no size and consumers such as Galaxy require an + # integer size for every file. + for record in self._all_records(folder_id): + if want_sizes and record.get("globalId", "")[:2] == FILE_PREFIX and record.get("size") is None: + try: + record = self.eln_client.get_file_info(record["id"]) + except ClientBase.ApiError: # keep listing even if one file cannot be read + logger.debug("could not read file details for %s", record.get("globalId")) + yield self._record_entry(record) + + def _info_of(self, path: str, target: Target) -> dict: + if target.kind == "root": + return make_entry("", True, raw={"name": GALLERY_FOLDER_NAME}) + if target.kind == "folder": + return self._entry(self.eln_client.get_folder(target.gid[2:]), True, target.gid) + return self._entry(self.eln_client.get_file_info(target.gid[2:]), False, target.gid) + + # ------------------------------------------------------------ folders + + @writes + def mkdir(self, path: str, create_parents: bool = True, exist_ok: bool = False, **kwargs) -> None: + path = self._strip_protocol(path) + if paths.is_root(path): + raise FileExistsError(errno.EEXIST, "the root exists", path) + parent, name = self._new_folder(path) + folder_id = self._folder_id(parent) + self._create_folder(path, name, self._all_records(folder_id), + lambda: self.eln_client.create_folder(name, folder_id), + FOLDER_PREFIXES, exist_ok) + + @deletes + def rmdir(self, path: str) -> None: + path = self._strip_protocol(path) + target = self._resolve(path) + self._remove_folder(path, target.gid if target.kind == "folder" else None, + self.eln_client.delete_folder, + NotADirectoryError(errno.ENOTDIR, f"{path!r} is not a folder", path)) + + def rm_file(self, path: str) -> None: + raise NotImplementedError("deleting Gallery files is not supported by the RSpace API") + + # ------------------------------------------------------------ files + + def _download(self, file_id: str, file: BinaryIO, chunk_size: int) -> None: + self.eln_client.download_file(file_id, file, chunk_size) + + # ------------------------------------------------------------ upload with section routing + + @writes + def upload_fileobj(self, path: str, file: BinaryIO, on_mismatch: Optional[str] = None, + **options: Any) -> Placement: + """ + Write ``file`` to the Gallery file at ``path``: the last segment is the name to give + the file and its parent is the folder to put it in. A file directly under the root + is left to the API's default target, the relevant section's "Api Inbox". + + :param on_mismatch: optional override of the filesystem-wide policy for + this call ("raise" or "reroute"); defaults to the value + passed to the constructor. + :return: a :class:`Placement` describing where the file ended up. + + The RSpace Gallery is split into media-type sections (Images, Documents, + Chemistry, ...) and a file may only be placed in a folder whose section + matches the file's media type. If ``path`` names a folder in the wrong + section the upload is rejected. Depending on the effective policy this + either raises a :class:`GallerySectionMismatch` naming the folder's + section (``"raise"``) or places the file in the correct section's inbox + and returns a Placement with ``rerouted=True`` (``"reroute"``). + """ + policy = check_policy(self.on_mismatch if on_mismatch is None else on_mismatch) + path = self._strip_protocol(path) + container, name = self._upload_target(path) + folder_id = None if paths.is_root(container) else self._folder_id(container) + try: + response = self.eln_client.upload_file(file, folder_id, **self._upload_kwargs(name)) + self.invalidate_cache() + return placement(self.eln_client, response, requested_path=container, rerouted=False) + except ClientBase.ApiError as err: + section = None if folder_id is None else folder_section(self.eln_client, folder_id) + if section is None: + raise + filename = options.get("filename") or name + guessed = classify_media_section(filename) + + if policy == ON_MISMATCH_REROUTE: + try: + file.seek(0) + except (AttributeError, OSError, ValueError): + pass + response = self.eln_client.upload_file(file, None, **self._upload_kwargs(name)) + self.invalidate_cache() + placed = placement(self.eln_client, response, requested_path=container, rerouted=True) + logger.info( + "RSpace Gallery: %s could not go in %s (section '%s'); placed in %s instead", + f"'{filename}'" if filename else "file", container, section, placed.path) + return placed + + # `from None` rather than `from err`: this exception reaches people through + # Galaxy, which prints the whole chain and clips it from the top, hiding the + # explanation at the bottom. The server's own reason is already quoted in the + # message, and the original error stays available on the exception's + # attributes, so nothing is lost by not stacking three tracebacks. + raise GallerySectionMismatch( + mismatch_message(f"'{filename}'" if filename else "the file", container, section, guessed, err), + folder_section=section, + folder_global_id="GF" + str(folder_id), + file_media_type=guessed, + response_status_code=getattr(err, "response_status_code", None), + ) from None diff --git a/rspace_client/fs/inventory.py b/rspace_client/fs/inventory.py new file mode 100644 index 0000000..1ea4b4f --- /dev/null +++ b/rspace_client/fs/inventory.py @@ -0,0 +1,321 @@ +""" +fsspec view of RSpace Inventory: every record is a folder that may hold files. + + (root) benches, plus the sections Containers, Samples, Templates + Containers top-level containers, streamed however many there are + Samples, Templates likewise + WB user1a a bench: its containers and subsamples, plus attachments + .../Freezer -80 a container: child containers and subsamples as folders, + attachments as files + .../Plasmid pUC19 a sample: its subsamples as folders, attachments as files + .../Plasmid pUC19 #1 a subsample: attachments, plus a shortcut folder + .../sample: Plasmid pUC19 the shortcut: that sample's attachments only (no subsamples, + so the tree has no cycles) + .../Plasmid a sample template: attachments + .../microscope an instrument: attachments + .../Safety Data an attachment-type template field: the one file it holds + +Segments are the records' own names under the default path style; a segment written as a +global ID (``IC200``, ``Freezer -80 [IC200]``) resolves whatever style is in force. + +Records carry files in two places: their own ``attachments``, listed as files in the record +folder, and template-defined **attachment fields**, listed as folders holding at most one +file each (the same shape as an ELN document field). Only ``attachment`` fields are shown; +the other field types cannot hold a file. An attachment field accepts a file only while it +is empty: RSpace would replace the current file rather than add to it, which a folder does +not suggest, so uploading into an occupied field raises ``FileExistsError``. Remove the +file first (with ``allow_delete=True``) and then upload. + +Samples are not stored *in* containers in RSpace, their subsamples are; the shortcut +folder is how you get from a subsample you found in a container to its sample's files. + +A record listing is one API call. A section streams: its records are fetched +``page_size`` at a time as the listing is consumed (``scandir`` keeps that laziness, ``ls`` +collects the list). Read-only by default: ``writable=True`` allows attaching files to any +record, ``allow_delete=True`` allows removing attachments. +""" +from __future__ import annotations + +import errno +from typing import Any, BinaryIO, Iterator, Optional + +from rspace_client.client_base import Pagination +from rspace_client.eln import eln +from rspace_client.inv import inv + +from . import paths +from .base import RSpaceFSBase, Target, client_or_new, deletes, make_entry, stream_pages, writes + +ATTACHMENT_PREFIX = "IF" +FIELD_PREFIX = "SF" # a template-defined field on a sample, template or instrument +CONTAINER_PREFIXES = ("IC", "BE") +RECORD_GETTERS = { + "IC": "get_container_by_id", + "BE": "get_workbench_by_id", # /containers/{id} answers 422 for a bench + "SA": "get_sample_by_id", + "SS": "get_subsample_by_id", + "IT": "get_sample_template_by_id", + "IN": "get_instrument_by_id", # instruments can sit on a bench and carry attachments +} +#: Record types the server will actually attach a file to, confirmed live against RSpace +#: 2.27. A bench answers "Unsupported global id type" and a sample template answers "Sample +#: Templates don't support file attachments yet". Both present as ordinary record folders, so +#: without this the Gallery upload happens first and its file is stranded when the link is +#: refused, and the RSpace API cannot delete a Gallery file. +ATTACHABLE_PREFIXES = ("IC", "SA", "SS", "IN", "SF") +SECTION_LISTERS = { # section -> (client method, collection key in the response) + "Containers": ("list_top_level_containers", "containers"), + "Samples": ("list_samples", "samples"), + "Templates": ("list_sample_templates", "templates"), +} +#: The named sections at this filesystem's root, in the order they are listed. +SECTIONS = tuple(SECTION_LISTERS) + + +class InventoryFilesystem(RSpaceFSBase): + """Target kinds: root | section | record | shortcut | field | attachment.""" + + protocol = ("rspace-inventory",) + FILE_KINDS = ("attachment",) + #: Inventory field types are number/date/string/text/uri/reference/attachment/time/radio/ + #: choice; only "attachment" can hold a file, and it holds at most one. + FILE_BEARING_FIELD_TYPES = ("attachment",) + CAN_LINK = True + + def __init__(self, server: Optional[str] = None, api_key: Optional[str] = None, *, + inv_client: Optional[inv.InventoryClient] = None, + eln_client: Optional[eln.ELNClient] = None, writable: bool = False, + allow_delete: bool = False, path_style: str = "name", + page_size: int = 100, via_gallery: Optional[bool] = None, + **storage_options: Any) -> None: + super().__init__(writable=writable, allow_delete=allow_delete, path_style=path_style, + **storage_options) + self.inv_client = client_or_new(inv_client, inv.InventoryClient, server, api_key, "InventoryFilesystem") + #: Put uploaded bytes in the Gallery and link them, rather than creating a file that + #: exists only inside Inventory. See ``upload_fileobj``. Needs an ELN client as well + #: as an Inventory one; left as None it turns itself on whenever one can be had, and + #: asking for it explicitly when one cannot is an error rather than a silent downgrade. + if eln_client is None and via_gallery is not False and server and api_key: + eln_client = eln.ELNClient(server, api_key) + self.eln_client = eln_client + if via_gallery and eln_client is None: + raise ValueError( + "via_gallery=True needs an ELN client as well as an Inventory one: pass " + "eln_client=, or server and api_key, or via_gallery=False to create files " + "that exist only inside Inventory") + self.via_gallery = bool(eln_client) if via_gallery is None else via_gallery + #: how many records to fetch per API request while streaming a listing; a + #: performance knob, not something a user of the filesystem sees. + self.page_size = max(1, int(page_size)) + + # ------------------------------------------------------------ path grammar + + def _resolve(self, path: str) -> Target: + if paths.is_root(path): + return Target("root") + segments = paths.segments(path) + section: Optional[str] = None + previous_gid: Optional[str] = None + target: Optional[Target] = None + first = 0 + if segments[0] in SECTIONS: + section = segments[0] + first = 1 + if first == len(segments): + return Target("section", section=section) + for index, gid in self._gids(segments, first): + prefix = gid[:2] + if prefix == ATTACHMENT_PREFIX: + if index != len(segments) - 1: # nothing lives below a file + raise FileNotFoundError(errno.ENOENT, f"{path!r} not found", path) + return Target("attachment", gid=gid, parent_gid=previous_gid) + if prefix == FIELD_PREFIX: + if previous_gid is None: + raise FileNotFoundError(errno.ENOENT, f"{path!r} not found", path) + target = Target("field", gid=gid, parent_gid=previous_gid) + previous_gid = gid + continue + if prefix not in RECORD_GETTERS: + raise FileNotFoundError(errno.ENOENT, f"{path!r} not found", path) + shortcut = prefix == "SA" and previous_gid is not None and previous_gid.startswith("SS") + previous_gid = gid + target = Target("shortcut" if shortcut else "record", gid=gid, section=section) + if target is None: + raise FileNotFoundError(errno.ENOENT, f"{path!r} not found", path) + return target + + # ------------------------------------------------------------ RSpace plumbing + + def _fetch(self, gid: str, include_content: bool = True) -> dict: + """Fetch a record. A container's contents cost extra to assemble server-side, so + ``include_content=False`` when only the record itself is wanted (``info``).""" + getter = getattr(self.inv_client, RECORD_GETTERS[gid[:2]]) + if gid[:2] in CONTAINER_PREFIXES and include_content: + return getter(gid[2:], include_content=True) + return getter(gid[2:]) + + def _record(self, gid: str) -> dict: + return self._fetch(gid, include_content=False) + + def _section_items(self, section: str) -> Iterator[dict]: + """Stream a section's records, fetching the next API page only when it is reached.""" + method, key = SECTION_LISTERS[section] + first_page = getattr(self.inv_client, method)(Pagination(page_number=0, page_size=self.page_size)) + return stream_pages(self.inv_client, first_page, key) + + # ------------------------------------------------------------ entry builders + + def _file_entry(self, attachment: dict, gid: Optional[str] = None) -> dict: + return self._entry(attachment, False, gid) + + @staticmethod + def _virtual_dir(name: str, extra: Optional[dict] = None) -> dict: + raw = {"name": name, "virtual": True} + raw.update(extra or {}) + return make_entry(name, True, raw=raw) + + def _field_entry(self, field: dict) -> dict: + raw = dict(field, fieldType=field.get("type")) + raw.pop("attachment", None) + return self._entry(raw, True) + + def _shortcut_entry(self, sample: dict) -> dict: + return self._entry(dict(sample, shortcut="sample"), True, name=f"sample: {sample.get('name', '')}") + + # ------------------------------------------------------------ children of a directory + + def _children(self, target: Target, names_only: bool = False) -> Iterator[dict]: + if target.kind == "root": + return self._root_children() + if target.kind == "section": + return (self._entry(item, True) for item in self._section_items(target.section)) + if target.kind == "field": + attachment = self._field(target.parent_gid, target.gid).get("attachment") + return iter([self._file_entry(attachment)] if attachment else []) + return self._record_children(target, self._fetch(target.gid)) + + def _root_children(self) -> Iterator[dict]: + for bench in self.inv_client.get_workbenches(): + yield self._entry(bench, True) + for section in SECTIONS: + yield self._virtual_dir(section, {"section": section}) + + def _record_children(self, target: Target, record: dict) -> Iterator[dict]: + prefix = target.gid[:2] + if target.kind == "record": + if prefix in CONTAINER_PREFIXES: + for location in record.get("locations", []): + content = location.get("content") + if content and content.get("globalId"): + yield self._entry(content, True) + elif prefix == "SA": + for sub in record.get("subSamples", []): + yield self._entry(sub, True) + elif prefix == "SS": + sample = record.get("sample") + if sample and sample.get("globalId"): + yield self._shortcut_entry(sample) + for field in self._file_fields(record): + yield self._field_entry(field) + for attachment in record.get("attachments", []): + yield self._file_entry(attachment) + + # ------------------------------------------------------------ info + + def _info_of(self, path: str, target: Target) -> dict: + if target.kind == "field": + return self._field_entry(self._field(target.parent_gid, target.gid)) + if target.kind == "root": + return make_entry("", True, raw={"name": "Inventory"}) + if target.kind == "section": + return self._virtual_dir(target.section, {"section": target.section}) + if target.kind == "attachment": + attachment = self.inv_client.get_attachment_by_id(target.gid[2:]) + return self._file_entry(attachment, target.gid) + record = self._fetch(target.gid, include_content=False) + if target.kind == "shortcut": + return self._shortcut_entry(record) + return self._entry(record, True, target.gid) + + # ------------------------------------------------------------ files + + @deletes + def rm_file(self, path: str) -> None: + path = self._strip_protocol(path) + target = self._resolve(path) + if target.kind != "attachment": + raise IsADirectoryError(errno.EISDIR, f"{path!r} is not an attachment", path) + self.inv_client.delete_attachment_by_id(target.gid[2:]) + self.invalidate_cache() + + def _download(self, file_id: str, file: BinaryIO, chunk_size: int) -> None: + self.inv_client.download_attachment_by_id(file_id, file, chunk_size) + + @writes + def upload_fileobj(self, path: str, file: BinaryIO, **options: Any) -> dict: + """ + Write ``file`` to ``path``: the last segment names the attachment to create and its + parent is the record (container, sample, subsample, instrument) or the attachment + field to put it on. Returns the new attachment record. + + By default the bytes go to the **Gallery** and the record is given a link to them, + which is what the web interface's "link from Gallery" does and what an ELN upload + already did. The alternative, which the API also offers, creates a file that exists + only inside Inventory: invisible in the Gallery, not reusable anywhere else, and + impossible to find again except through the record it hangs off. Pass + ``via_gallery=False`` for that behaviour. + """ + path = self._strip_protocol(path) + container, name = self._upload_target(path) + target = self._resolve(container) + if target.kind == "field": + # An attachment field holds one file, and RSpace *replaces* it on a second upload + # rather than adding. A folder looks like something you add to, so silently + # destroying its contents would be a nasty surprise: refuse, and say what to do. + # Checked before anything is uploaded, so a refusal leaves nothing behind. + existing = self._field(target.parent_gid, target.gid).get("attachment") + if existing: + raise FileExistsError( + errno.EEXIST, + f"{container!r} already holds {existing.get('name')!r}; an attachment field " + f"takes a single file and uploading would replace it. Remove the existing " + f"file first.", container) + elif target.kind not in ("record", "shortcut"): + raise NotImplementedError( + f"{container!r}: files attach to a record or one of its attachment fields") + if target.gid[:2] not in ATTACHABLE_PREFIXES: + # Checked before the Gallery upload: the server would refuse the link afterwards + # and the uploaded file could never be deleted. + raise NotImplementedError( + f"{container!r}: RSpace does not accept attachments on " + f"{'a bench' if target.gid[:2] == 'BE' else 'a sample template'}. " + f"Attach to a container, sample, subsample or instrument instead.") + result = self._attach(target.gid, file, name) + self.invalidate_cache() + return result + + def _attach(self, parent_gid: str, file: BinaryIO, name: str) -> dict: + """Put ``file`` on the record or field ``parent_gid``, through the Gallery unless + ``via_gallery`` is off.""" + if not self.via_gallery: + return self.inv_client.upload_attachment_by_global_id( + parent_gid, file, **self._upload_kwargs(name)) + # No folder id: the Gallery files it into the section its media type belongs to, so + # this cannot hit the section-mismatch rule that a chosen folder would. + uploaded = self.eln_client.upload_file(file, **self._upload_kwargs(name)) + # by global id, not through link(): a field global id does not resolve as a path on + # its own, because a field is only addressable underneath its record. + return self.inv_client.attach_gallery_file_by_global_id(parent_gid, uploaded["globalId"]) + + @writes + def link(self, path: str, media_file_gid: str) -> dict: + """Attach a file already in the Gallery to the record or attachment field at ``path``, + without copying the bytes. The attachment reports the Gallery file it points at as + ``mediaFileGlobalId``, and removing it leaves the Gallery file alone.""" + path = self._strip_protocol(path) + target = self._resolve(path) + if target.kind not in ("record", "shortcut", "field"): + raise NotImplementedError(f"{path!r}: a Gallery file links to a record or an attachment field") + result = self.inv_client.attach_gallery_file_by_global_id(target.gid, media_file_gid) + self.invalidate_cache() + return result diff --git a/rspace_client/fs/paths.py b/rspace_client/fs/paths.py new file mode 100644 index 0000000..0abb447 --- /dev/null +++ b/rspace_client/fs/paths.py @@ -0,0 +1,142 @@ +""" +Path grammar helpers shared by the RSpace filesystems. + +A path is a slash-delimited sequence of segments. Below a filesystem root each +segment addresses an RSpace record, and there are three ways of writing one: + + name ``microscope.png`` the record's own name + labelled ``microscope.png [GL100]`` name and global ID + id ``GL100`` the bare global ID + +The *path style* decides which of these a listing produces. A segment carrying a global +ID is accepted under every style, so a path copied from a labelled or id listing resolves +anywhere; a bare name only resolves under the ``name`` style, where the lookup exists. +A segment carrying a global ID resolves in one call and survives a rename. A bare +name has to be looked up among its parent's children, and where a folder holds two +records with the same name the listing appends the global ID to the second and +later ones, so every segment in a listing stays unique and addressable. + +Each filesystem adds its own vocabulary on top of this (the Inventory sections, for +instance); what lives here is only what all of them share. +""" +from __future__ import annotations + +import re +from typing import Optional, Tuple + +from posixpath import basename + +GLOBAL_ID = re.compile(r"^[A-Z]{2}\d+$") +LABELLED = re.compile(r"^(?P