[botandbang]

What and why

This builds on the plugin system from #67, #68, #74, #79 and #82, and on the analysis file writers from the beat grid, phrase and waveform PR it branches from.

Until now a drive could be built from rekordbox's collection (collection export) or from another drive, never from anything else. A plugin could read a drive's library and export it to another program, but not bring a library in. With this PR a plugin can read a library from somewhere on the computer (another program's collection, a folder of tagged files) and locallibrary builds a drive from it, through the same path rekordbox's collection takes. The plugin names audio files by path, so everything here is about never copying a file the user didn't grant.

Protocol and SDK (the sources feature)

• library/read {source, options, roots, cursor?, context} answers {tracks, playlists, blob?, next?}, a page at a time; the host passes each next back as cursor until a page has none (sources.rs). roots are the folders the user granted for this build.
• A track is its audio file (absolute), its tags, a key (Camelot or as rekordbox names it), a BPM, cues, a beat grid, a waveform blob and My Tags (sources.rs). A cue's colour is written as cue edits take one. A playlist is a Folder/Name path and its tracks' paths.
• A page too big for a message is a blob in the request's scratch folder holding a page as JSON.
• [[contributes.sources]] name, title, options needs the new capability fs:read (sources.rs, manifest.rs). The feature is optional, so the protocol stays at 0.2.
• WaveformBlob gets a JSON Schema, and loclib-plugin asks for loclib-model's schemars feature itself, since SourceTrack holds a beat grid; before, it only had it through the workspace.

What the host checks

• Plugins::plan_source_export checks the source and its options against the manifest, reads every page, and refuses a cursor answered twice, so a plugin can't page forever (plugin/sources.rs).
• The folders granted must be absolute, there, and not on the drive being written (plugin/sources.rs).
• Each track's path is resolved with its links and must land inside a granted folder, off the drive written to, be a regular file with an extension players play, not be empty, and open as audio. The host reads its length, sample rate and bit depth itself rather than taking the plugin's word (plugin/sources.rs). A file listed twice is kept once.
• What doesn't pass is left out and counted in a warning with a few examples: tracks, playlist entries for them, My Tags that aren't Category/Tag, keys that aren't keys, beat grids out of order, waveforms that aren't waveforms, cues in a slot already taken or past H, loops that end before they start, colours players don't have (plugin/sources.rs, plugin/sources.rs).
• Each audio file the build copies is hashed after planning, and the copy is checked against that hash, so a file swapped or edited between the plan and the write is refused rather than copied (collection_export.rs, library_write.rs).

Building the drive

• collection_export now takes a source-neutral Collected library: rekordbox's collection fills it as before, and the plugin host fills it with the files it checked (collection_export.rs). Both go through the same layout (Contents/<artist>/<album>/), library_write and convert::build_databases, so a source gets what an export of the collection gets: only changed files copied, a backup of what's replaced, history lists saved first, My Tags in OneLibrary.
• A track that brings cues, a beat grid or a waveform gets new .DAT and .EXT files made with the analysis writers, its cues read back and compared before the files are kept (analysis_edit.rs, collection_export.rs). One whose analysis can't be written that way goes without it, with a warning. The made files wait in a temporary folder until they're copied, and one the drive already holds byte for byte isn't copied again.
• The drive remembers which source it was built from. Building again from the same source carries the playlist selection over; building from another source (or over a collection export) counts as a library the user has to ask to replace.
• The export's planning is shared with collection export (api/collection.rs). The plan says which folders were read; it's named <plugin>.source.<source> and its JSON has source.

Command line

• collection export DRIVE --from PLUGIN:SOURCE --grant FOLDER (--grant again for more, --option KEY=VALUE for the source's options, --dry-run, --replace) (collection_cmd.rs). --db and --key are refused with --from. The plan prints the folders read and what was left out.
• plugin info lists a plugin's library sources, plugin list says it adds them, and fs:read reads "read audio files in folders you grant it, to build a drive from".

Sidecar

• plugins.source.plan {plugin, source, options, grants, drive, playlists?, all?, replace?} answers a plan shaped like collection.export.plan's, applied by plugins.source.apply (rpc/methods/plugins/sources.rs). Its plans are kept with the other plugin plans and taken only by plugins.source.apply (rpc.rs). plugins.list has sources.
• docs/rpc/ is regenerated.

Player

• "build from ..." in the palette and a drive's menu, for a drive plugged in, opens a form for the folders it may read (~/Music to begin with, several split by commas), whether to replace a library the drive holds, and the source's options (tui/plugins/sources.rs). The plan is confirmed as "Build DRIVE from the library source PLUGIN:SOURCE?" (or "Replace the library on ...") (tui/edits.rs).

Design choices

• The user grants folders for each build instead of the manifest naming them. A plugin can't know where someone's music is, and a grant kept from one build to the next would let a later build read folders nobody looked at again; given each time, the folders are in the plan the user confirms. The manifest still declares fs:read, so plugin list says up front that it reads files.
• Hard links inside a granted folder are copied. Unlike a request's scratch folder, which a plugin fills itself, a music folder is the user's, and hard-linked libraries are a common way to keep one copy of a file in two places.
• Only files the build copies are hashed. Hashing every file on every build would read the whole library twice for an unchanged drive; a file that isn't copied isn't read at all.
• Phrases, artwork and history lists aren't part of a page yet: a source's tracks get the analysis players need to cue and beat-match, and nothing locallibrary can't read back.
• The echo example has no library source. Adding one would give it fs:read, which many tests match its capability list against; the tests that need a source add it to echo's manifest themselves.

How it was tested

New and updated tests:

• loclib-plugin: a_source_answers_its_library_a_page_at_a_time (the SDK) and a_source_is_granted_reading_folders. The Hegel properties a_drawn_manifest_is_valid_and_reads_back_as_it_was_written and each_kind_of_mistake_is_refused_saying_where draw sources, with a new mistake (a source without fs:read); capabilities_read_back_from_their_names_and_nothing_else covers fs:read.
• Host (crates/locallibrary/tests/it/plugins/sources.rs): a_source_builds_a_drive_from_the_audio_files_it_was_granted (a file outside the grant, a link out of it, text named as audio, a relative path, a missing playlist entry; tags, cues, a grid and a waveform read back from the drive; building again copies nothing), a_source_answers_in_pages_and_blobs, what_a_track_brings_that_cant_be_written_is_left_out, grants_are_checked_and_files_copied_as_they_were_checked (a file changed after planning is refused as it's copied), and the Hegel properties the_drive_holds_exactly_the_granted_audio_files_and_their_cues, every_track_left_out_is_counted_with_why (each reason a track is left out, counted in one warning with the first three examples exactly), pages_and_blobs_add_up_to_the_library (any page size, pages in blobs, playlists with folders and missing entries, a source that pages forever or can't read its JSON), what_tracks_bring_is_written_or_counted_as_left_out (keys, beat grids, waveforms, My Tags, cues and colours, read back from the drive, with a count per kind left out), files_are_copied_only_as_they_were_checked (a file that grows while the plan hashes it refuses the plan; one changed after it refuses the copy) and only_absolute_folders_off_the_drive_are_granted.
• The test plugin's sources read every file in their folders, with what a .json file beside each says, and a folder's playlists.json; options page, answer in blobs or never stop (test-plugin/plugin.rs).
• Command line and sidecar (the app's tests/it/plugins/sources.rs): a_drive_is_built_from_a_plugins_library_source and the_sidecar_builds_a_drive_from_a_library_source; the plan cache's model test plans_are_kept_and_taken_as_modelled covers plugins.source, and the Hegel property plugins_are_listed_and_described_in_text_as_in_json draws library sources. The Hegel property collection_export_from_a_source_is_checked_then_written draws --from, --db, --option, --grant, --replace, a dry run, --yes or neither, and --json, and checks each refusal and what's written.
• Player (tui/plugins/tests/sources.rs): a_drive_is_built_from_a_library_source, and the Hegel property a_drive_is_built_from_a_source_once_confirmed (from the palette or the drive's menu, with drawn folders, replacing or not, cancelled while planning, then confirmed or declined).

Needs testing on players

• A stick built from a library source plays on CDJ-3000s, XDJ-XZ/RX3 and an older player reading export.pdb only: tracks, playlists, tags and keys show as they do from a collection export.
• Cues, loops and hot cue colours a source gives show where they should; beat grids line up with the music.
• Waveforms made from a source's three bands: the blue overview and detail, and how players with a colour overview (PWV4, which isn't written) show a track without one.
• A track whose source brings nothing to analyse has no analysis files: players should load it and analyse it as they do any track rekordbox hasn't.

Checklist

• Tests cover the change
• For the website: Storybook stories cover any new UI states (no website change)
• For locallibrary: apps/locallibrary/CHANGELOG.md has a line under Unreleased, if people using it will notice
• The READMEs, docs/ and --help still say what the code does (apps/locallibrary/docs/rpc/ too, for a sidecar change)

[github-actions]

Coverage Report for apps/web

Status	Category	Percentage	Covered / Total
🟢	Lines	92.58% (🎯 91%) 🟰 ±0 pp	4879 / 5270
🟢	Statements	90.96% (🎯 89%) 🟰 ±0 pp	7517 / 8264
🟢	Functions	92.19% (🎯 90%) 🟰 ±0 pp	1901 / 2062
🟢	Branches	78.94% (🎯 77%) 🟰 ±0 pp	3427 / 4341

File Coverage

No changed files found.

Generated in workflow #290 for commit 73b112f by the Vitest Coverage Report Action

Size

Compared with main. KiB are 1,024 bytes.

	This pull request	main	Change
Client JavaScript	584.5 KiB	584.5 KiB	0 B (0%)
Client JavaScript, brotli	193.6 KiB	193.6 KiB	0 B (0%)
Client CSS	84.8 KiB	84.8 KiB	0 B (0%)
Client CSS, brotli	12.3 KiB	12.3 KiB	0 B (0%)

[github-actions]

Rust coverage

From run 278. Both are held to the thresholds in .cargo/config.toml.

Tests	Lines	Functions	Regions
All	97.42% (75791/77800)	96.61% (11962/12382)	95.75% (166138/173515)
Hegel property tests alone	95.65% (74415/77800)	94.37% (11685/12382)	93.81% (162777/173515)