[botandbang]

What and why

This builds on the plugin system from #67, #68, #74, #79, #82 and #85.

locallibrary could read a track's beat grid, phrases and waveforms from its analysis files (ANLZ0000.DAT, .EXT, .2EX), but the only things it could write there were cue points and a whole-grid time shift. An analyzer plugin could only answer columns (numbers, strings, booleans). With this PR locallibrary writes beat grids, phrases and waveforms into the analysis files as a journaled, undoable edit, and an analyzer can work them out. What an analyzer works out is only drawn, in the player, for a track whose drive has none, until the user asks for it to be written.

Writing a section of an analysis file

• One function replaces, adds or removes a single tagged section (tags.rs). A section that's missing goes where rekordbox 6 puts it, by the tag order seen in its exports across the three kinds of file (tags.rs). Every other section comes through byte for byte and in its place, which it checks on what it wrote, and anything after the declared file length is kept. A file holding the tag twice is refused rather than guessed at (tags.rs).
• Each writer reads back what it wrote with the existing readers and fails when that isn't what it was given, so nothing is written that locallibrary's own reader would take differently.

Beat grids (PQTZ, in .DAT)

• write_beat_grid writes the beats as rekordbox lays them out: header 0x18, the two unexplained header words kept as the file has them (rekordbox 6's for a new section), the count, then each beat's number in its bar, tempo in hundredths of a BPM and time in ms (grid.rs).
• checked_grid refuses what the format can't hold: no beats, a beat number outside 1-4, a tempo outside 0.01-655.35 BPM (NaN and infinities included), and beats that aren't strictly in time order (grid.rs).
• The .EXT's extended grid (PQT2) is taken out when a new grid is written (grid.rs). Its body isn't understood well enough to write, and leaving one that contradicts the new grid seemed worse than leaving none. Shifting a whole grid by some milliseconds (the existing shift_times) still keeps it.

Phrases (PSSI, in .EXT)

• write_song_structure writes the mood and phrases, or removes them (structure.rs). Rewriting keeps the file's header bytes (lighting bank included) and every byte of each phrase entry whose start, kind and fill-in stay the same, so the variation flags rekordbox set survive. A new entry gets flags of 0.
• It's masked with the same XOR mask the reader undoes, offset by the phrase count, when the file's was masked (and for a new section, as rekordbox 6 writes them) (structure.rs). The reader takes a mood over 20 for masked, so when masking would give a mood of 20 or less (a mid-mood track with exactly 53 phrases), it's written unmasked instead, which reads back right.
• Phrases are refused unless they start on beat 1 or later and follow one another with no gaps, each of a kind the track's mood has a name for, with any fill-in on beat 1 or later (structure.rs).

Waveforms (PWAV, PWV2, PWV3, PWV5, PWV6, PWV7)

• write_waveform writes a tag's columns as given, keeping the file's header fields and using rekordbox 6's for a new section (waves.rs, waves.rs). Only whole columns are taken.
• waveform_from_bands works out every tag except PWV4 from one three-band waveform (lows, mids and highs, 0-255 each, 150 columns a second, Bands in loclib-model) (waves.rs). The previews are pooled to rekordbox's widths (400, 100 and 1200 columns), each band the loudest of the columns it covers. Blue columns are the loudest band's height in 5 bits with the highs' share as whiteness in 3; PWV2 is a 4-bit height; PWV5 is red, green and blue (lows, mids, highs against the loudest) in 3 bits each and the height in 5; the 3-band tags are mids, highs and lows, shifted to the 0-63 and 0-127 ranges beat-link draws them from.
• A plugin's waveform at another rate is pooled to 150 columns a second, and one that would come to more than a day of audio is refused rather than pooled out to fill memory (waves.rs).
• The readers also read PWV6 and PWV7 from the .2EX for a track that has no other waveforms, drawn in a new 3-band style (anlz.rs).

New analysis files

• new_analysis_files makes a .DAT and .EXT for an audio file: the PMAI header and PPTH path as rekordbox 6 writes them, and the empty cue lists it writes for a track without cues, for the writers above to add to (tags.rs). Nothing in this PR uses it yet: the edit below needs a track's existing files. It's there for building a drive from another library, which will need files for tracks rekordbox never analysed.

The edit (set_analysis)

• Edit::SetAnalysis {track, beats?, phrases?, waveform?} writes any of the three into one track's analysis files (edit.rs, analysis_edit.rs). It's made like a cue edit: the files are backed up first, written through the commit path's atomic replace, and recorded in the journal, so undo puts them back byte for byte and edits replay makes it again after rekordbox exports to the drive.
• A grid brings the track's BPM with it: planning adds an edit_tracks with the tempo most of its beats have, recorded with it so a replay doesn't add another (edit.rs). Cues stay where they are when the grid moves, and the plan says how many sat on beats of the old grid. A beat past the track's end is refused, and phrases running past the grid's last beat are warned about.
• A waveform is given in base64 and staged on the drive by its hash, in .locallibrary/waveforms/<hash>.bands, so the journal names it and a replay writes exactly the same tags. The journal keeps only the staged files its entries name and carries them to another drive when it's copied, as it already did for artwork images, which now share that code (staged.rs).
• The journal's format goes to version 4, written only when an entry holds a set_analysis edit, so a drive that never had one stays readable by older builds (edit.rs).
• The edit is refused, as it's planned and with nothing on the drive changed, for a track with no analysis files, one whose .DAT is missing, phrases for a track with no .EXT, and a waveform for a track whose files hold rekordbox's colour overview (PWV4) or the .2EX's PWVC (analysis_edit.rs). Those two can't be worked out from another waveform, so that track's waveforms stay rekordbox's. An analysis path outside PIONEER/USBANLZ/ is refused too, as cue edits already refuse it.

Analyzers that work them out (the analysis feature)

• Analyzer fields may now be of type beats, phrases or waveform, one of each at most (hooks/analyzers.rs, hooks/analysis.rs). A waveform's value is {blob, columns_per_sec}, the blob a file in the request's scratch folder holding the packed bands. Output::of reads a value as its type says, and FieldValue::beats, phrases and waveform make them (hooks/analysis.rs).
• The feature has no method of its own. A manifest declaring such a field needs it, so a plugin built against an SDK without it is refused as it starts.
• The host keeps these fields apart from the columns: they never show in the track list, filters or sorts. They're checked (a grid as the edit checks one, a waveform read from its blob and put at 150 columns a second), and anything that isn't what it says fails that track, saying why (plugin/analyzers.rs).
• They're kept in an overlay store beside the field store, keyed the same way, in <store>/cache/overlay/<plugin>/<analyzer>/<version>/ (fields/overlay.rs).
• Plugins::plan_apply_analysis turns what one analyzer kept into an edit plan: one set_analysis per track that would change, planned and confirmed as any edit is (plugin/analysis.rs). It only writes where a track has none unless asked to overwrite. Phrases count the analyzer's own beats, so they're only written with its grid or onto the same one. Tracks the analyzer hasn't done yet, and those whose files can't take what's asked, are left out with a warning.
• An operation can plan the same edit with the new analysis:plan capability, giving its waveform as a file in its scratch folder, which the host reads as it plans (plugin/operations.rs).

Command line

• analyze --apply grid,phrases,waveform (with --overwrite) runs the analyzer and then plans writing what it worked out (plugin_cmd/analyze.rs). Keys and BPMs are applied in a run of their own, and several analyzers that could answer must be narrowed to one.
• plugin list and plugin info describe analysis:plan in plain words.

Sidecar

• plugins.apply_analysis.plan {plugin, analyzer, drive, parts, tracks?, overwrite?} and plugins.apply_analysis.apply (rpc/methods/plugins/analysis.rs). docs/rpc/ is regenerated.

Player

• When a track starts, the deck fills in what its drive lacks from the overlay store, each part from the first analyzer that has it, and says so under the track's details ("grid and waveform from echo.heard, not on the drive"). This is read on the background thread that already loads a track's analysis, never on the frame path (tui/app.rs).

Design choices

• No PQT2 writer. Its body has two bytes a beat whose first doesn't count beats in the bar in rekordbox 6 exports and whose second nobody has explained, so a written one could only be a guess. A new grid takes it out instead.
• No PWV4 writer, and waveform edits are refused for tracks that have one. Its first two bytes per column aren't understood. Removing it would leave a track with a colour detail waveform and no colour overview, and writing the rest beside rekordbox's PWV4 would leave the overview and the detail disagreeing.
• Tracks without analysis files are refused. Making their files needs paths in rekordbox's USBANLZ layout written into the databases as well, which belongs with building whole drives from another library.
• The waveform models aren't merged. The new Bands type covers what plugins give, every tag is worked out from it, and it draws through the existing two-layer renderer, so merging the colour and mix waveform models wasn't needed to draw plugin waveforms.
• Beats use the existing Beat model (beat_number, bpm, time_ms), which the readers and the XML export already use, rather than a second shape for the same thing.
• The library list (browse) doesn't draw overlays. It draws no grid, phrases or waveform from the drive either, so there's nothing for an overlay to fill in there.
• There's no apply action in the player yet. Writing is done from the command line or the sidecar, both of which show the plan and confirm it.

How it was tested

New and updated tests:

• Writers (crates/locallibrary/tests/it/props_anlz_write.rs): rekordbox_files_rewrite_byte_for_byte rewrites the grid, phrases and every waveform tag of rekordcrate's real rekordbox 6 files with what they hold and gets the same bytes. Hegel properties beat_grids_read_back_as_written, phrases_read_back_as_written and waveforms_read_back_as_written write drawn data into real, synthetic and new files and check that the reader and rekordcrate's independent parser read it back, that every other section is unchanged, and that writing again changes nothing. waveforms_are_worked_out_from_bands checks each tag's mapping, bands_over_a_day_are_refused the length limit, and writers_take_damaged_files_without_panicking damages section headers and cuts files short. Example tests cover the refusals, changing_a_phrase_keeps_the_others_entries and the tag order of new files.
• The edit (crates/locallibrary/tests/it/analysis_edit.rs): a_beat_grid_is_written_with_its_bpm_undone_and_replayed, phrases_are_written_into_the_ext, a_waveform_is_written_staged_and_replayed, cues_on_beats_stay_put_when_the_grid_moves, a_waveform_is_refused_for_a_track_with_rekordboxs_colour_overview and analysis_that_cant_be_written_is_refused. The Hegel property any_grid_phrases_and_waveform_read_back_and_undo applies drawn grids, phrases and waveforms and restores the backup byte for byte, analysis_edits_that_cant_be_made_change_nothing draws each way an edit is refused and checks that the drive is unchanged and no waveform is staged, and any_analysis_edit_warns_of_what_it_leaves_and_replays_as_made draws a moved or kept grid, phrases, a waveform on a track with or without an .EXT and cues on the old grid's beats, checks the warnings for cues left off the new grid and for a waveform with only an overview to go in, then replays the journal on a drive that already has the edit (nothing changes), after rekordbox exports again (the same files, the waveform read from its staged copy) and with the .DAT gone (skipped with a warning), or sets a second grid, which has no PQT2 left to lose.
• Analyzers (crates/locallibrary/tests/it/plugins/analysis.rs), running the test plugin, whose heard analyzer now answers structured fields: structured_fields_are_kept_apart_from_the_columns, structured_fields_that_arent_what_they_say_fail_the_track, the_analysis_is_written_as_an_edit, only_what_the_analyzer_works_out_is_written, structured_fields_need_the_analysis_feature and an_operations_analysis_edit_takes_analysis_plan, and the Hegel properties what_the_drive_lacks_is_drawn_from_the_overlay, any_structured_answer_is_kept_and_planned_as_given, overlays_read_back_as_kept_and_refuse_what_isnt_theirs, writing_an_analysis_is_refused_for_what_isnt_there, waveforms_over_a_day_fail_the_track, any_structured_answer_is_kept_or_fails_the_track_saying_why (nulls and an empty grid keep nothing; a beat out of its bar, out of tempo range or out of order, and a waveform file that's missing or isn't three bytes a column fail the track with the reason) and any_kept_analysis_is_planned_as_asked_and_left_out_as_said, which draws what the analyzer kept for each track, the parts asked, overwrite, one track or all and whether a track's files can take a waveform, then checks the plan's summary, each track's set_analysis edit and the warnings for tracks not analysed yet, parts the tracks already have, phrases on another grid and tracks left out, and that the drive matches the plan once it's applied.
• loclib-plugin's manifest tests and the SDK's hello test cover the new field types, the feature and the capability. loclib-model has tests for Bands. The app has command-line tests for analyze --apply with the new parts and the player's deck line, and the Hegel property analyze_apply_writes_the_parts_asked_where_it_can (apps/locallibrary/tests/it/commands/analysis.rs) runs analyze --apply with drawn parts, --overwrite, --dry-run, --json and tracks whose audio the analyzer fails on. It checks the run line, the plan's summary and warnings, and the grids on the drive afterwards, and that the refused runs (a key or BPM asked alongside, --min-confidence, several analyzers and none named, an analyzer that doesn't work the part out) are refused before anything runs, and that a run where every track fails is an error. A key or BPM asked alone is written from the analyzer chosen for it, the sound plugin's over heard's.

Needs testing on players

None of this has been tried on hardware yet. Each needs a drive written by this branch on a CDJ or XDJ:

1. A track whose .EXT lost its PQT2 after a new grid: does a CDJ-3000 (and a nexus 2 player) still load it, show the grid and quantize to it?
2. A written PQTZ grid: beat and bar markers, sync and quantize land where they should, including a grid with tempo changes.
3. Written PSSI phrases: the phrase bar on the player, and Lighting mode on a mixer or rekordbox lighting reading them. Also a mid-mood track with exactly 53 phrases, which is written unmasked.
4. Waveforms worked out from bands: the blue (PWAV, PWV2, PWV3), colour detail (PWV5) and 3-band (PWV6, PWV7) waveforms look like the track on players that draw each.
5. The layout of new files from new_analysis_files (PMAI, PPTH, empty cue lists, then sections in rekordbox's order), once something uses it.
6. Cues after a grid move: memory and hot cues stay at their times and still play from there.

Checklist

• Tests cover the change
• For the website: Storybook stories cover any new UI states (no website change)
• For locallibrary: apps/locallibrary/CHANGELOG.md has a line under Unreleased, if people using it will notice
• The READMEs, docs/ and --help still say what the code does (apps/locallibrary/docs/rpc/ too, for a sidecar change)

[github-actions]

Coverage Report for apps/web

Status	Category	Percentage	Covered / Total
🟢	Lines	92.58% (🎯 91%) 🟰 ±0 pp	4879 / 5270
🟢	Statements	90.96% (🎯 89%) 🟰 ±0 pp	7517 / 8264
🟢	Functions	92.19% (🎯 90%) 🟰 ±0 pp	1901 / 2062
🟢	Branches	78.94% (🎯 77%) 🟰 ±0 pp	3427 / 4341

File Coverage

No changed files found.

Generated in workflow #287 for commit 10fdc6e by the Vitest Coverage Report Action

Size

Compared with main. KiB are 1,024 bytes.

	This pull request	main	Change
Client JavaScript	584.5 KiB	584.5 KiB	0 B (0%)
Client JavaScript, brotli	193.6 KiB	193.6 KiB	0 B (0%)
Client CSS	84.8 KiB	84.8 KiB	0 B (0%)
Client CSS, brotli	12.3 KiB	12.3 KiB	0 B (0%)

[github-actions]

Rust coverage

From run 275. Both are held to the thresholds in .cargo/config.toml.

Tests	Lines	Functions	Regions
All	97.43% (75171/77153)	96.67% (11867/12276)	95.76% (164664/171948)
Hegel property tests alone	95.65% (73799/77153)	94.44% (11593/12276)	93.82% (161328/171948)