Skip to content

Update to Litestream 0.5.17 - #78

Open
cole-robertson wants to merge 2 commits into
fractaledmind:mainfrom
Rebulk:litestream-0.5-core
Open

Update to Litestream 0.5.17#78
cole-robertson wants to merge 2 commits into
fractaledmind:mainfrom
Rebulk:litestream-0.5-core

Conversation

@cole-robertson

Copy link
Copy Markdown

Second of two, stacked on #76 (the runner). Depends on #76: this branch contains that commit too, so review the second commit here, or wait for #76 to merge and this diff becomes just the bump. Contains only the version bump and what it forces; the dashboard redesign is a later PR, and so is IPC support, which #73 already implements and which should land on top of this as an optional enrichment rather than the only path.

What changes

  • Binary 0.3.13 → 0.5.17. Tarballs pinned by sha256 for all four assets (Upstream::CHECKSUMS), verified in memory before extraction. A sidecar exe/<platform>/litestream.sha256 records what was packaged, so a cached binary from an earlier version is re-downloaded rather than shipped. No zip assets upstream any more, so rubyzip is dropped.
  • Commands. generations, snapshots, wal are gone in 0.5; ltx and status replace them. status expands a relative database path: Litestream 0.5 filters status by the absolute path while ltx and restore match the config string, so without this the documented --database=storage/production.sqlite3 form printed an empty table. The three removed methods and rake tasks remain for one release as tombstones that raise with a pointer to ltx.
  • Dashboard, minimal. Reads status -json (local, no daemon) and ltx -level all -json (replica). A command failure is isolated to that database; anything else raises. Process detection via systemctl/ps is unchanged and still matches a 0.5 daemon. This deliberately does not use the 0.5 IPC socket: it is off by default and only reachable from the process tree running replicate, so on Kamal or Docker with a separate replicate role a socket-based dashboard shows nothing. The socket does expose better data where it is reachable (a real last-sync timestamp, and a daemon-confirmed replica TXID from sync -wait), which is what Litestream v0.5.9 update #73 builds on. The two compose well: keep these CLI calls as the baseline that always works, and read the socket to enrich them when the config enables it and the file exists.
  • Config. Generator template, dummy config and README in the 0.5 shape: single replica: per database, root-level snapshot: block. README's restore section lists the 0.5 flags; an "Upgrading from 0.3" section covers the config edit, the databases key rename (replicasreplica), the tombstones, and rollback.

Not touched: Puma plugin, engine, VerificationJob, executable, auth, version.rb (left for the release commit).

Verified

  • Fresh Rails 8 app consuming the packaged native gem (bundled binary, no override): install generator, Puma plugin replicating to a local replica, dashboard over HTTP with and without basic auth and with the daemon stopped, the restore button, every rake task including the tombstones, and VerificationJob.

  • 85 tests, 0 failures (the 39 tests for the removed commands are replaced by ltx/status/tombstone tests); standardrb clean; fork CI green.

  • Native Gems workflow on the fork: all six platform packages build; linux-x86_64, darwin-arm64 and plain-ruby installs pass. (darwin-x86_64-install never starts because the workflow asks for the retired macos-13 runner; same on upstream.)

  • rake gem:x86_64-linux from a clean tree: download, checksum OK, gem built; installed into a fresh, isolated GEM_HOME, litestream version → 0.5.17. Planting a wrong sidecar checksum triggers a re-download.

  • Real 0.5.17 binary against a replica holding both 0.3 generations/ and 0.5 ltx/ history: status and ltx in JSON and table modes, the tombstones, and Litestream.databases in the dummy app returning status plus 16 LTX rows.

  • Running in production at Rebulk (Nightrail Cloud): the readiness probe (status + ltx, six databases, every five minutes) and the weekly restore drill go through this gem.

Commands.run joined argv into one shell string, ignored the exit status
and discarded stderr, so a failing command returned "" (or [] after
table parsing) and looked like success. Arguments with spaces broke, and
argv reached the shell unescaped.

The runner now uses Open3.popen3 with an argv array, raises
CommandFailedException with the exit status and stderr on failure, and
takes an explicit output mode: :table (the existing header/rows
parsing), :raw, or :json when a caller passes json: true (Litestream
>= 0.5). In JSON mode the two opt-in restore skips, which print one
logfmt line on stdout with exit 0, come back as {"skipped" => true,
"message" => ...} so callers can tell "did nothing" from data.

timeout: runs the command in its own process group and TERMs then KILLs
it on expiry, raising CommandTimeoutException with the child reaped.
The LITESTREAM_INSTALL_DIR note prints once per process.
The bundled binary moves from 0.3.13 to 0.5.17. Release tarballs are
pinned by sha256 in Upstream::CHECKSUMS and verified in memory before
extraction; a sidecar exe/<platform>/litestream.sha256 records what was
packaged so a cached binary from an earlier version is re-downloaded
instead of shipped. The darwin zip assets are gone upstream, so rubyzip
goes too.

Litestream 0.5 replaced generations, snapshots and wal with TXID-numbered
LTX files. Commands gains ltx and status; the three removed methods and
rake tasks stay for one release as tombstones that explain the change.
The dashboard reads status (local, no daemon) and ltx (replica) through
the same commands, isolates a command failure to the one database, and
lets anything else raise. Generator template, dummy config and README
use the 0.5 shape: one replica: per database, root snapshot: block.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant