Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion content/docs/(overview)/quickstart.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ The CLI asks for a password for the owner, then for the setup code the server pr
</Tab>
<Tab value="In the browser">

Open `http://localhost:8600/setup`, enter the setup code the server printed, and choose the owner's email and password.
Open `http://localhost:8600/setup`, enter the setup code the server printed, and choose the owner's email and password. **Claiming signs you in**, and setup goes on from there to add a passkey, name the instance, connect something and read how it's backed up.

</Tab>
</Tabs>
Expand Down
19 changes: 19 additions & 0 deletions content/docs/(reference)/changelog.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,25 @@ Each release of Marfa ships the server and its container image, the CLI for macO

**Until the first public release, Marfa does not upgrade a database in place.** A build refuses a database whose schema differs from its own or that holds retired tables or settings, and leaves the file unchanged. If the build refuses your database, [Upgrade](/guides/upgrade) covers exporting with the old build and restoring into fresh storage. A version change alone does not cause a refusal.

## v0.0.17

Released October 20, 2026. [v0.0.17 on GitHub](https://github.com/withmarfa/marfa/releases/tag/v0.0.17)

For running an instance:

- **A blob folder that has gone is no longer read as an empty one.** When the disk store's folder or its `.marfa-store` marker is missing, as when its drive is unmounted while the server runs, `GET /health` answers `blob_storage` as `down` with status `503`. The `blob-integrity` job strikes none of that store's copies, counts the store in `unreachable` in its result, and still checks the stores it can reach. A blob read answers `503 blob_store_unavailable` rather than `404 blob_not_found` when a store that holds a copy can't answer and no other store returns the bytes. A restore that finds the disk store's marker missing, or naming another store, when it places a blob answers the same and writes nothing. Replication doesn't end its run on such a store, and the `enrichment-sweep` job doesn't count the attempt against the file. [Deploy](/guides/deploy#keep-it-running) covers the health check.
- **The bin purge keeps going through a backlog.** `trash-purge` removes a limited number of items in a run. While items past the retention remain after a run that removed some, the job runs again at once instead of waiting for its next interval, so an instance that bins more items in an interval than a run removes no longer falls behind. [Housekeeping](/guides/housekeeping#empty-the-bin) covers the job.
- **A restore keeps an item or an edge as the archive holds it.** An item whose type the instance no longer registers, such as after a forced delete, or that lacks a field its type later required, is restored with the properties and `schema_version` the archive holds. A restore also keeps an edge whose edge type was force-deleted, and no longer judges an end that is an item of a force-deleted type against the edge type's constraints. Such an edge is still skipped for the other reasons, such as a missing end. An export no longer refuses these with `409 export_rows_unrestorable`, so an instance that holds them can be exported again. An item of a `core` or `system` type that the build ships is still held to its shape. [Back up and restore](/guides/back-up-and-restore#restore-an-archive) covers it.
- **A restore refuses an archive that lacks one of its four files.** An archive without `manifest.json`, `items.ndjson`, `edges.ndjson` or `types.ndjson` is refused `400 validation_error`, naming each missing entry in the message and in `details.entries`. Before, it answered `200` with counts of zero.
- **A restore or an upload is no longer cut off while its bytes arrive.** An admitted upload has no total time limit. `MARFA_UPLOAD_IDLE_TIMEOUT_MS`, 60000 milliseconds unless set, ends one that receives no bytes for that long with `408 request_timeout`, and a restore then writes nothing and an upload stores no blob. The owner's recent sign-in is judged when a restore starts, so a restore that starts within five minutes of the sign-in isn't refused for finishing later. A session that signed in more than five minutes before the restore starts is refused `403 forbidden` without waiting for the body to finish arriving, and one that ends during the upload is answered `401 unauthorized`. [Settings](/guides/settings#every-environment-setting) lists the setting.
- **`marfa export --output` keeps the last whole archive.** When the new archive lacks blobs, the CLI doesn't replace the file at the path. It keeps the archive beside it as `<path>.incomplete`, replacing only an earlier file of that name, and exits `1` with `blobs_missing`. The message says whether the path still holds an earlier file. If the CLI can't write that name, it keeps the archive in a hidden file beside the path and names the file. [Back up and restore](/guides/back-up-and-restore#when-an-archive-lacks-a-files-bytes) covers it.
- **Claiming in a browser signs the owner in, and setup is six steps.** A claim that carries the instance's own `Origin` leaves the browser signed in, as a sign-in does, and that counts as a sign-in within the last five minutes, so adding a passkey needs no second prompt. A claim that sends no `Origin`, such as a script's, and a claim that is refused set no session. If the sign-in that follows a claim fails, the owner stays claimed and signs in at the page that follows. Setup runs as six steps: the setup code, the owner's details, a passkey, a name for the instance, connecting, and backups. [Quickstart](/quickstart#claim-the-instance) covers claiming.
- **The sign-in page says how long a lockout lasts.** When password sign-in or a device code lookup is refused for too many attempts, the page states the wait in whole minutes, rounded up, and the sign-in page says that a passkey still signs in. The wait runs to the time another attempt is allowed, the latest end among the limits that are spent. The sign-in page also links to a Forgot password page at `/auth/forgot-password`, which gives the command that recovers the owner on the instance's machine. [Troubleshooting](/guides/troubleshooting) covers it.

For apps and devices:

- **An export ends with an error when its credential ends.** If the key or signed-in app that started `GET /export` is revoked, expires or is disconnected while the response is open, the response ends with an error instead of ending cleanly, in both formats and while an archive sends a blob. The same happens if the credential is narrowed so that it no longer reads something the export has still to send. A narrowing that takes away nothing the export has still to send changes nothing. An archive export whose key is revoked while it reads the selection, before the response begins, gets `401 unauthorized`. [Back up and restore](/guides/back-up-and-restore#what-an-export-checks) covers it.

## v0.0.16

Released October 19, 2026. [v0.0.16 on GitHub](https://github.com/withmarfa/marfa/releases/tag/v0.0.16)
Expand Down
22 changes: 15 additions & 7 deletions content/docs/guides/back-up-and-restore.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,8 @@ S3_PREFIX=blobs litestream restore -config deploy/litestream.yml -o restored.db

The [deployment guide](https://github.com/withmarfa/marfa/blob/main/deploy/README.md#restoring-by-hand) has the same command for a point in time.

**A copy of the volume that is older than the bucket needs a prefix of its own.** Started under the bucket's current `S3_PREFIX`, it becomes the bucket's newest copy of the database. The deployment guide's [restore of a backup older than the bucket](https://github.com/withmarfa/marfa/blob/main/deploy/README.md#restore-a-backup-that-is-older-than-the-bucket) gives the steps.

## Export an archive

**An archive holds what its caller can read.** A live owner browser session can export without a key or recent reauthentication. For automated exports with the CLI, make a key that reads everything and use it only for exports. The steps below use that key.
Expand Down Expand Up @@ -95,17 +97,17 @@ types.ndjson
blobs/sha256:a0feecd7233d9f5f3ec784c6bdb36587c651e2ec60437c27b4bf135de39149d7
```

If the server fails partway through an export, it ends the connection early, and the file doesn't unpack. **`marfa export --output` writes to a file beside the path and renames it only once the stream has ended whole.** A stream cut short removes that file, leaves any earlier file at the path as it was, and ends with exit code 3, so try again.
If the server fails partway through an export, it ends the connection early, and the file doesn't unpack. **`marfa export --output` writes to a hidden file beside the path and renames it over the path only once the stream has ended whole and the archive lacks no blobs.** A stream cut short removes that file, leaves any earlier file at the path as it was, and ends with exit code 3, so try again.

</Step>
<Step>

### Check what the archive lacks

The manifest names each file's bytes the archive couldn't carry, under `missing_blobs`. Read it from the archive:
The manifest names each file's bytes the archive couldn't carry, under `missing_blobs`. When `marfa export` exits with code 1 and `blobs_missing`, the archive is at `marfa-archive.tar.gz.incomplete`, not at `marfa-archive.tar.gz`. Read the manifest from it:

```bash
tar -xzOf marfa-archive.tar.gz manifest.json
tar -xzOf marfa-archive.tar.gz.incomplete manifest.json
```

The manifest's other fields are left out here:
Expand All @@ -132,7 +134,7 @@ An empty `missing_blobs` means the archive holds every blob its items name. The

An export never claims a blob it doesn't hold. When no store holds a whole copy of a blob, such as after a disk lost the file, the export **leaves the blob out of the archive** and names it in `missing_blobs`, with its `mime_type` and `size_bytes`. The rest of the instance is still backed up: one lost file costs the backup that file, not everything.

`marfa export` writes the archive, then **exits with code 1 and `blobs_missing`**, naming up to ten of the hashes and counting the rest. A script that reads only the exit code can't take such an archive for a whole one. The HTTP answer is still `200`, so read the manifest if you export without the CLI.
`marfa export` **exits with code 1 and `blobs_missing`**, naming up to ten of the hashes and counting the rest. **With `--output`, it doesn't replace the file at the path.** It keeps the archive beside it as `<path>.incomplete`, replacing only an earlier file of that name, so the last whole backup stays where a script reads it. If it can't write that name, it keeps the archive in a hidden file beside the path and names the file in its message. A script that reads only the exit code can't take such an archive for a whole one. The HTTP answer is still `200`, so read the manifest if you export without the CLI.

A file item can name bytes the instance has no record of, as after a restore that lacked them. The export names such a blob too, with `null` for both `mime_type` and `size_bytes`, because nothing knows them. **Every later backup keeps naming a file's blob it lacks until the bytes are uploaded again**, with `marfa blobs upload` or [Upload a blob](/api-reference/blobs/uploadBlob). Only a file item's `blob_ref` counts: a hash that appears in a note's text is not named.

Expand All @@ -142,9 +144,11 @@ A store that fails, or doesn't answer within 15 seconds, is not a lost file. The

**An export isn't taken at one instant**: an item created while it runs may not be in it. Stop your apps first if the archive must hold every write.

**An export ends when its credential does.** If the key or signed-in app that started it is revoked, expires or is disconnected, or is narrowed so that it no longer reads something the export has still to send, the server ends the response with an error, in both formats, and an archive cut short this way doesn't unpack. A narrowing that takes away nothing the export has still to send changes nothing.

**Before it sends a byte, Marfa checks the whole archive the way a restore into an empty instance would.** An archive that Marfa sends restores into an empty instance of the same build without skipping an item or an edge, even when writes landed between its pages. If a row fails the check, Marfa reads the selection once more. If the second read fails too, Marfa answers one of two ways, and neither sends an archive:

- **`409 export_rows_unrestorable`:** both reads refused the same row, and the row didn't change between them. The cause is in your data, such as a row that a type's replacement schema no longer accepts. `details.row` names the row by `kind`, `id`, `type` and `field`, with the `code` a restore refuses it with or the reason it skips it. Correct the row or its type, then export again.
- **`409 export_rows_unrestorable`:** both reads refused the same row, and the row didn't change between them. The cause is in your data, such as a type replaced to become an event while one of its rows holds a recurrence rule that can't be read. `details.row` names the row by `kind`, `id`, `type` and `field`, with the `code` a restore refuses it with or the reason it skips it. Correct the row or its type, then export again.
- **`503 export_unrestorable`:** writes changed the refused row or its type between the two reads, or the second read refused a different row, so the data may be fine once the writes stop. Export again, or stop your apps first.

The server prepares the whole archive before it sends the first byte, in a folder inside the blob folder, so leave room on that disk for about as much as the archive's text takes uncompressed. An export that would take the volume below `MARFA_DISK_RESERVE_BYTES` is refused `507 insufficient_storage`, as an upload or a restore is.
Expand All @@ -153,7 +157,7 @@ A [source filter](/guides/settings#what-the-enforcement-levers-do) leaves items

## Restore an archive

**Only the owner can restore**, through the private socket or signed in to the instance within the last five minutes. A key is refused `403 forbidden`, whatever it holds. The target can be a new instance or the one the archive came from:
**Only the owner can restore**, through the private socket or signed in to the instance within the five minutes before the restore starts. A key is refused `403 forbidden`, whatever it holds. The target can be a new instance or the one the archive came from:

```bash
marfa --socket /srv/marfa/data/control/marfa.sock restore marfa-archive.tar.gz
Expand Down Expand Up @@ -183,17 +187,21 @@ A restored `system.connection` item for an app has `status: revoked`. It records

**An edge to an item in the bin is restored**, as it was exported. A restore skips an edge only for one of four reasons, which `edges_skipped_reasons` counts: `already_present`, `endpoint_missing` (the instance holds no item under an endpoint's ID, in the bin or out of it), `malformed`, or the error code that `POST /edges` would answer for the edge, such as `edge_cycle`. An archive's types restore in any order, including types that name each other in `compatible_with`.

**A restore takes an item as the archive holds it** when its type isn't registered, as after a forced delete, or when it lacks a field its type later required or holds a value the type's field no longer accepts. An item of a `core` or `system` type that the build ships is still held to its shape. A restore also keeps an edge whose edge type isn't registered, and doesn't judge an end whose type isn't registered against the edge type's constraints; the other reasons to skip an edge still apply.

An item the instance already holds is counted in `duplicates` and left as it is, so replaying an archive never overwrites work done since. Restoring the same archive again into the same instance answers `"imported": 0` and `"duplicates": 5`.

`blobs_missing` lists the hashes the archive names but doesn't carry bytes for, that no store of the instance holds either. The restore still writes the items, and **writes no record for those blobs**, so a file that names one is a file with no bytes until you upload them again with `marfa blobs upload`. `marfa restore` prints the answer and exits with code 1 and `blobs_missing` when the list isn't empty, naming up to ten of the hashes. The same check covers an archive whose blob entry is missing or doesn't hash to its name.

A restore writes in one transaction. If it's refused, fails or is interrupted, it leaves nothing behind, so you can run it again. While it runs, other writes wait for it, so let it answer before you start your apps.

**A restore has no time limit while its bytes arrive.** If none arrives for `MARFA_UPLOAD_IDLE_TIMEOUT_MS`, 60 seconds unless you set it, the server answers `408 request_timeout` and writes nothing. A browser session that ends during the upload is answered `401 unauthorized`, also with nothing written.

**An archive never holds keys, webhooks, the configuration or the audit log**: make keys, register webhooks and set the configuration again on the new instance.

Before Marfa's first release, only the build that wrote an archive is sure to read it, so **an archive isn't a long-term backup**. A restore refuses the whole archive, and writes nothing, for one of these reasons:

- **`400 validation_error`:** the file isn't a valid archive, is at another format version, repeats an entry, has an invalid row, holds a file over 64 MiB, or holds text that isn't well-formed UTF-16. For that text, the answer names the entry or the row and the JSON pointer of the text.
- **`400 validation_error`:** the file isn't a valid archive, lacks `manifest.json`, `items.ndjson`, `edges.ndjson` or `types.ndjson` (`details.entries` names each), is at another format version, repeats an entry, has an invalid row, holds a file over 64 MiB, or holds text that isn't well-formed UTF-16. For that text, the answer names the entry or the row and the JSON pointer of the text.
- **`400 invalid_schema`:** an archived type is invalid or carries a key the schema doesn't define.
- **`400 invalid_properties`:** an item sets a property its type doesn't declare, and strict mode names that type.
- **`409 conflict`:** the archive redefines a type the instance registers differently, carries a core edge type, or carries a snapshot ID the instance already holds.
Expand Down
2 changes: 1 addition & 1 deletion content/docs/guides/deploy.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,7 @@ Claim it with `marfa --socket <path> setup open`, as the account that runs the s

## Keep it running

`GET /health` answers without a credential. It answers `503` when the database, a write, the disk or blob storage is down, and `200` otherwise, a degraded component included. **Point your supervisor or monitor at it**; the container checks it every 30 seconds.
`GET /health` answers without a credential. It answers `503` when the database, a write, the disk or blob storage is down, and `200` otherwise, a degraded component included. **Blob storage is down when the blob folder or its `.marfa-store` marker is gone**, as when its drive is unmounted while the server runs, and the instance strikes none of that store's copies while it is. **Point your supervisor or monitor at it**; the container checks it every 30 seconds.

`SIGTERM` or `SIGINT` stops the server within eight seconds, so **give it ten before a runtime kills the process**.

Expand Down
Loading
Loading