From 2165e60bc4701bf0643d39e53483714bff63910d Mon Sep 17 00:00:00 2001 From: August Cayzer Date: Wed, 21 Oct 2026 17:15:35 +0100 Subject: [PATCH 1/3] docs(guides): rewrite the type, sync, archive and deploy guides --- content/docs/guides/define-a-type.mdx | 243 ++++++--------------- content/docs/guides/deploy.mdx | 179 ++++++++------- content/docs/guides/export-and-restore.mdx | 114 +++++----- content/docs/guides/keep-in-sync.mdx | 208 ++++++++++-------- content/docs/guides/settings.mdx | 2 +- 5 files changed, 331 insertions(+), 415 deletions(-) diff --git a/content/docs/guides/define-a-type.mdx b/content/docs/guides/define-a-type.mdx index 6ce5ecbc..12443472 100644 --- a/content/docs/guides/define-a-type.mdx +++ b/content/docs/guides/define-a-type.mdx @@ -3,48 +3,11 @@ title: Define a type description: Register a type for items no shipped type fits, then change it as your app grows, or delete it. --- -When none of the types Marfa ships fits what your app keeps, you register a type of your own, and Marfa checks the properties you declare on every item of it. The examples register a type for receipt photos, `app.example.receipt`, built on the shipped `core.file.image`, then change it and delete it. [Items and types](/items-and-types) explains what a type is and when to build on another. The TypeScript examples use a client made as [the TypeScript SDK](/typescript-sdk) shows. +When no type Marfa ships fits what your app keeps, register one of your own, and Marfa checks the properties it declares on every item of that type. This guide registers a type for receipt photos, `app.example.receipt`, then changes it and deletes it. [Items and types](/items-and-types) explains what a type is and when to build on another. - - +## Write the definition -### Choose an identifier - -The identifier's namespace says who the type is for. Use `app..` for your app's own types, `user.` for types you keep for yourself, and a name of your own, such as `acme.deal`, for types you publish for others. The Catalog's [Type rules](/catalog/your-own-types#identifier) page lists what an identifier must meet, including the roots Marfa keeps. - -Name a type with a **singular noun**, `user.recipe` and not `user.recipes`, and write words within a segment and in property names in `snake_case`, as the shipped types do with `blob_ref` and `mime_type`. - - - - -### Declare the properties - -`fields` holds one entry for each property, by name. Each entry needs a `type`, and can refine it: - - - -| Field | Type | Description | Default | -| --- | --- | --- | --- | -| `type` | `string` | One of the [field kinds](/catalog/field-kinds). | | -| `required` | `boolean` | Whether every item of the type must hold the property. | `false` | -| `description` | `string` | What the property holds. | | -| `enum_values` | `string[]` | The values an enum allows. An enum needs it. | | -| `items_type` | `string` | The type of each element of an array. An array needs it, and a write is refused if an element isn't a value of that type. | | -| `maxLength` | `integer` | The longest value a string or enum allows. | | -| `searchable` | `boolean` | Set to false to keep the property out of search. | `true` | - - - -A property **can't share a name** with a field every item has, such as `source` or `version`. The Catalog lists the [reserved field names](/catalog/your-own-types#reserved-field-names). Keep the time the thing happened in the item's own `occurred_at` rather than a property. [Register a type](/api-reference/types/registerType) lists everything a type can declare, such as its label, display hints, merge policy, version policy and retention policy. - -A property you don't declare is **kept by default**. Marfa checks the declared properties, and accepts an item that carries others and serves them back. To refuse an undeclared property, name the type in the instance's strict mode: a write that carries one is then refused `400 invalid_properties`, with `details.code` set to `unknown_property`. Strict mode holds for exactly the types it names, not the types that inherit from them. [What the enforcement levers do](/guides/settings#what-the-enforcement-levers-do) shows how to turn it on. - - - - -### Register the type - -Put the definition in a file. This one makes receipts a child of `core.file.image`, so each receipt is an image with a merchant and a total: +Keep the definition in a file. Each call on this page sends the file as it stands: ```json title="receipt.json" { @@ -58,7 +21,15 @@ Put the definition in a file. This one makes receipts a child of `core.file.imag } ``` -To start from a scaffold instead of a blank file, `marfa types scaffold` prints a definition from the identifier and the fields you name. It needs no server, working copy or key, and sends nothing: +- **`id`** says who the type is for: `app..` for your app's own types, `user.` for types you keep for yourself, or a name of your own, such as `acme.deal`, for types you publish. Name a type with a singular noun, and write words in `snake_case`. [Type rules](/catalog/your-own-types#identifier) lists what an identifier must meet. +- **`parent`** makes the type a child of another. A receipt is an image with a merchant and a total, so it takes every property of `core.file.image` as well as its own. +- **`fields`** declares the type's own properties. Each needs a `type`, one of the [field kinds](/catalog/field-kinds). `required` defaults to `false`, an enum needs `enum_values` and an array needs `items_type`. [Type rules](/catalog/your-own-types#fields) lists every key a field takes. + +A property can't share a name with a field every item has, such as `source` or `version`: the Catalog lists the [reserved field names](/catalog/your-own-types#reserved-field-names). Keep the time the thing happened in the item's own `occurred_at`. [Register a type](/api-reference/types/registerType) lists the rest a type can declare, such as its label, display hints and policies. + +Marfa keeps a property the type doesn't declare and serves it back. To refuse one, name the type in the `strict_mode` lever, as [Settings](/guides/settings#what-the-enforcement-levers-do) shows. + +To start from a scaffold, `marfa types scaffold` prints a definition. It needs no server or key: ```bash marfa types scaffold app.example.receipt --parent core.file.image \ @@ -66,9 +37,27 @@ marfa types scaffold app.example.receipt --parent core.file.image \ --required merchant > receipt.json ``` -A field is `NAME:TYPE`, an enum is `NAME:enum:one,two` and an array is `NAME:array:TYPE`. The command refuses an identifier or a field that registration would refuse, and says why. It can't set a field's `description` or `maxLength`, so edit the file to add them. +A field is `NAME:TYPE`, an enum `NAME:enum:one,two` and an array `NAME:array:TYPE`. The command refuses what registration would refuse, and says why. It can't set a field's `description` or `maxLength`, so add them to the file. + +### Make it readable as another type + +To make a type readable as another without taking its properties, name that type in `compatible_with`. A connector does this so a person can keep one of its feed items as an item of a core type. A recipe that requires a `body` can be read as a note: + +```json title="recipe.json" +{ + "id": "user.recipe", + "compatible_with": ["core.note"], + "fields": { + "title": { "type": "string" }, + "body": { "type": "string", "required": true }, + "servings": { "type": "integer" } + } +} +``` + +Marfa checks the claim against the [compatibility rules](/catalog/your-own-types#compatible-with) when you register or replace the type, and refuses one that doesn't hold with `422 compatible_with_violation`, naming the property. Later, it refuses the same way a replacement or deletion of the target that would break the claim, and `details.dependent_type_ids` names the types that claim it. A compatible type inherits nothing, so a listing of `core.note` leaves recipes out. -Then register it: +## Register the type ```bash tab="CLI" tab-group="client" marfa types register --file receipt.json @@ -79,32 +68,17 @@ POST /types Authorization: Bearer Content-Type: application/json -{ - "id": "app.example.receipt", - "parent": "core.file.image", - "fields": { - "merchant": { "type": "string", "required": true, "description": "Where the purchase was made" }, - "total": { "type": "number" }, - "currency": { "type": "string", "maxLength": 3 } - } -} + ``` ```ts tab="TypeScript" -const { data, error } = await marfa.POST("/types", { - body: { - id: "app.example.receipt", - parent: "core.file.image", - fields: { - merchant: { type: "string", required: true, description: "Where the purchase was made" }, - total: { type: "number" }, - currency: { type: "string", maxLength: 3 }, - }, - }, -}); +import { readFile } from "node:fs/promises"; + +const definition = JSON.parse(await readFile("receipt.json", "utf8")); +const { data, error } = await marfa.POST("/types", { body: definition }); ``` -Marfa answers `201` with the type as it stored it. It made the label `Receipt` from the identifier, and set the type's `version` to 0. That version is **yours to keep**: Marfa stores the number you send and never changes it. +The TypeScript examples use a client made as [the TypeScript SDK](/typescript-sdk) shows. Marfa answers `201` with the type as it stored it: ```json { @@ -117,43 +91,28 @@ Marfa answers `201` with the type as it stored it. It made the label `Receipt` f "total": { "type": "number" }, "currency": { "type": "string", "maxLength": 3 } }, - "parent": "core.file.image" + "parent": "core.file.image", + "provenance": { + "origin": "user", + "created_by": { "kind": "key", "id": "01a15f3c-13ee-70ba-8ba1-6e00b9a76552", "name": "admin" } + } } } ``` -An identifier someone already registered answers `409 type_already_exists`. +Marfa made the label from the identifier, and `provenance.created_by` names who registered the type. The `version` is **yours to keep**: Marfa stores the number you send, 0 when you send none, and never changes it. An identifier that is already registered is refused `409 type_already_exists`. - - +`marfa types get app.example.receipt` (`GET /types/{id}`) answers the type with the properties it inherits, such as `blob_ref`, `mime_type`, `width` and `height`. An item must hold **every required property along the chain**, so a receipt needs a `blob_ref`, a `mime_type` and a `merchant`. A write that lacks one is refused `400 invalid_properties`, naming it. -### Check what Marfa stored +## Change the type -Read the type back: +A replacement **sends the whole definition**, so change the file and send all of it. To add an optional category, add `"version": 1` and this field to `receipt.json`: -```bash tab="CLI" tab-group="client" -marfa types get app.example.receipt -``` - -```http tab="HTTP" -GET /types/app.example.receipt -Authorization: Bearer -``` - -```ts tab="TypeScript" -const { data } = await marfa.GET("/types/{id}", { - params: { path: { id: "app.example.receipt" } }, -}); +```json +"category": { "type": "enum", "enum_values": ["travel", "meals", "supplies"] } ``` -The answer holds every property a receipt has: its own, and those it inherits from `core.file` and `core.file.image`, such as `blob_ref`, `mime_type`, `width` and `height`. An item of the type must hold **every required property** along the chain, so a receipt needs a `blob_ref`, a `mime_type` and a `merchant`. One without a `merchant` is refused with `400 invalid_properties`, naming it. - - - - -## Compare a definition without registering it - -[Normalize a type](/api-reference/types/normalizeType) validates a candidate and makes its declared defaults explicit, **without changing the registry** or adding an audit entry. It needs no write grant, so any credential can use it to decide whether a connector needs to register or replace its type. For the `receipt.json` definition above: +To check what would change first, normalize the file. Normalizing changes nothing, and any credential can do it, so a connector uses it to decide whether its type needs registering or replacing: ```bash tab="CLI" tab-group="client" marfa types normalize --file receipt.json @@ -164,74 +123,16 @@ POST /types/normalize Authorization: Bearer Content-Type: application/json -{ - "id": "app.example.receipt", - "parent": "core.file.image", - "fields": { - "merchant": { "type": "string", "required": true, "description": "Where the purchase was made" }, - "total": { "type": "number" }, - "currency": { "type": "string", "maxLength": 3 } - } -} + ``` ```ts tab="TypeScript" -const { data, error } = await marfa.POST("/types/normalize", { - body: { - id: "app.example.receipt", - parent: "core.file.image", - fields: { - merchant: { type: "string", required: true, description: "Where the purchase was made" }, - total: { type: "number" }, - currency: { type: "string", maxLength: 3 }, - }, - }, -}); -if (error) throw new Error(error.error.message); +const { data, error } = await marfa.POST("/types/normalize", { body: definition }); ``` -The answer's `type` is your canonical declared definition. `registered_type` is the registered declared definition, and `resolved_type` adds its inherited fields and resolved policies. Both registered forms are `null` if the identifier is not registered. To detect a definition change, compare the declared forms: a resolved form includes ancestor fields your candidate never declared. - -Marfa **refuses unknown schema keys** rather than ignoring them, including inside field definitions and policies. [Type rules](/catalog/your-own-types#fields) lists the keys a field definition takes. Normalization validates the candidate against the same schema and parent-chain rules as registration. - -## Make it readable as another type - -A child takes every property of its parent. When your type should instead be readable as another type without taking its properties, name that type in `compatible_with`. +The answer's `type` is your definition with its defaults made explicit, and `registered_type` is the registered one in the same form. Compare those two. `resolved_type` adds what the registered type inherits, so it never matches. Both registered forms are `null` when the identifier isn't registered. A key that Marfa doesn't know, anywhere in a definition, is refused `400 invalid_schema`. -A connector's type does this so that a person can keep one of its feed items as a library item of a core type. A recipe can be read as a note if it requires a `body`, as a note does: - -```json title="recipe.json" -{ - "id": "user.recipe", - "compatible_with": ["core.note"], - "fields": { - "title": { "type": "string" }, - "body": { "type": "string", "required": true }, - "servings": { "type": "integer" } - } -} -``` - -Marfa checks the claim when you register or replace the type, by the rule in [Type rules](/catalog/your-own-types#parent-and-inheritance). A claim that doesn't hold is refused with `422 compatible_with_violation`, naming the property and what it should be. A compatible type **inherits nothing**, and a listing of `core.note` leaves recipes out. - -Marfa keeps the claim true afterward. It refuses, with the same `422 compatible_with_violation`, a replacement or deletion that would break another type's claim, such as a change to a target's properties that leaves `user.recipe` unreadable as it. `details.dependent_type_ids` lists the claimants, so change or remove their claims first. A platform type such as `core.note` can't be replaced or deleted at all. - -## Change the type - -`PUT /types/{id}` **replaces the whole definition**, so send everything you want to keep. This adds an optional `category` and sets the type's `version` to 1: - -```json title="receipt.json" -{ - "parent": "core.file.image", - "version": 1, - "fields": { - "merchant": { "type": "string", "required": true, "description": "Where the purchase was made" }, - "total": { "type": "number" }, - "currency": { "type": "string", "maxLength": 3 }, - "category": { "type": "enum", "enum_values": ["travel", "meals", "supplies"] } - } -} -``` +Then replace the type: ```bash tab="CLI" tab-group="client" marfa types replace app.example.receipt --file receipt.json @@ -242,41 +143,23 @@ PUT /types/app.example.receipt Authorization: Bearer Content-Type: application/json -{ - "parent": "core.file.image", - "version": 1, - "fields": { - "merchant": { "type": "string", "required": true, "description": "Where the purchase was made" }, - "total": { "type": "number" }, - "currency": { "type": "string", "maxLength": 3 }, - "category": { "type": "enum", "enum_values": ["travel", "meals", "supplies"] } - } -} + ``` ```ts tab="TypeScript" -await marfa.PUT("/types/{id}", { +const { data, error } = await marfa.PUT("/types/{id}", { params: { path: { id: "app.example.receipt" } }, - body: { - parent: "core.file.image", - version: 1, - fields: { - merchant: { type: "string", required: true, description: "Where the purchase was made" }, - total: { type: "number" }, - currency: { type: "string", maxLength: 3 }, - category: { type: "enum", enum_values: ["travel", "meals", "supplies"] }, - }, - }, + body: definition, }); ``` A replacement **changes no item you already hold**. An item meets a property you made required the next time its properties are written. A property you leave out is gone from the type, but items keep their values for it, and a later write isn't refused for them. -Name the **`version`** on every replacement: one that leaves it out sets it back to 0. [Make it readable as another type](#make-it-readable-as-another-type) covers a replacement that would break another type's claim. +Send the `version` every time: a replacement without one sets it back to 0. You can't replace or delete a type Marfa ships. ## Delete the type -`DELETE /types/{id}` removes a type that **no item holds** and no other type names as its parent or in `compatible_with`: +Marfa deletes a type that **no item holds**, in any state and the bin included, and that no other type names as its parent or in `compatible_with`: ```bash tab="CLI" tab-group="client" marfa types delete app.example.receipt @@ -293,10 +176,10 @@ await marfa.DELETE("/types/{id}", { }); ``` -Marfa refuses to delete a type that another type names as its parent, with `409 type_has_subtypes`, and one that items still hold, in any state and the bin included, with `409 type_in_use`. A type that other types name in `compatible_with` is refused as [described above](#make-it-readable-as-another-type). +Otherwise it refuses with `409 type_in_use` when items hold the type, `409 type_has_subtypes` when a type names it as its parent, and `422 compatible_with_violation` when a type names it in `compatible_with`. -`force=true` (the CLI's `--force`) deletes a type that items still hold. It doesn't override a subtype or a `compatible_with` claim. The items stay, but until the type is registered again, Marfa refuses any write to their properties, tier or time, so they can't be edited. You can still tag, archive or move them to the bin. +`force=true` (the CLI's `--force`) deletes a type that items still hold. It doesn't override a subtype or a `compatible_with` claim. The items stay, but until the type is registered again, Marfa refuses any write to their properties, tier or time. You can still tag them, archive them or move them to the bin. ## What your key needs @@ -309,8 +192,8 @@ Each operation needs a key whose type map grants write on the type's identifier, | Replace | `metadata.types:write` when the change only adds optional properties no item holds a value under, or changes the label, a description, the display hints or the version. Any other change, such as removing a property or making one required, needs the `schema.write` permission. | | Delete | `schema.write`. | -A parent or `compatible_with` target that Marfa doesn't ship also needs write on it, unless the type already names it, so a connector can update its type after its permissions narrow. When the type map lacks a write it needs, Marfa answers `403 type_not_permitted`, and `details.grant` names the target. +A new parent or `compatible_with` target that Marfa doesn't ship also needs write on that type; one the type already names doesn't, so a connector can update its type after its permissions narrow. A missing write is refused `403 type_not_permitted`, and `details.grant` names it. -A replacement that needs `schema.write` from a key that lacks it is refused `403 forbidden`, and `details.changes` lists each part that needs it. Split those parts off, or send the replacement with a key that holds `schema.write`. [Manage keys](/guides/manage-keys) shows how to give a key its maps and permissions, and the [Catalog](/catalog/permissions) lists the permissions. +A replacement that needs `schema.write` from a key without it is refused `403 forbidden`, and `details.changes` lists each part that needs it. Send those parts with a key that holds `schema.write`. [Manage keys](/guides/manage-keys) shows how to give a key its maps and permissions. The rules behind this guide are in the [types](https://github.com/withmarfa/marfa/blob/main/conformance/spec/types.md) chapter of the contract. diff --git a/content/docs/guides/deploy.mdx b/content/docs/guides/deploy.mdx index 7e5e1cd4..6396e756 100644 --- a/content/docs/guides/deploy.mdx +++ b/content/docs/guides/deploy.mdx @@ -13,56 +13,16 @@ An instance is one server process and one SQLite database, so there's no databas | A container | A server you already run. | [Run it in a container](#run-it-in-a-container), with one volume. | | Railway | The quickest way to an instance on the internet, billed to you by Railway. | The [Marfa template](https://railway.com/new/template/5z9-Ja) creates the server, a volume and a backup bucket, and sets the secrets and the address, as [the Railway steps](https://github.com/withmarfa/marfa/blob/main/deploy/railway.md) say. | -The server runs [housekeeping jobs](/guides/housekeeping) and backups in the background, so **turn off any host feature that stops an idle service**, such as Railway's Serverless mode. +The server runs [housekeeping jobs](/guides/housekeeping) and backups in the background, so turn off any host feature that stops an idle service, such as Railway's Serverless mode. -## What the server needs - -The server reads its settings from its environment at start. An instance you keep needs these seven: - -```bash title="marfa.env" -NODE_ENV=production -PORT=8600 -SQLITE_PATH=/srv/marfa/data/marfa.db -BLOB_PATH=/srv/marfa/data/blobs -API_KEY_SALT= -MARFA_AUTH_SECRET= -MARFA_AUTH_BASE_URL=https://marfa.example.com -``` - -With `NODE_ENV=production`, the server refuses to start without the two secrets and the address, or with a weak secret. **Outside production it falls back to defaults that suit only a trial.** [Settings](/guides/settings#every-environment-setting) defines every setting. - -### The data directory - -The database and the blobs are the whole of your data, so **put them on storage that survives a restart**: a container volume, or a disk you keep. With the paths above, the directory looks like this once the server has started: - - - - - - - - - - - - - - -[Back up and restore](/guides/back-up-and-restore) says which files a copy needs. - -### The private socket - -A command run on the same machine, as the account that runs the server, reaches the instance through `control/marfa.sock` with the owner's authority and no key. You use it to claim the instance, make the first key and [recover the owner's password](/guides/sign-in-as-the-owner#recover-the-owners-password): - -```bash -marfa --socket /srv/marfa/data/control/marfa.sock setup open -``` +## Run it in a container -The CLI refuses `--socket` while `MARFA_API_URL` or `MARFA_API_KEY` is set. **The socket carries no key, so the server refuses operations on your data**, such as items, types and search, with `401 unauthorized` ([`device/local-socket-needs-key`](https://github.com/withmarfa/marfa/blob/main/conformance/spec/device.md#devicelocal-socket-needs-key)). Run those with a key. +Every release publishes the server image to `ghcr.io/withmarfa/marfa-server` for `linux/amd64` and `linux/arm64`, tagged with its release number and `latest`. The image sets `NODE_ENV=production`, the port and the paths, with the database, the blobs and the private socket under `/data`. -The socket's folder must belong to the server's account and be readable by no one else. On Linux, install the `acl` package, which the server uses to check this. A start replaces a socket that a crash left behind, and refuses one that another process listens on ([`instance/control-live-refused`](https://github.com/withmarfa/marfa/blob/main/conformance/spec/instance.md#instancecontrol-live-refused)): find that process before you remove the socket. + + -### The secrets +### Generate the secrets Generate each secret once: @@ -70,107 +30,144 @@ Generate each secret once: openssl rand -hex 32 ``` -**Keep a copy of both outside the data directory**, because neither is stored in it. +Keep a copy of both outside the data directory, because neither is stored in it: | Secret | What it does | If it changes | | --- | --- | --- | -| `API_KEY_SALT` | Hashes keys and app tokens. The database holds only the hashes. | The server refuses every existing key and token with `401 unauthorized` until it starts again with the original ([`instance/salt-refuses-credentials`](https://github.com/withmarfa/marfa/blob/main/conformance/spec/instance.md#instancesalt-refuses-credentials)). | +| `API_KEY_SALT` | Hashes keys and app tokens. The database holds only the hashes. | The server refuses every existing key and token with `401 unauthorized` until it starts again with the original. | | `MARFA_AUTH_SECRET` | Signs sign-in cookies and the links the server makes. | Every browser is signed out. | -### The address - -`MARFA_AUTH_BASE_URL` is the URL your devices reach the instance at, and the server uses it as the OAuth issuer and in every link it makes. **The owner can claim the instance and sign in only over `https`**, or over `http` on `localhost`, `127.0.0.1` or `[::1]`. - -Devices reach it on your local network, from anywhere over Tailscale, which these docs suggest, or through a Cloudflare Tunnel at an address on your own domain. - -**Behind a proxy or tunnel, tell the server which caller address to believe.** Otherwise every request seems to come from the proxy, and the limit of ten password sign-in attempts in fifteen minutes from one address ([`keys-and-oauth/sign-in-address-limit`](https://github.com/withmarfa/marfa/blob/main/conformance/spec/keys-and-oauth.md#keys-and-oauthsign-in-address-limit)) lets anyone's wrong guesses lock the owner out. + + -- **`TRUSTED_PROXY_CIDRS`** names the proxy's address ranges. The server believes `X-Forwarded-For` from any address in them, so **they must hold only your proxy**. -- **`TRUSTED_PROXY_HEADER`** names a header the platform's edge overwrites with the caller's address, for a host whose edge addresses you can't list. Set it only when nothing reaches the server except through that edge, or any client can choose its own address. +### Write the settings -A proxy on the same machine as a container, such as Tailscale Serve or a tunnel, reaches it from the gateway of Docker's network. Set `TRUSTED_PROXY_CIDRS` to that network's subnet, and run nothing on the network that you don't control, because every container there is trusted too. This shows the default network's subnet: +Put the secrets and the instance's [address](#the-address) in `marfa.env`, with the `S3_` settings if you use an [object store](#an-object-store). Keep the file out of any repository. -```bash -docker network inspect bridge --format '{{json .IPAM.Config}}' +```bash title="marfa.env" +API_KEY_SALT= +MARFA_AUTH_SECRET= +MARFA_AUTH_BASE_URL=https://marfa.example.com ``` -### An object store - -Set `S3_BUCKET` and the other `S3_` settings, and the server keeps a second copy of every blob in an S3-compatible bucket. The container also [streams the database there](/guides/back-up-and-restore#with-a-bucket). - -**Give each instance that shares a bucket its own `S3_PREFIX`**, such as `trial/blobs`. An instance keeps its blobs and its database replica under its prefix ([`instance/replica-other-prefix`](https://github.com/withmarfa/marfa/blob/main/conformance/spec/instance.md#instancereplica-other-prefix)). Two instances with one prefix write one replica, and a new volume of either starts on whichever database was copied last. - -A prefix keeps replicas apart, not access: each instance holds credentials for the whole bucket, so **share a bucket only between instances of one owner**. +In production, the server refuses to start without these three, or with a weak secret. [Settings](/guides/settings#every-environment-setting) defines every setting. -**Put the bucket on a different site from the instance.** A link the bucket signs serves a blob from the bucket's own host, without the protections the instance puts on its own downloads ([`blobs/download-sandbox`](https://github.com/withmarfa/marfa/blob/main/conformance/spec/blobs.md#blobsdownload-sandbox)). A bucket under the instance's domain could let an uploaded HTML file run as a page of the instance. - -## Run it in a container - -Every release publishes the server image to `ghcr.io/withmarfa/marfa-server` for `linux/amd64` and `linux/arm64`, tagged with its release number and `latest`. **Pull a release by its tag, not `latest`**, so a restart never changes the build and an [upgrade](/guides/upgrade) can name both builds. - - + -### Pull the image and start it +### Start the container -Put `API_KEY_SALT`, `MARFA_AUTH_SECRET`, `MARFA_AUTH_BASE_URL` and the `S3_` settings in `marfa.env`, and keep it out of any repository. The image sets `NODE_ENV`, `PORT` and the paths, with the database, the blobs and the socket under `/data`. Start the container with one volume there: +Pull a release by its tag, not `latest`, so a restart never changes the build and an [upgrade](/guides/upgrade) can name both builds. Start it with one volume at `/data`: src/generated/server-image.md -**`-p 127.0.0.1:8600:8600` publishes the port on the loopback address only**, for a proxy or tunnel on the same machine. A bare `-p 8600:8600` opens it on every network interface, on Linux past the host's firewall. - -Without `S3_BUCKET`, **no copy leaves the machine**, so [back up](/guides/back-up-and-restore) the volume yourself. +`-p 127.0.0.1:8600:8600` publishes the port on the loopback address only, for a proxy or tunnel on the same machine. A bare `-p 8600:8600` opens it on every network interface, on Linux past the host's firewall. -To run code that no release holds, [build the image yourself](https://github.com/withmarfa/marfa/blob/main/deploy/README.md#building-the-image-yourself). +Without `S3_BUCKET`, no copy leaves the machine, so [back up](/guides/back-up-and-restore) the volume yourself. To run code that no release holds, [build the image yourself](https://github.com/withmarfa/marfa/blob/main/deploy/README.md#building-the-image-yourself). ### Claim the instance -Ask the CLI in the image for a setup link, then open it in a browser to choose the owner's email and password: +Ask the CLI in the image for a setup link: ```bash docker exec marfa marfa --socket /data/control/marfa.sock setup open --no-browser ``` -The CLI prints the link as `url`. It works once, for five minutes. +The CLI prints the link as `url`. It works once, for five minutes. Open it in a browser and choose the owner's email and password. ## Run it without a container -The server needs Node.js 22.14 or later. Build it in a clone of the repository: +The container is the tested route. Without it, the server needs Node.js 22.14 or later. Build it in a clone of the repository: ```bash pnpm install --frozen-lockfile pnpm --filter "@withmarfa/server..." build ``` -Start it from `packages/server`, with the settings in its environment, under a supervisor such as launchd or systemd: +Give the server the three settings from the container steps, plus `NODE_ENV`, `PORT` and the two paths. Outside `production`, it falls back to defaults that suit only a trial. + +```bash title="marfa.env" +NODE_ENV=production +PORT=8600 +SQLITE_PATH=/srv/marfa/data/marfa.db +BLOB_PATH=/srv/marfa/data/blobs +API_KEY_SALT= +MARFA_AUTH_SECRET= +MARFA_AUTH_BASE_URL=https://marfa.example.com +``` + +Start it from `packages/server` under a supervisor, such as launchd or systemd. The `--import` loads [telemetry](/guides/telemetry). ```bash cd packages/server node --import ./dist/instrumentation.js dist/index.js ``` -The `--import` loads [telemetry](/guides/telemetry). Claim the instance with `marfa --socket setup open`, as the account that runs the server. **The container is the tested route.** +Then claim the instance as the account that runs the server, with `marfa --socket /srv/marfa/data/control/marfa.sock setup open`. -## Keep it running +## The address + +`MARFA_AUTH_BASE_URL` is the URL your devices reach the instance at. The server uses it as the OAuth issuer and in its links. **The owner can claim the instance and sign in only over `https`**, or over `http` on `localhost`, `127.0.0.1` or `[::1]`. + +Devices reach the instance on your local network, from anywhere over Tailscale, which these docs suggest, or through a Cloudflare Tunnel at an address on your own domain. + +**Behind a proxy or tunnel, tell the server which caller address to believe.** Otherwise every request seems to come from the proxy, and the limit of ten password attempts from one address in fifteen minutes lets anyone's wrong guesses lock the owner out. Set one of these: + +- **`TRUSTED_PROXY_CIDRS`** names the proxy's address ranges. The server believes `X-Forwarded-For` from any address in them, so they must hold only your proxy. A proxy beside a container, such as Tailscale Serve, reaches it from Docker's network, whose subnet `docker network inspect bridge --format '{{json .IPAM.Config}}'` shows. Every container there is then trusted, so run nothing there you don't control. +- **`TRUSTED_PROXY_HEADER`** names a header the platform's edge overwrites with the caller's address, for a host whose edge addresses you can't list. Set it only when nothing reaches the server except through that edge, or any client can choose its address. -**Point your supervisor or monitor at `GET /health`**, which needs no credential. It answers `503` when the database, a write, the disk or blob storage is down, as when the blob drive is unmounted, and `200` otherwise. The container checks it every 30 seconds. +## An object store + +Set `S3_BUCKET` and the other `S3_` settings, and the server keeps a second copy of every blob in an S3-compatible bucket. The container also [streams the database there](/guides/back-up-and-restore#with-a-bucket). + +- **Give each instance that shares a bucket its own `S3_PREFIX`**, such as `trial/blobs`. An instance keeps its blobs and its database replica under its prefix. Two instances with one prefix write one replica, and a new volume of either starts on whichever database was replicated last. +- **Share a bucket only between instances of one owner.** A prefix keeps replicas apart, not access: each instance holds credentials for the whole bucket. +- **Put the bucket on a different site from the instance.** A link the bucket signs serves a blob without the instance's download protections, so a bucket under the instance's domain could let an uploaded HTML file run as one of its pages. + +## The data directory + +The database and the blobs are the whole of your data, so put them on storage that survives a restart, such as a container volume. With the paths in `marfa.env` above, the started server's directory holds these files: + + + + + + + + + + + + + + +[Back up and restore](/guides/back-up-and-restore) says which files a copy needs. + +`control/marfa.sock` is the private socket. A command run on the same machine, as the server's account, reaches the instance through it with the owner's authority. You use it to claim the instance, make the first key and [recover the owner's password](/guides/sign-in-as-the-owner#recover-the-owners-password). + +- The socket carries no key, so the server refuses item properties, edges, tags, search, counts, events and exports through it with `401 unauthorized`. Run those with a key. +- The CLI refuses `--socket` while `MARFA_API_URL` or `MARFA_API_KEY` is set. +- The socket's folder must belong to the server's account and be readable by no one else. On Linux, install the `acl` package, which the server uses to check this. +- A start replaces a socket that a crash left behind, and refuses one that another process listens on. Find that process before you remove the socket. + +## Keep it running -`SIGTERM` or `SIGINT` stops the server within eight seconds, so give it ten before a runtime kills the process. +Point your supervisor or monitor at `GET /health`, which needs no credential. It answers `503` when the database, a write, the disk or blob storage is down, such as an unmounted blob drive, and `200` otherwise. The container checks it every 30 seconds. -When the server stops, its exit status tells your supervisor whether to start it again: +`SIGTERM` or `SIGINT` stops the server within about eight seconds, so give it ten before a runtime kills the process. When the server stops, its exit status tells your supervisor whether to start it again: | Status | Meaning | Your supervisor | | --- | --- | --- | | `0` | A clean stop. | Starts it again if you want it running. | | `1` | A failed start or stop, such as a port in use or a setting outside its rules. | Starts it again. | -| `78` | The database was written by another build. The server changed nothing. | **Doesn't start it again**: see [Upgrade](/guides/upgrade). | +| `78` | The database was written by another build. The server changed nothing. | Doesn't start it again. See [Upgrade](/guides/upgrade). | The container doesn't exit on `78`. It stays up and unhealthy with the reason in its log, so a restart policy can't loop on it. -The rules are in the contract's [instance chapter](https://github.com/withmarfa/marfa/blob/main/conformance/spec/instance.md), and the container recipe in the [deployment guide](https://github.com/withmarfa/marfa/blob/main/deploy/README.md). +The rules behind this page are in the [instance](https://github.com/withmarfa/marfa/blob/main/conformance/spec/instance.md), [device](https://github.com/withmarfa/marfa/blob/main/conformance/spec/device.md), [keys and OAuth](https://github.com/withmarfa/marfa/blob/main/conformance/spec/keys-and-oauth.md) and [blobs](https://github.com/withmarfa/marfa/blob/main/conformance/spec/blobs.md) chapters of the contract, and the container recipe is in the [deployment guide](https://github.com/withmarfa/marfa/blob/main/deploy/README.md). diff --git a/content/docs/guides/export-and-restore.mdx b/content/docs/guides/export-and-restore.mdx index 4863ca2f..81c68114 100644 --- a/content/docs/guides/export-and-restore.mdx +++ b/content/docs/guides/export-and-restore.mdx @@ -3,18 +3,20 @@ title: Export and restore an archive description: Export your items, edges, types and files to an archive, and restore it into the same instance or another, such as a new build. --- -An archive carries your content to another instance and leaves the instance's own setup behind: **no keys, webhooks, configuration or audit log**. It's how you move data to a new build when you [upgrade](/guides/upgrade). It **isn't a long-term backup**, because only the build that wrote it is sure to read it ([`search-and-filters/restore-version`](https://github.com/withmarfa/marfa/blob/main/conformance/spec/search-and-filters.md#search-and-filtersrestore-version)). To recover this instance from a failed disk, [back up the data directory](/guides/back-up-and-restore) instead. +An archive carries your content to another instance: items, edges, the types and edge types you registered, and file bytes. It leaves the instance's own setup behind, so it holds no keys, webhooks, configuration or audit log. It's how you move data to a new build when you [upgrade](/guides/upgrade). + +An archive isn't a long-term backup, because only the build that wrote it is sure to read it. To recover an instance from a failed disk, [back up the data directory](/guides/back-up-and-restore) instead. ## Export an archive -An archive holds what its caller can read. The owner **reads everything**, and exports from a browser without a key. The Export and restore page, at `/auth/owner/export`, downloads an archive of everything, or of one type and the types under it. The trash is never in it. To export from a script, use a key that reads every type, edge type and extension and holds `blobs.manage`. **Without `blobs.manage`, the archive leaves out the contents of files that only an earlier version names**, and its manifest doesn't list them as missing. +An archive holds what its caller can read. The owner can export everything from a browser, on the **Export and restore** page at `/auth/owner/export`: choose everything or one type in **What to export**, then select **Export**. That page always leaves out the trash. To export from a script, follow these steps. ### Make an export key -On the instance's machine, under the account that runs the server: +On the instance's machine, under the account that runs the server, make a key that reads every type, edge type and extension: ```bash marfa --socket /srv/marfa/data/control/marfa.sock keys create --label backup --source backup \ @@ -22,13 +24,15 @@ marfa --socket /srv/marfa/data/control/marfa.sock keys create --label backup --s --permission blobs.manage ``` -The CLI shows the key once. Set it as `MARFA_API_KEY` wherever you export from. +The CLI shows the key once. Set it as `MARFA_API_KEY` wherever you export from. Without `blobs.manage`, the archive leaves out the bytes of files that only an earlier version of an item names, and its manifest doesn't list them as missing. ### Export +`state=any` includes the trash, which an export leaves out otherwise: + ```bash tab="CLI" tab-group="client" marfa export --format archive --state any --output marfa-archive.tar.gz ``` @@ -38,13 +42,17 @@ GET /export?format=archive&state=any Authorization: Bearer ``` -The answer is a `.tar.gz` file of the items the key can read, with their tags, extensions and earlier versions, the edges between them, the blobs they name, and every type and edge type registered on the instance. **`state=any` includes the trash**, which an export leaves out otherwise. +The CLI writes to a hidden file beside the path and renames it over the path only when the export ended whole, so the file at the path is always the last whole archive the command wrote. Any other ending leaves the path as it was and exits with a nonzero [exit code](/cli#exit-codes). ### Check the file +An archive holds four files, plus one file for each blob it carries: + + + ```bash tar -tzf marfa-archive.tar.gz ``` @@ -54,85 +62,89 @@ manifest.json items.ndjson edges.ndjson types.ndjson -blobs/sha256:a0feecd7233d9f5f3ec784c6bdb36587c651e2ec60437c27b4bf135de39149d7 +blobs/sha256:f1062b81e1771fa1b881c035d40485418ecb4c74fa1b427c1155506a26382c46 ``` -An export that was cut short doesn't unpack. With `--output`, the CLI writes to a hidden file beside the path and renames it over the path only when the export ended whole and lacks no files, so a file at the path is **always the last whole archive** the command wrote there. Any other ending leaves the path as it was and exits with a nonzero [exit code](/cli#exit-codes), so a script notices. + + +An export that was cut short doesn't unpack. -### When an archive lacks a file's bytes +## Leave out the files -An export never claims a blob it doesn't hold. When no store holds a whole copy of a blob, such as after a disk lost the file, the archive **leaves the blob out** and the manifest names it under `missing_blobs`, with its `mime_type` and `size_bytes`. One lost file costs the backup that file, not everything else. +An archive without file bytes is much smaller. Add `files=false` to the request, or turn off **Include files** on the **Export and restore** page. The CLI has no option for it. -With `--output`, the CLI keeps such an archive as `.incomplete` and exits with code 1 and `blobs_missing`. Over HTTP the answer is still `200`. In both, read the manifest: +```http +GET /export?format=archive&files=false +Authorization: Bearer +``` + +The manifest then names every file under `missing_blobs`, and a restore reports in `blobs_missing` each one the target doesn't hold. + +## When an archive lacks a file's bytes + +When no store holds a whole copy of a blob, as after a disk lost the file, the export leaves that blob out and names it under `missing_blobs` in the manifest. One lost file costs the archive that file, not everything else. Over HTTP the answer is still `200`. The CLI keeps the archive as `.incomplete`, leaves the file at the path as it was, and exits with code 1 and `blobs_missing`. + +The manifest's blob fields show what the archive lacks: + + ```bash -tar -xzOf marfa-archive.tar.gz.incomplete manifest.json +tar -xzOf marfa-archive.tar.gz.incomplete manifest.json | jq '{blob_count, blobs, missing_blobs}' ``` -The manifest's other fields are left out here: - ```json { "blob_count": 0, "blobs": {}, "missing_blobs": { - "sha256:a0feecd7233d9f5f3ec784c6bdb36587c651e2ec60437c27b4bf135de39149d7": { - "mime_type": "image/jpeg", - "size_bytes": 2048 + "sha256:f1062b81e1771fa1b881c035d40485418ecb4c74fa1b427c1155506a26382c46": { + "mime_type": "image/png", + "size_bytes": 73 } } } ``` -An empty `missing_blobs` means the archive holds every blob its items name. A file item can also name bytes the instance has no record of, as after a restore that lacked them, and the manifest gives `null` for both fields. Every later backup names a missing blob **until you upload its bytes again**: + + +`mime_type` and `size_bytes` are `null` when the instance has no record of the bytes, as after a restore that lacked them. An empty `missing_blobs` means the archive holds every blob its items name. Every later export names the blob until you upload its bytes again: ```bash marfa blobs upload ``` -A store that fails, or doesn't answer within 15 seconds, is not a lost file. Unless another store holds the blob whole, the export is refused with `503 blob_store_unavailable` before it begins, so try again once the store answers. - -### Leave out the files - -An archive without file bytes is **much smaller**. Add `files=false` to the export, or turn off Include files on the Export and restore page. The CLI has no option for it. - -```http -GET /export?format=archive&files=false -Authorization: Bearer -``` - -The manifest then names every file under `missing_blobs`, and a restore reports in `blobs_missing` each one the target doesn't hold ([`search-and-filters/export-without-files`](https://github.com/withmarfa/marfa/blob/main/conformance/spec/search-and-filters.md#search-and-filtersexport-without-files)). +A store that fails is not a lost file. Unless another store holds the blob whole, the export is refused with `503 blob_store_unavailable` before it begins, so try again once the store answers. -### What an export checks +## What an export checks - **It isn't taken at one instant.** An item created while it runs may not be in it, so stop your apps first if the archive must hold every write. -- **It ends when its credential does.** If the key or app that started it is revoked, expires, is disconnected or is narrowed below what is left to send, the response ends with an error and the file doesn't unpack. Export again. -- **It is checked against a restore.** Before it sends a byte, Marfa checks that the archive restores into an empty instance of the same build without skipping a row, and reads the selection once more if a row fails. A row that fails both times unchanged refuses the export with `409 export_rows_unrestorable`, named in `details.row`: correct the row or its type, then export again. If writes changed it between the reads, the answer is `503 export_unrestorable`: export again once the writes stop. -- **It needs room.** The server prepares the archive in the blob folder first, so leave that disk room for about as much as the archive's text takes uncompressed. An export that would take it below `MARFA_DISK_RESERVE_BYTES` is refused `507 insufficient_storage`. -- **A source filter hides items.** [A source filter](/guides/settings#what-the-enforcement-levers-do) leaves items out of an export, the owner's too. Check that neither the instance nor the export key carries one that hides what you want kept. +- **It ends when its credential does.** If the key or app that started it is revoked, expires, is disconnected or loses access to what is left to send, the response ends with an error and the file doesn't unpack. Export again. +- **It's checked against a restore.** Before it sends a byte, Marfa checks that the archive restores into an empty instance of the same build, and reads the selection again if a row fails. A row that fails twice refuses the export with `409 export_rows_unrestorable` and is named in `details.row`: correct the row or its type, then export again. If writes changed the row between the reads, the answer is `503 export_unrestorable`: export again once the writes stop. +- **It needs room.** The server builds the archive in the blob folder first, so leave that disk room for about the archive's uncompressed text. An export that would take the disk below `MARFA_DISK_RESERVE_BYTES` is refused `507 insufficient_storage`. +- **A source filter hides items.** A [source filter](/guides/settings#what-the-enforcement-levers-do) on the instance or the key leaves items out of an export, the owner's too. [Export items and edges](/api-reference/export-and-restore/exportData) lists every answer. ## Restore an archive -A restore **adds what the instance lacks and replaces nothing**. An item the instance already holds is counted in `duplicates` and left as it is, so replaying an archive never overwrites work done since. The target can be a new instance or the one the archive came from. +A restore adds what the instance lacks and replaces nothing. An item the instance already holds is counted in `duplicates` and left as it is, so replaying an archive never overwrites work done since. The target can be a new instance or the one the archive came from. + +Only the owner can restore, through the private socket or signed in to the instance within the last five minutes. A key is refused `403 forbidden`, whatever it holds. In a browser, on the **Export and restore** page, select **Choose file...**, choose the archive, then select **Restore**. An older sign-in is asked to confirm in a **Confirm it's you** dialog first. + +The answer counts what the restore did. Through the socket, into a fresh instance: -**Only the owner can restore**, through the private socket or signed in to the instance within the five minutes before the restore starts. A key is refused `403 forbidden`, whatever it holds. + ```bash marfa --socket /srv/marfa/data/control/marfa.sock restore marfa-archive.tar.gz ``` -In a browser, choose the archive on the Export and restore page and select Restore. A sign-in older than five minutes is asked to confirm it's the owner before the upload starts. - -The answer counts what the restore did. Into a fresh instance: - ```json { - "imported": 5, + "imported": 6, "duplicates": 0, "edges_imported": 1, "edges_skipped": 0, @@ -146,19 +158,21 @@ The answer counts what the restore did. Into a fresh instance: } ``` -Restoring the same archive again answers `"imported": 0` and `"duplicates": 5`. Check two fields: + + +The same archive restored again answers `"imported": 0`, `"duplicates": 6` and `"edges_skipped_reasons": { "already_present": 1 }`. Check two fields: -- **`blobs_missing`** lists the blobs the archive names but carries no bytes for, and no store of the instance holds. The restore still writes the items and **writes no record for those blobs**, so a file that names one has no bytes until you [upload them again](#when-an-archive-lacks-a-files-bytes). The CLI then exits with code 1 and `blobs_missing`. -- **`edges_skipped_reasons`** counts why each edge was skipped, such as `already_present` or `endpoint_missing` ([`search-and-filters/restore-edge-skips`](https://github.com/withmarfa/marfa/blob/main/conformance/spec/search-and-filters.md#search-and-filtersrestore-edge-skips)). An edge to an item in the trash is restored. +- **`blobs_missing`** lists the blobs the archive names but carries no bytes for, and no store of the instance holds. The restore still writes the items but no record for those blobs, so their files have no bytes until you [upload them again](#when-an-archive-lacks-a-files-bytes). The CLI then exits with code 1 and `blobs_missing`. +- **`edges_skipped_reasons`** counts why edges were skipped, such as `already_present` or `endpoint_missing`. An edge to an item in the trash is restored. -A restore writes in one transaction. If it's refused, fails or is interrupted, it leaves nothing behind, so you can run it again. While it runs, other writes wait, so let it answer before you start your apps. +A restore writes in one transaction. If it's refused, fails or is interrupted, it leaves nothing behind, so you can run it again. Other writes wait while it runs, so let it answer before you start your apps. -### What a restore leaves for you +A restore refuses the whole archive when the file isn't a valid archive, lacks one of its four files, names another format version, or conflicts with what the instance registers. [Restore from an archive](/api-reference/export-and-restore/restoreArchive) lists each refusal. If a newer build refuses your archive, [keep running the old build](/guides/upgrade#if-the-new-build-refuses-the-archive). -- **Keys, webhooks and configuration.** Make keys, register webhooks and set the configuration again on the new instance. -- **Connected apps.** A restored `system.connection` item for an app has `status: revoked`. It records past access rather than reconnecting the app, so sign the app in again. An item an app wrote gets `restored:` in front of its source, so a later sign-in can't overwrite it. -- **Rows as the archive holds them.** A restore takes an item whose type is no longer registered, or whose type has since changed its fields, exactly as archived, so an archive of your instance restores into an empty instance of the same build. An item of a `core` or `system` type that the build ships is still held to its shape. +## What a restore leaves for you -A restore refuses the whole archive, and writes nothing, when the file isn't a valid archive, lacks one of its four files, names another format version, or conflicts with what the instance registers. [Restore from an archive](/api-reference/export-and-restore/restoreArchive) lists each refusal. If a newer build refuses your archive, [keep running the old build](/guides/upgrade#if-the-new-build-refuses-the-archive). +- **Keys, webhooks and configuration.** Make keys, register webhooks and set the configuration again. +- **Connected apps.** A restored `system.connection` item has `status: revoked`: it records past access rather than reconnecting the app, so sign the app in again. An item an app wrote gets `restored:` in front of its source, so a later sign-in can't overwrite it. +- **Rows as archived.** A restore takes an item exactly as archived, even when its type is no longer registered or has changed its fields. An item of a `core` or `system` type that the build ships is still held to its shape. The rules behind this page are in the [export](https://github.com/withmarfa/marfa/blob/main/conformance/spec/search-and-filters.md#export) and [restore](https://github.com/withmarfa/marfa/blob/main/conformance/spec/search-and-filters.md#restore) sections of the search and filters chapter of the contract. diff --git a/content/docs/guides/keep-in-sync.mdx b/content/docs/guides/keep-in-sync.mdx index be32d937..82bce432 100644 --- a/content/docs/guides/keep-in-sync.mdx +++ b/content/docs/guides/keep-in-sync.mdx @@ -3,82 +3,62 @@ title: Keep a copy in sync description: Follow an instance's event stream from a cursor, so a copy of your data stays current and picks up where it stopped after a break. --- -Some programs keep their own copy of an instance's data: a search index, a cache, a report that should never go stale. You read what you need once, then apply each change from the [event stream](/events) as it happens, and save a cursor so that after a break you receive everything you missed and nothing twice. An app built on the [Swift SDK](/swift-sdk), and the [CLI](/cli), do this for you; follow this guide when you build your own. +A search index, a cache or a report keeps its own copy of an instance's data. Read what you need once, then apply each change from the [event stream](/events) and save a cursor, so that after a break you receive everything you missed and nothing twice. The Swift SDK and the CLI do this for you. -You need a credential that can read the types you copy. To watch the stream by hand, run `marfa events`, which prints one frame a line, or `marfa events --from ` to start after a cursor. +You need a credential that can read the types you copy. A copy that answers reads offline in the instance's place, as a device does, needs more: the [read views chapter of the contract](https://github.com/withmarfa/marfa/blob/main/conformance/spec/read-views.md) says what. -## Start from the head of the log +## What the stream sends -The first time you run, you have no cursor. Take the **head of the log** before you read anything, so that every change made while you read is still waiting in the log after it. +`marfa events` prints one frame a line. From cursor `4`, narrowed to notes: - - - -### Take the head of the log + -Open the [stream](/api-reference/event-stream/streamEvents) with no cursor: - -```http -GET /events -Authorization: Bearer +```bash +marfa events --type core.note --edges none --from 4 ``` -Its first frame, `stream_cursor`, names the latest event in the log, or `0` if the log is empty. Keep its **`cursor`** and close the stream. - ```text -event: stream_cursor -data: {"event_type":"stream_cursor","cursor":"78"} +- stream_cursor 6 +6 item.created 01a15f3c-db7b-745c-a178-be82b8caa2a0 +- stream_live 6 ``` - - - -### Read the state you need - -List the items you copy, page by page, until `next_cursor` is `null`. [List, filter and search](/guides/list-filter-search) shows how. - - - - -### Follow the stream from the head - -Open the stream again with the head as your cursor, as the next section shows. - - - - -A change you read in the second step can arrive again as an event in the third. Apply each event by replacing what you hold for the item or edge with the one in the frame, and **an event you already reflect changes nothing**. + -## Follow the stream +The stream opens with `stream_cursor`, which names the **head of the log**, the latest event. It replays every event after your cursor, sends `stream_live` when it has caught up, then sends each new event as it happens. Event 5 changed a task, so the filter left it out and the IDs skip it. -Send your cursor in the `Last-Event-ID` header. Marfa opens with `stream_cursor`, replays every event after your cursor, sends `stream_live` when it has caught up, and then sends each new event as it happens. Here the log held one event after 78: - -```http -GET /events -Authorization: Bearer -Last-Event-ID: 78 -``` +Over HTTP, the [stream](/api-reference/event-stream/streamEvents) takes your cursor in the `Last-Event-ID` header and answers server-sent events. From cursor `5`, with the item cut short: ```text : connected event: stream_cursor -data: {"event_type":"stream_cursor","cursor":"79"} +data: {"event_type":"stream_cursor","cursor":"6"} -id: 79 +id: 6 event: item.created -data: {"event_type":"item.created","item":{"id":"01a12323-60d1-722b-a9e2-d590f2a35d87","type":"core.note","state":"active","tier":"library","properties":{"body":"Finish the chapter on tides.","title":"Reading list"},"created_at":"2026-10-10T00:08:03.537Z","updated_at":"2026-10-10T00:08:03.537Z","occurred_at":"2026-10-10T00:08:03.537Z","version":1,"schema_version":0,"source":"laptop"},"metadata":{"item_id":"01a12323-60d1-722b-a9e2-d590f2a35d87","tags":[],"extensions":{}}} +data: {"event_type":"item.created","item":{"id":"01a15f3c-db7b-745c-a178-be82b8caa2a0","type":"core.note",...},"metadata":{...}} event: stream_live -data: {"event_type":"stream_live","cursor":"79"} +data: {"event_type":"stream_live","cursor":"6"} ``` -Only events carry an `id:` line. The frames that mark the stream's progress, and the ones that end it, carry none, so the last `id:` you received is **always a cursor you can resume from**. Lines that start with `:` are comments: Marfa sends `:ping` every 30 seconds while the stream is quiet. +Only events carry an `id:` line, so **the last `id:` you received is always a cursor you can resume from**. Lines that start with `:` are comments: Marfa sends `:ping` every 30 seconds while the stream is quiet. The Catalog's [Events](/catalog/events) page lists every event and its fields. + +## Build the sync + +`sync.ts` keeps a copy of every note in a JSON file. On its first run it takes the head of the log before it reads the notes, so every change made while it reads waits in the log, then follows the stream from that head. -A browser `EventSource` can't send an `Authorization` header, and the TypeScript SDK doesn't read streams, so **read the stream with `fetch`**. The code in this guide is one file, `sync.ts`, and it starts by reading frames: + + + +### Read the frames + +A browser `EventSource` can't send an `Authorization` header, and the TypeScript SDK doesn't read streams, so read the stream with `fetch`: ```ts title="sync.ts" -import { createClient, pages } from "@withmarfa/client"; +import { readFile, writeFile } from "node:fs/promises"; +import { createClient, pages, type components } from "@withmarfa/client"; const baseUrl = process.env.MARFA_API_URL!; const credential = process.env.MARFA_API_KEY!; @@ -109,7 +89,7 @@ async function* readFrames(body: ReadableStream>): Async } async function open(cursor?: string): Promise>> { - const response = await fetch(`${baseUrl}/events?type=core.note`, { + const response = await fetch(`${baseUrl}/events?type=core.note&edges=none`, { headers: { Authorization: `Bearer ${credential}`, ...(cursor === undefined ? {} : { "Last-Event-ID": cursor }), @@ -122,7 +102,36 @@ async function open(cursor?: string): Promise + + +### Keep the copy and its cursor together + +Save the cursor **in the same place as the copy, in step with what you apply**: here, one file written after each event. An item event carries the item as it stands after the change, so `apply` replaces what the copy holds, or removes it for `item.purged`. An event the copy already reflects changes nothing. + +```ts title="sync.ts" +type Item = components["schemas"]["Item"]; +type Copy = { cursor?: string; notes: Record }; +const file = "notes-copy.json"; +let copy: Copy = await readFile(file, "utf8").then(JSON.parse, () => ({ notes: {} })); + +function apply(event: { event_type: string; item: Item }) { + if (event.event_type === "item.purged") delete copy.notes[event.item.id]; + else copy.notes[event.item.id] = event.item; +} + +async function saveCursor(cursor: string) { + copy.cursor = cursor; + await writeFile(file, JSON.stringify(copy)); +} +``` + + + + +### Follow the stream + +`follow` applies each event and saves its ID. It returns the frame that ended the stream, or `closed` if the connection dropped. ```ts title="sync.ts" async function follow(cursor: string): Promise { @@ -139,45 +148,32 @@ async function follow(cursor: string): Promise { case "cursor_ahead": return frame.event; default: - await apply(data); - if (frame.id !== undefined) await saveCursor(frame.id); + apply(data); + await saveCursor(frame.id!); } } return "closed"; } ``` -Save the `cursor` of **`stream_live`** too. When you narrow the stream, the last event you received can be far behind the head, and resuming from `stream_live` skips what the replay already covered. - -## Resume after a break - -Keep the cursor in step with what you apply, in the same place as your copy, and send it as `Last-Event-ID` when you open the stream again. Marfa sends each event after it **exactly once**, also across a restart of the server. Send the cursor as the stream wrote it; a malformed one is refused with `400 validation_error`. - -## When the stream ends - -A stream doesn't end of its own accord. What you see when it does tells you what to do: +Save the `cursor` of `stream_live` too: on a narrow stream, the last event can be far behind the head. Marfa sends each event after your cursor exactly once, also across a restart of the server. -| You see | It means | What to do | -| --- | --- | --- | -| The connection closes with no frame | A network or proxy fault. | After a short wait, open the stream again from your saved cursor. | -| `stream_incomplete` | The stream couldn't go on, and its `reason` says why, such as `server_stopping` or `reader_behind`. Everything after your cursor is still in the log. | Open the stream again from your saved cursor. For `credential_ended`, the key was revoked or has expired: use one that's valid. For `reader_behind`, take frames off the connection as fast as they arrive and apply them from a queue. | -| `catchup_too_old` | The log no longer holds the events after your cursor. By default it keeps them for at least seven days. | Read the state again and start from the head of the log. | -| `cursor_ahead` | Your cursor is past the log's head, as after the instance was restored from a backup. | Discard the cursor, read the state again and start from the head of the log. | - -`marfa events` prints the frame it read, then exits `1` for `cursor_ahead` and `catchup_too_old`, `3` for a `stream_incomplete` that a new stream can mend, and `5` for `credential_ended`. A script need not read the frames. [Exit codes](/cli#exit-codes) lists every status. + + -`rebuild` does the three steps at the top of this guide, with the [TypeScript SDK](/typescript-sdk) for the listing, and returns the head it started from. The listing and the stream both ask for `core.note` in every state, so the copy holds exactly what the stream keeps current. +### Start from the head of the log -The loop after it follows the stream for as long as the program runs. `clear` and `loadCursor` are yours too: +`rebuild` takes the head from the first frame, lists every note in every state with the [TypeScript SDK](/typescript-sdk), and saves the head as the cursor. The listing and the stream both ask for notes and no edges, so the copy holds exactly what the stream keeps current. ```ts title="sync.ts" async function rebuild(): Promise { - let head = "0"; + let head: string | undefined; for await (const frame of readFrames(await open())) { - head = JSON.parse(frame.data).cursor; // the first frame is stream_cursor - break; + if (frame.event === "stream_cursor") head = JSON.parse(frame.data).cursor; + break; // closes the stream } - await clear(); + if (head === undefined) throw new Error("The stream didn't name the head of the log"); + copy = { notes: {} }; const notes = pages(async (cursor) => { const { data, error } = await marfa.GET("/items", { params: { query: { type: "core.note", state: "any", cursor } }, @@ -185,45 +181,71 @@ async function rebuild(): Promise { if (error) throw new Error(error.error.message); return data; }); - for await (const item of notes) await apply({ event_type: "item.created", item }); + for await (const item of notes) apply({ event_type: "item.created", item: item as Item }); await saveCursor(head); return head; } +``` + +When Marfa can't read the head in time, the first frame is `stream_live` with a `null` cursor, so `rebuild` throws. + + + + +### Run it + +The loop resumes from the saved cursor after a break, and rebuilds when the cursor can't be used: -let cursor = await loadCursor(); // undefined on the first run +```ts title="sync.ts" +let cursor = copy.cursor; // undefined on the first run for (;;) { cursor ??= await rebuild(); const ended = await follow(cursor); - if (ended === "catchup_too_old" || ended === "cursor_ahead") cursor = undefined; - else cursor = await loadCursor(); + cursor = ended === "catchup_too_old" || ended === "cursor_ahead" ? undefined : copy.cursor; await new Promise((resolve) => setTimeout(resolve, 1000)); } ``` -A stream can also be refused before it opens, and `open` throws. After `429 rate_limited`, wait the seconds `Retry-After` names and try again. After `503 stream_capacity_exhausted`, the instance is serving as many streams as it allows: try again later. +Run it as an ES module, with `MARFA_API_URL` and `MARFA_API_KEY` set. + + + -Any other refusal, such as `403 type_not_permitted` or `400 unknown_type`, is a fault in the credential or the request, so **fix it rather than retry**. [Stream change events](/api-reference/event-stream/streamEvents) lists every answer. +## When the stream ends + +A stream doesn't end of its own accord. What ends it tells you what to do: + +| You see | It means | What to do | +| --- | --- | --- | +| The connection closes with no frame | A network or proxy fault. | Wait a moment, then resume from your saved cursor. | +| `stream_incomplete` | The stream couldn't go on, for the `reason` it names, such as `server_stopping`. Everything after your cursor is still in the log. | Resume from your saved cursor. For `credential_ended`, use a key that's valid. For `reader_behind`, take frames off the connection as they arrive and apply them from a queue. | +| `catchup_too_old` | The log no longer holds the events after your cursor. By default it keeps them for at least seven days. | Rebuild from the head of the log. | +| `cursor_ahead` | Your cursor is past the log's head, as after the instance was restored from a backup. | Discard the cursor and rebuild from the head of the log. | + +`marfa events` exits `1` for `cursor_ahead` and `catchup_too_old`, `3` for a `stream_incomplete` that a new stream can mend, and `5` for `credential_ended`, so a script needn't read the frames. [Exit codes](/cli#exit-codes) lists every status. + +A stream can also be refused before it opens, and then `open` throws: + +- **`429 rate_limited`:** wait the seconds `Retry-After` names, then try again. +- **`503 stream_capacity_exhausted`:** the instance is serving as many streams as it allows. Try again later. +- **Anything else,** such as `403 type_not_permitted`, `400 unknown_type`, or `400 validation_error` for a cursor the stream didn't write, is a fault in the credential or the request. Fix it rather than retry. + +[Stream change events](/api-reference/event-stream/streamEvents) lists every answer. ## Narrow the stream -`type` limits item events to a type and its subtypes. It takes up to ten types or patterns, separated by commas, such as `core.note,app.*`. Edge events come **whatever `type` says**; send `edges=none` to leave them out. +`type` limits item events to a type and its subtypes. It takes up to ten types or patterns, separated by commas, such as `core.note,app.*`. Edge events come whatever `type` says, so send `edges=none` to leave them out, as `sync.ts` does. ```bash tab="CLI" tab-group="client" -marfa events --type core.note --type core.task --edges none --from 78 +marfa events --type core.note --type core.task --edges none --from 4 ``` ```http tab="HTTP" GET /events?type=core.note,core.task&edges=none Authorization: Bearer -Last-Event-ID: 78 +Last-Event-ID: 4 ``` -A cursor works under **any filter**, and you can change the filter between connections. The IDs you receive skip the events the filter leaves out. - -## Build a working copy of your own - -A copy that stands in for the instance, answering reads offline the way a device does, **needs more than the plain stream**. It must show that what it listed, what it read and what it follows were all read with the same access, and start again when that access changes. - -That proof is a read view, carried on the copy stream, `GET /events?edges=all©=1`. The [read views chapter of the contract](https://github.com/withmarfa/marfa/blob/main/conformance/spec/read-views.md) says what such a copy must do. +A cursor works under any filter, so you can change the filter between connections. The rules behind this guide are in the [events](https://github.com/withmarfa/marfa/blob/main/conformance/spec/events.md) chapter of the contract. diff --git a/content/docs/guides/settings.mdx b/content/docs/guides/settings.mdx index 40faf00d..394d4001 100644 --- a/content/docs/guides/settings.mdx +++ b/content/docs/guides/settings.mdx @@ -33,7 +33,7 @@ Most defaults suit any instance. Check these first: | Setting | Check it when | Read more | | --- | --- | --- | -| `NODE_ENV` | Always. `production` refuses a start without `API_KEY_SALT`, `MARFA_AUTH_SECRET` and `MARFA_AUTH_BASE_URL`. | [Deploy](/guides/deploy#what-the-server-needs) | +| `NODE_ENV` | Always. `production` refuses a start without `API_KEY_SALT`, `MARFA_AUTH_SECRET` and `MARFA_AUTH_BASE_URL`. | [Deploy](/guides/deploy#run-it-without-a-container) | | `SQLITE_PATH`, `BLOB_PATH` | Always. They say where your data lives. | [Back up and restore](/guides/back-up-and-restore) | | `TRUSTED_PROXY_CIDRS`, `TRUSTED_PROXY_HEADER` | A proxy or tunnel sits in front of the server. | [Deploy](/guides/deploy#the-address) | | `S3_PREFIX` | Instances share a bucket. Each needs its own. | [Deploy](/guides/deploy#an-object-store) | From 01a36caa2b71fde16fe3091fbd497de68642d988 Mon Sep 17 00:00:00 2001 From: August Cayzer Date: Wed, 21 Oct 2026 17:16:31 +0100 Subject: [PATCH 2/3] docs(guides): keep the private socket anchor and fix a changelog link --- content/docs/(reference)/changelog.mdx | 2 +- content/docs/guides/deploy.mdx | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/content/docs/(reference)/changelog.mdx b/content/docs/(reference)/changelog.mdx index 2d13972a..c7c9b6f7 100644 --- a/content/docs/(reference)/changelog.mdx +++ b/content/docs/(reference)/changelog.mdx @@ -390,7 +390,7 @@ For the API and the CLI: - **The CLI warns once before it sends a credential over plain `http`** to a host other than this machine, and `--allow-http` says that's intended. - **A device client that can't refresh its token is refused `offline_access`**, rather than approved without a refresh token. - **Version history names the sign-in that wrote each version**, and carries a cursor to continue its pages. [History](/versions#history) covers it. -- **`POST /types/normalize` and `marfa types normalize` compare a candidate with its registered declared and resolved definitions** without writing it. [Compare a definition without registering it](/guides/define-a-type#compare-a-definition-without-registering-it) covers it. +- **`POST /types/normalize` and `marfa types normalize` compare a candidate with its registered declared and resolved definitions** without writing it. [Change the type](/guides/define-a-type#change-the-type) covers it. - **Unknown type and edge type schema keys are refused.** - **A connector registration shows persistent problems apart from its state and runs.** A connector replaces or clears its problems under its hold. diff --git a/content/docs/guides/deploy.mdx b/content/docs/guides/deploy.mdx index 6396e756..82ea0a6f 100644 --- a/content/docs/guides/deploy.mdx +++ b/content/docs/guides/deploy.mdx @@ -149,7 +149,9 @@ The database and the blobs are the whole of your data, so put them on storage th [Back up and restore](/guides/back-up-and-restore) says which files a copy needs. -`control/marfa.sock` is the private socket. A command run on the same machine, as the server's account, reaches the instance through it with the owner's authority. You use it to claim the instance, make the first key and [recover the owner's password](/guides/sign-in-as-the-owner#recover-the-owners-password). +### The private socket + +`control/marfa.sock` lets a command run on the same machine, as the server's account, reach the instance with the owner's authority. You use it to claim the instance, make the first key and [recover the owner's password](/guides/sign-in-as-the-owner#recover-the-owners-password). - The socket carries no key, so the server refuses item properties, edges, tags, search, counts, events and exports through it with `401 unauthorized`. Run those with a key. - The CLI refuses `--socket` while `MARFA_API_URL` or `MARFA_API_KEY` is set. From 560e38e79cfd734a530b2c32079fd5c523c89777 Mon Sep 17 00:00:00 2001 From: August Cayzer Date: Wed, 21 Oct 2026 17:24:41 +0100 Subject: [PATCH 3/3] docs(guides): tighten review fixes in the four rewritten guides --- content/docs/guides/define-a-type.mdx | 4 ++-- content/docs/guides/deploy.mdx | 4 ++-- content/docs/guides/export-and-restore.mdx | 4 ++-- content/docs/guides/keep-in-sync.mdx | 8 +++++++- 4 files changed, 13 insertions(+), 7 deletions(-) diff --git a/content/docs/guides/define-a-type.mdx b/content/docs/guides/define-a-type.mdx index 12443472..9203446a 100644 --- a/content/docs/guides/define-a-type.mdx +++ b/content/docs/guides/define-a-type.mdx @@ -55,7 +55,7 @@ To make a type readable as another without taking its properties, name that type } ``` -Marfa checks the claim against the [compatibility rules](/catalog/your-own-types#compatible-with) when you register or replace the type, and refuses one that doesn't hold with `422 compatible_with_violation`, naming the property. Later, it refuses the same way a replacement or deletion of the target that would break the claim, and `details.dependent_type_ids` names the types that claim it. A compatible type inherits nothing, so a listing of `core.note` leaves recipes out. +Marfa checks the claim against the [compatibility rules](/catalog/your-own-types#compatible-with) when you register or replace the type, and refuses one that doesn't hold with `422 compatible_with_violation`, naming the property. It then refuses the same way any replacement or deletion of the target that would break the claim, and `details.dependent_type_ids` names the types that claim it. A compatible type inherits nothing, so a listing of `core.note` leaves recipes out. ## Register the type @@ -130,7 +130,7 @@ Content-Type: application/json const { data, error } = await marfa.POST("/types/normalize", { body: definition }); ``` -The answer's `type` is your definition with its defaults made explicit, and `registered_type` is the registered one in the same form. Compare those two. `resolved_type` adds what the registered type inherits, so it never matches. Both registered forms are `null` when the identifier isn't registered. A key that Marfa doesn't know, anywhere in a definition, is refused `400 invalid_schema`. +The answer's `type` is your definition with its defaults made explicit, and `registered_type` is the registered one in the same form. Compare those two, not `resolved_type`, which adds what the registered type inherits. Both registered forms are `null` when the identifier isn't registered. An unknown schema key, anywhere in a definition, is refused `400 invalid_schema`. Then replace the type: diff --git a/content/docs/guides/deploy.mdx b/content/docs/guides/deploy.mdx index 82ea0a6f..d7aaac42 100644 --- a/content/docs/guides/deploy.mdx +++ b/content/docs/guides/deploy.mdx @@ -102,7 +102,7 @@ MARFA_AUTH_SECRET= MARFA_AUTH_BASE_URL=https://marfa.example.com ``` -Start it from `packages/server` under a supervisor, such as launchd or systemd. The `--import` loads [telemetry](/guides/telemetry). +Start it from `packages/server` under a supervisor, such as launchd or systemd, with these settings in its environment: the server doesn't read `marfa.env` itself. The `--import` loads [telemetry](/guides/telemetry). ```bash cd packages/server @@ -162,7 +162,7 @@ The database and the blobs are the whole of your data, so put them on storage th Point your supervisor or monitor at `GET /health`, which needs no credential. It answers `503` when the database, a write, the disk or blob storage is down, such as an unmounted blob drive, and `200` otherwise. The container checks it every 30 seconds. -`SIGTERM` or `SIGINT` stops the server within about eight seconds, so give it ten before a runtime kills the process. When the server stops, its exit status tells your supervisor whether to start it again: +`SIGTERM` or `SIGINT` stops the server within eight seconds, so give it ten before a runtime kills the process. When the server stops, its exit status tells your supervisor whether to start it again: | Status | Meaning | Your supervisor | | --- | --- | --- | diff --git a/content/docs/guides/export-and-restore.mdx b/content/docs/guides/export-and-restore.mdx index 81c68114..e226f9d8 100644 --- a/content/docs/guides/export-and-restore.mdx +++ b/content/docs/guides/export-and-restore.mdx @@ -3,7 +3,7 @@ title: Export and restore an archive description: Export your items, edges, types and files to an archive, and restore it into the same instance or another, such as a new build. --- -An archive carries your content to another instance: items, edges, the types and edge types you registered, and file bytes. It leaves the instance's own setup behind, so it holds no keys, webhooks, configuration or audit log. It's how you move data to a new build when you [upgrade](/guides/upgrade). +An archive carries your content to another instance: items with their tags, extensions and earlier versions, edges, the types and edge types you registered, and file bytes. It leaves the instance's own setup behind, so it holds no keys, webhooks, configuration or audit log. It's how you move data to a new build when you [upgrade](/guides/upgrade). An archive isn't a long-term backup, because only the build that wrote it is sure to read it. To recover an instance from a failed disk, [back up the data directory](/guides/back-up-and-restore) instead. @@ -42,7 +42,7 @@ GET /export?format=archive&state=any Authorization: Bearer ``` -The CLI writes to a hidden file beside the path and renames it over the path only when the export ended whole, so the file at the path is always the last whole archive the command wrote. Any other ending leaves the path as it was and exits with a nonzero [exit code](/cli#exit-codes). +The CLI writes to a hidden file beside the path and renames it over the path only when the export ended whole and lacks no files, so the file at the path is always the last whole archive the command wrote. Any other ending leaves the path as it was and exits with a nonzero [exit code](/cli#exit-codes). diff --git a/content/docs/guides/keep-in-sync.mdx b/content/docs/guides/keep-in-sync.mdx index 82bce432..fcbd15d7 100644 --- a/content/docs/guides/keep-in-sync.mdx +++ b/content/docs/guides/keep-in-sync.mdx @@ -3,7 +3,7 @@ title: Keep a copy in sync description: Follow an instance's event stream from a cursor, so a copy of your data stays current and picks up where it stopped after a break. --- -A search index, a cache or a report keeps its own copy of an instance's data. Read what you need once, then apply each change from the [event stream](/events) and save a cursor, so that after a break you receive everything you missed and nothing twice. The Swift SDK and the CLI do this for you. +A search index, a cache or a report keeps its own copy of an instance's data. Read what you need once, then apply each change from the [event stream](/events) and save a cursor, so that after a break you receive everything you missed and nothing twice. The [Swift SDK](/swift-sdk) and the [CLI](/cli) do this for you. You need a credential that can read the types you copy. A copy that answers reads offline in the instance's place, as a device does, needs more: the [read views chapter of the contract](https://github.com/withmarfa/marfa/blob/main/conformance/spec/read-views.md) says what. @@ -29,6 +29,12 @@ The stream opens with `stream_cursor`, which names the **head of the log**, the Over HTTP, the [stream](/api-reference/event-stream/streamEvents) takes your cursor in the `Last-Event-ID` header and answers server-sent events. From cursor `5`, with the item cut short: +```http +GET /events +Authorization: Bearer +Last-Event-ID: 5 +``` + ```text : connected