diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index e3c07fb9c..ba02cf157 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -282,6 +282,14 @@ jobs: # The bridge's pure parts; `node --test` needs no install. - run: node --test catalog/bailey/bridge/images/bridge/src/*.test.js + postgres-credential: + name: Postgres credential container tests + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v7 + # The container's pure parts, on the standard library alone. + - run: python3 catalog/postgres/database/images/credential/test_bootstrap.py + # Kilo Code's Tangle plugin (`tangle/kilo-code/`): its tests are # TypeScript run straight by `node --test`, which strips the types # itself from Node 22.18 on, so they need no install either. @@ -318,7 +326,7 @@ jobs: # The name the main ruleset requires (see the `rust` job). editor: name: graph + editor - needs: [graph, connect, syntax, vscode, bridge, tangle, browser] + needs: [graph, connect, syntax, vscode, bridge, postgres-credential, tangle, browser] if: ${{ !cancelled() }} runs-on: ubuntu-latest env: diff --git a/CHANGELOG.md b/CHANGELOG.md index ed17499c2..0c274d71f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,101 @@ What changed between versions of weft that you need to know about when you update. To update, run `weft maintenance on`, pull and run `./setup.sh` again, then run `weft maintenance off`. +## 2026-10-31 (dbefc004) + +**What you have to change after this update, if you use a PostgresDatabase:** + +- **Clients outside the program sign in as `app`, no longer as `weft`.** The + password, host and port stay the same; `weft infra env --into .env + --set DATABASE_USER=User` writes the user again. +- **A frontend's pool must expect a connection idle for a minute to close.** + The database now closes it. Set the pool's idle timeout under a minute, or + have it test a connection before using it. +- **An extension that needs the superuser goes in the node's new `extensions` + setting** (`["pg_trgm", "postgres_fdw"]`). The superuser makes each one on + every start, and the program, signed in as `app`, uses it. + +**A PostgresDatabase no longer runs out of connections when the program +scales.** Postgres serves a fixed number of connections, while its clients +keep changing: workers scale, a change of worker settings runs old and new +workers side by side, and a frontend connects too. Each worker used to keep up +to 20 connections open whether it used them or not, so once enough workers ran +at once, Postgres answered `too many clients` to them and to everything else +using that database. A PostgresDatabase now runs a pooler in front of Postgres, +in the same unit. The pooler never opens more connections than Postgres takes, +and a client that finds them all busy waits for one, so a run waits instead of +failing. Watching queries (PostgresWatchQuery) get a quarter of the +connections the pooler opens (what is left after 3 kept for the superuser and +2 spare), at least one, so a crowd of them never leaves the other queries +without a connection, or the other way round. If you want to see who holds the +connections, the node's card now shows them: Postgres's own by user and state, +each of the pooler's two shares with its clients served and waiting, and the +clients by address. + +Your existing PostgresDatabase nodes get the pooler on their next `weft infra +upgrade`. When the database starts again it gets a new role, `app`, which +becomes the owner of everything the superuser owned in it (except objects an +extension made). After that: + +- **Through the pooler a client reaches this one database only**, so a client + that also used `postgres` or another database made beside it can no longer + reach those. The superuser `weft` can only sign in from inside the unit. An + extension listed in `extensions` that this Postgres does not have fails the + start, and `weft infra status` names it. +- **A database whose name the pooler cannot read as written gets a second + name for clients.** Your database keeps its name and its data. When the + name is letters, digits, `_` and `-` only, at most 57 characters, and none + of `pgbouncer`, `all`, `sameuser`, `samerole`, `samegroup` or + `replication`, clients use it as before; otherwise they sign in by a name the + pooler makes from it (`db_my_app_1a2b3c4d`), which the connection carries + and the card shows under Database. A client of yours outside the program + takes it from there. +- **The database closes a connection that sat idle for a minute.** That gives + back a connection held by a worker that lost its CPU between requests, or by + an old worker a rollout left behind. If a client of yours keeps a connection idle on purpose (one that `LISTEN`s), have it lift that + limit for its own session when it connects, the way PostgresWatchQuery does: + `SELECT set_config('idle_session_timeout', '0', false) FROM pg_settings + WHERE name = 'idle_session_timeout'`. +- **The least `memory` is 565Mi** (it was 886Mi). At 1Gi Postgres serves 23 + connections, at 2Gi 59, at 4Gi 131. +- **A new database needs Postgres 14 or later.** One you already have on an + older Postgres keeps running, but it cannot close idle connections, so its + card and `weft infra status` warn of it and list the commands that dump it + and restore it into a new node on 17. + +**Every Postgres node closes a pooled connection that sat idle for 5 +seconds**, and never hands out one idle for longer, whichever database it talks +to. + +**A stopping worker gives back what its runs shared** (`ctx.shared`) as soon +as its last run has ended, instead of when the process goes, so a pool's +connections return to the database while the worker writes its last records. + +**On Cloud Run, a change of worker settings retires the old workers.** Once +the new settings serve, weft moves the calls the old ones took to them and +removes that program's revisions under any other settings, so their copies stop +instead of running beside the new ones. Which settings are current is decided +from what the project has stored, so a dispatcher copy that briefly still held +the old settings and brought them back up never wins: that revision takes no +calls and scales to zero. It goes at the next settings change or activation, +unless it is the service's latest, which Cloud Run does not let anyone delete; +then it goes once a later settings change or build replaces it. + +**When an infra node's setup fails, you are told why.** `weft infra start`, +`weft activate` and the other commands that wait on a setup used to answer +`infra setup failed` and nothing else; they now give the error the setup run +failed with (a trigger's setup the same), as `weft runs` shows it. + +**A unit that knows why it is not ready says so in `weft infra status`.** When +a container's health check answers an error with a reason in its body (JSON +`{"error": "..."}`, or plain text), the status shows that reason after the +status code. + +**A node's fake tests hold a published connection to its service's values.** +A value the service does not declare, or a required one missing, fails the +run the way the access store refuses it on an install; before, the fake rig +stored anything and only a real run found out. + ## 2026-10-31 (ed587b61) **On a cloud install, how many holders run follows how busy they are**, not diff --git a/TODO.md b/TODO.md index b46a761eb..6a2f4e0fe 100644 --- a/TODO.md +++ b/TODO.md @@ -633,25 +633,6 @@ minute on one project. [Update Notice Warning] If we touch `crates/weft-engine/src/door/limits.rs` or `/v1/door/tick`, revisit this entry. -## Several workers of a project can open more Postgres connections than the database takes - -**Problem.** The Postgres node keeps one pool per worker and database -(`catalog/postgres/postgres.rs`, `MOST_AT_ONCE`), held to 20 connections -so a few workers together stay under Postgres's default limit of 100. -That is a guess about how many workers there are. A project scaled to -many workers can still open more connections than the database accepts, -and the runs past it fail to connect. - -**Direction.** Put a connection pooler (PgBouncer, in transaction mode) -in front of the database inside the Postgres node's own infra, so every -worker talks to the pooler and the pooler holds the few real -connections. The worker's own limit can then be generous again. The open -question is a database the user brings, which has no infra of ours to -put the pooler in. - -[Update Notice Warning] If we touch `catalog/postgres/postgres.rs`'s -pool or the Postgres node's infra, revisit this entry. - ## A Python process runs one script at a time **Problem.** `ExecPython` keeps as many `python3` processes as the worker diff --git a/catalog/bailey/bridge/mod.rs b/catalog/bailey/bridge/mod.rs index 5d38b13a5..c4dcf0880 100644 --- a/catalog/bailey/bridge/mod.rs +++ b/catalog/bailey/bridge/mod.rs @@ -64,6 +64,7 @@ impl Node for BaileyBridgeNode { }], keep_on_terminate: Vec::new(), health_rules: None, + notes: Vec::new(), }) } diff --git a/catalog/computer/computer/mod.rs b/catalog/computer/computer/mod.rs index 8856ac70b..59032b726 100644 --- a/catalog/computer/computer/mod.rs +++ b/catalog/computer/computer/mod.rs @@ -104,6 +104,7 @@ impl Node for ComputerNode { endpoints, keep_on_terminate: Vec::new(), health_rules: None, + notes: Vec::new(), }) } diff --git a/catalog/postgres/access/metadata.json b/catalog/postgres/access/metadata.json index 742bc30ad..80e28c08e 100644 --- a/catalog/postgres/access/metadata.json +++ b/catalog/postgres/access/metadata.json @@ -69,6 +69,13 @@ "secret": false, "optional": true, "placeholder": "require (or verify-ca / verify-full; disable turns TLS off)" + }, + { + "name": "watchDatabase", + "label": "Database for watching queries", + "secret": false, + "optional": true, + "placeholder": "the same as Database (a pooler's separate name for long-held connections)" } ] } diff --git a/catalog/postgres/database/images/credential/bootstrap.py b/catalog/postgres/database/images/credential/bootstrap.py index f9eba7451..808e1315a 100644 --- a/catalog/postgres/database/images/credential/bootstrap.py +++ b/catalog/postgres/database/images/credential/bootstrap.py @@ -8,59 +8,106 @@ changes, which is what keeps the database answering to the password it was created with. - serve Run beside Postgres. Answers the password to whoever asks - over the network, then stops answering once the asker - proves it stored it, by sending the password back. - Proof, not a bare say-so: otherwise anything able to reach - this port could retire a password nobody holds, and the - database would be locked away with no way back. + serve Run beside Postgres and the pooler. First it sets the database + up for the program (`converge`): the program's own role, which + owns the program's database and everything in it and is no + superuser, signing in with the password on the shared volume; + then it tells the pooler who may sign in. Every start does it + again, so a database made before any of this existed is brought + to the same shape. Until it is done, `/health` says not ready. + + Then it answers the password to whoever asks over the network, + and stops answering once the asker proves it stored it, by + sending the password back. Proof, not a bare say-so: otherwise + anything able to reach this port could retire a password nobody + holds, and the database would be locked away with no way back. That same proof doubles as the answer to "is what I hold still this database's password", which is how a caller finds out its disk was replaced underneath it. It also serves the graph: `/live` shows the database and user - to sign in as and whether the password is still readable, and - its one button, `/action` with - `reset_password`, mints a new password, sets it on the - database over the unix socket (which needs no password), and - makes it readable again. Then it tells weft the new password - (`WEFT_VALUES_URL`, the agent beside this unit), which writes it - into the connection the node published, so the program's next - run signs in with it; the card shows it until the node's next - run retires it. - -Where the files live and which port to serve on come from the node -that declares this container, so the two sides cannot drift. + to sign in as, whether the password is still readable, and who + is connected (to Postgres and to the pooler, read when the card + asks, never on a timer); its one button, `/action` with + `reset_password`, mints a new password, sets it on the program's + role over the unix socket (which needs no password), tells the + pooler, and makes it readable again. Then it tells weft the new + password (`WEFT_VALUES_URL`, the agent beside this unit), which + writes it into the connection the node published, so the + program's next run signs in with it; the card shows it until the + node's next run retires it. + +Where the files live, which ports and names to use, all come from the +node that declares this container, so the two sides cannot drift. """ +import csv +import hashlib +import hmac +import base64 import http.server +import io import json import os import secrets import subprocess import sys +import threading +import time import urllib.error import urllib.request +# SYNC: the credential container's settings <-> catalog/postgres/database/mod.rs (credential_env) SECRET_DIR = os.environ.get("WEFT_SECRET_DIR") PASSWORD_FILE = os.environ.get("WEFT_PASSWORD_FILE") PORT = os.environ.get("WEFT_CREDENTIAL_PORT") SOCKET_DIR = os.environ.get("WEFT_SOCKET_DIR") +POSTGRES_PORT = os.environ.get("WEFT_POSTGRES_PORT") ADMIN_USER = os.environ.get("WEFT_ADMIN_USER") +APP_USER = os.environ.get("WEFT_APP_USER") +# The database as Postgres has it, and the names the pooler lets clients +# in by: the one a client signs in with and the watching queries' own. DATABASE = os.environ.get("WEFT_DATABASE") +POOLER_DATABASE = os.environ.get("WEFT_POOLER_DATABASE") +WATCH_DATABASE = os.environ.get("WEFT_WATCH_DATABASE") +# The extensions the program's database needs, as a JSON list of names. +EXTENSIONS_JSON = os.environ.get("WEFT_EXTENSIONS") +# The Postgres this unit runs, the first major that closes idle sessions, +# and what to warn of when it is older (empty when it is not). +POSTGRES_MAJOR = os.environ.get("WEFT_POSTGRES_MAJOR") +CLOSES_IDLE_SESSIONS_FROM = os.environ.get("WEFT_CLOSES_IDLE_SESSIONS_FROM") +POSTGRES_NOTE = os.environ.get("WEFT_POSTGRES_NOTE") +DATA_DIR = os.environ.get("WEFT_DATA_DIR") +IDLE_SESSION_SECONDS = os.environ.get("WEFT_IDLE_SESSION_SECONDS") +POOLER_DIR = os.environ.get("WEFT_POOLER_DIR") +POOLER_PORT = os.environ.get("WEFT_POOLER_PORT") +POOLER_USERS_FILE = os.environ.get("WEFT_POOLER_USERS_FILE") # Where this unit tells weft what changed of what the node handed it: set # by weft on every container of an infra unit, never by the node. # SYNC: WEFT_VALUES_URL <-> crates/weft-core/src/infra/resolve.rs (VALUES_URL_ENV) VALUES_URL = os.environ.get("WEFT_VALUES_URL") -if not SECRET_DIR or not PASSWORD_FILE or not PORT or not SOCKET_DIR or not ADMIN_USER or not DATABASE: - sys.exit( - "WEFT_SECRET_DIR, WEFT_PASSWORD_FILE, WEFT_CREDENTIAL_PORT, WEFT_SOCKET_DIR, " - "WEFT_ADMIN_USER and WEFT_DATABASE must be set by the node" - ) +_FROM_THE_NODE = { + "WEFT_SECRET_DIR": SECRET_DIR, "WEFT_PASSWORD_FILE": PASSWORD_FILE, "WEFT_CREDENTIAL_PORT": PORT, + "WEFT_SOCKET_DIR": SOCKET_DIR, "WEFT_POSTGRES_PORT": POSTGRES_PORT, "WEFT_ADMIN_USER": ADMIN_USER, + "WEFT_APP_USER": APP_USER, "WEFT_DATABASE": DATABASE, "WEFT_POOLER_DATABASE": POOLER_DATABASE, + "WEFT_WATCH_DATABASE": WATCH_DATABASE, "WEFT_EXTENSIONS": EXTENSIONS_JSON, "WEFT_POSTGRES_MAJOR": POSTGRES_MAJOR, + "WEFT_CLOSES_IDLE_SESSIONS_FROM": CLOSES_IDLE_SESSIONS_FROM, "WEFT_DATA_DIR": DATA_DIR, + "WEFT_IDLE_SESSION_SECONDS": IDLE_SESSION_SECONDS, "WEFT_POOLER_DIR": POOLER_DIR, + "WEFT_POOLER_PORT": POOLER_PORT, "WEFT_POOLER_USERS_FILE": POOLER_USERS_FILE, +} +_missing = [name for name, value in _FROM_THE_NODE.items() if not value] +if _missing: + sys.exit(f"{', '.join(_missing)} must be set by the node") +if POSTGRES_NOTE is None: + sys.exit("WEFT_POSTGRES_NOTE must be set by the node, empty when there is nothing to warn of") if not VALUES_URL: sys.exit("WEFT_VALUES_URL must be set by weft on every container of an infra unit") PORT = int(PORT) +CLOSES_IDLE_SESSIONS = int(POSTGRES_MAJOR) >= int(CLOSES_IDLE_SESSIONS_FROM) +EXTENSIONS = json.loads(EXTENSIONS_JSON) +if not isinstance(EXTENSIONS, list) or not all(isinstance(name, str) and name for name in EXTENSIONS): + sys.exit(f"WEFT_EXTENSIONS must be a JSON list of names, got {EXTENSIONS_JSON}") SEALED_FILE = os.path.join(SECRET_DIR, "sealed") # SYNC: credential routes <-> catalog/postgres/database/mod.rs @@ -73,8 +120,24 @@ ACTION_PATH = "/action" RESET_ACTION = "reset_password" +# Set once `converge` finished: the program's role is in place and the +# pooler knows who may sign in. `/health` answers ready from then on. +CONVERGED = threading.Event() +# Why the program cannot sign in yet, while `converge` waits or tries +# again: what `/health` answers until it is done, which the unit's +# readiness check carries to `weft infra status`. +NOT_READY = {"why": "the database is still being set up for the program"} +# How long `converge` waits before trying again after the database refused. +CONVERGE_RETRY_SECONDS = 30 +# One change of the password at a time: two presses at once would leave +# the pooler told one password and the database holding the other. +CHANGING = threading.Lock() + def mint() -> None: + refused = too_old_for_a_new_database(CLOSES_IDLE_SESSIONS, os.path.exists(os.path.join(DATA_DIR, "PG_VERSION"))) + if refused is not None: + sys.exit(f"credential: {refused}") # Made on EVERY boot, before any other container starts, because # the container that serves the password is given this directory # alone rather than the whole disk. A directory that already @@ -87,76 +150,391 @@ def mint() -> None: write_password(new_password()) +def too_old_for_a_new_database(closes_idle_sessions: bool, has_data: bool) -> str | None: + """Why this start must not make a database, or None. Run before Postgres + starts, the one moment that can tell a new disk from one an older weft + filled: an existing database on a Postgres that does not close idle + sessions keeps running (`POSTGRES_NOTE` says what it lacks), a new one + is never made on it.""" + if closes_idle_sessions or has_data: + return None + return ( + f"Postgres {POSTGRES_MAJOR} cannot be used for a new database: this node needs {CLOSES_IDLE_SESSIONS_FROM} or " + f"later, which closes the connections a client leaves idle. Set the node's `version` to 17 and start it again" + ) + + def new_password() -> str: # url-safe: the value travels in a connection string, and it is # quoted into SQL by the reset below, so no quote can be in it. return secrets.token_urlsafe(32) -def write_password(password: str) -> None: +def write_file(path: str, text: str) -> None: + """Write `text` to `path`, whole or not at all, readable by the infra unit's group.""" # Written under a temporary name and renamed once complete AND on # disk, so neither a crash mid-write nor a node reboot can leave a - # half password, or lose one Postgres has already adopted. - tmp = PASSWORD_FILE + ".partial" + # half file, or lose a password Postgres has already adopted. + tmp = path + ".partial" with open(tmp, "w", encoding="utf-8") as f: - f.write(password) + f.write(text) f.flush() os.fsync(f.fileno()) - # Group-readable, because Postgres runs as a different user in the - # same infra unit and reads this file to set its own password. The group - # is the unit's fsGroup, which only its own containers are in. + # Group-readable, because Postgres and the pooler run as other users + # in the same infra unit and read these files. The group is the + # unit's fsGroup, which only its own containers are in. os.chmod(tmp, 0o640) - os.replace(tmp, PASSWORD_FILE) - dir_fd = os.open(SECRET_DIR, os.O_RDONLY) + os.replace(tmp, path) + dir_fd = os.open(os.path.dirname(path), os.O_RDONLY) try: os.fsync(dir_fd) finally: os.close(dir_fd) +def write_password(password: str) -> None: + write_file(PASSWORD_FILE, password) + + def sealed() -> bool: return os.path.exists(SEALED_FILE) -def reset_password() -> tuple[str | None, str]: - """Mint a new password, set it on the database, make it readable again. +class Refused(Exception): + """What psql said when the database or the pooler refused.""" + + +def psql(script: str, variables: dict, database: str | None = None, pooler: bool = False) -> str: + """Run `script` as the superuser and hand back what it printed as CSV. + + Over Postgres's unix socket, which trusts a local connection without + a password, or with `pooler`, over the pooler's own socket to its + admin console. The script arrives on STDIN, not through `-c`: + variables (`:'name'` quotes as a literal, `:"name"` as an identifier) + are a psql feature, and psql only expands them while READING a + script, so a value never touches the SQL text itself. Text handed to + `-c` goes to the server as it stands, which is why the graph's reset + button once failed with `syntax error at or near ":"`.""" + host, port, database = (POOLER_DIR, POOLER_PORT, "pgbouncer") if pooler else (SOCKET_DIR, POSTGRES_PORT, database or DATABASE) + args = ["psql", "-h", host, "-p", port, "-U", ADMIN_USER, "-X", "-q", "--csv", "-v", "ON_ERROR_STOP=1"] + for name, value in variables.items(): + args += ["-v", f"{name}={value}"] + # The database by `PGDATABASE`, never `-d`: psql reads a `-d` holding + # `=`, or starting `postgres://`, as a whole connection string, and a + # database may be named anything. + run = subprocess.run(args, input=script, capture_output=True, text=True, env={**os.environ, "PGDATABASE": database}) + if run.returncode != 0: + raise Refused((run.stderr or run.stdout).strip() or f"psql exited {run.returncode}") + return run.stdout + + +def rows(printed: str) -> list[dict]: + """What psql printed as CSV for ONE statement, as one dict per row. + Several results back to back cannot be told apart, so a caller that + reads rows sends one statement at a time. A one-column row whose value + is empty prints as an empty line, which reads as no fields at all.""" + records = list(csv.reader(io.StringIO(printed))) + return [dict(zip(records[0], record or [""])) for record in records[1:]] if records else [] + + +# Every object in the program's database the superuser owns that the +# program's role must own to use, change or drop the way the program +# made it: what an older database made before the program had its own +# role holds, and what a person operating the database as the superuser +# adds later. Members of an extension stay where the extension put them, +# a sequence owned by a column moves with its table, and a table's row +# type and an array type move with what they belong to. +OWNED_BY_ADMIN = r""" +WITH admin AS (SELECT oid FROM pg_roles WHERE rolname = :'admin'), + member AS (SELECT classid, objid FROM pg_depend WHERE deptype = 'e'), + mine AS (SELECT oid FROM pg_namespace WHERE nspname !~ '^pg_' AND nspname <> 'information_schema') +SELECT format('ALTER SCHEMA %I OWNER TO %I', n.nspname, :'app') + FROM pg_namespace n + WHERE n.nspowner = (SELECT oid FROM admin) AND n.oid IN (SELECT oid FROM mine) + AND (n.oid) NOT IN (SELECT objid FROM member WHERE classid = 'pg_namespace'::regclass) +UNION ALL +SELECT format('ALTER %s %s OWNER TO %I', + CASE c.relkind WHEN 'v' THEN 'VIEW' WHEN 'm' THEN 'MATERIALIZED VIEW' WHEN 'S' THEN 'SEQUENCE' + WHEN 'f' THEN 'FOREIGN TABLE' ELSE 'TABLE' END, + c.oid::regclass, :'app') + FROM pg_class c + WHERE c.relowner = (SELECT oid FROM admin) AND c.relnamespace IN (SELECT oid FROM mine) + AND c.relkind IN ('r', 'p', 'v', 'm', 'S', 'f') + AND c.oid NOT IN (SELECT objid FROM member WHERE classid = 'pg_class'::regclass) + AND NOT (c.relkind = 'S' AND EXISTS (SELECT FROM pg_depend d + WHERE d.classid = 'pg_class'::regclass AND d.objid = c.oid + AND d.refclassid = 'pg_class'::regclass AND d.deptype IN ('a', 'i'))) +UNION ALL +SELECT format('ALTER ROUTINE %s OWNER TO %I', p.oid::regprocedure, :'app') + FROM pg_proc p + WHERE p.proowner = (SELECT oid FROM admin) AND p.pronamespace IN (SELECT oid FROM mine) + AND p.oid NOT IN (SELECT objid FROM member WHERE classid = 'pg_proc'::regclass) +UNION ALL +SELECT format('ALTER %s %s OWNER TO %I', CASE t.typtype WHEN 'd' THEN 'DOMAIN' ELSE 'TYPE' END, t.oid::regtype, :'app') + FROM pg_type t + WHERE t.typowner = (SELECT oid FROM admin) AND t.typnamespace IN (SELECT oid FROM mine) + AND (t.typtype IN ('d', 'e', 'r') + OR (t.typtype = 'c' AND (SELECT relkind FROM pg_class WHERE oid = t.typrelid) = 'c')) + AND t.oid NOT IN (SELECT objid FROM member WHERE classid = 'pg_type'::regclass) +UNION ALL +SELECT format('ALTER LARGE OBJECT %s OWNER TO %I', l.oid, :'app') + FROM pg_largeobject_metadata l + WHERE l.lomowner = (SELECT oid FROM admin) +UNION ALL +SELECT format('ALTER STATISTICS %s OWNER TO %I', format('%I.%I', ns.nspname, s.stxname), :'app') + FROM pg_statistic_ext s JOIN pg_namespace ns ON ns.oid = s.stxnamespace + WHERE s.stxowner = (SELECT oid FROM admin) AND s.stxnamespace IN (SELECT oid FROM mine) +UNION ALL +SELECT format('ALTER COLLATION %s OWNER TO %I', format('%I.%I', ns.nspname, co.collname), :'app') + FROM pg_collation co JOIN pg_namespace ns ON ns.oid = co.collnamespace + WHERE co.collowner = (SELECT oid FROM admin) AND co.collnamespace IN (SELECT oid FROM mine) + AND co.oid NOT IN (SELECT objid FROM member WHERE classid = 'pg_collation'::regclass) +UNION ALL +SELECT format('ALTER CONVERSION %s OWNER TO %I', format('%I.%I', ns.nspname, cv.conname), :'app') + FROM pg_conversion cv JOIN pg_namespace ns ON ns.oid = cv.connamespace + WHERE cv.conowner = (SELECT oid FROM admin) AND cv.connamespace IN (SELECT oid FROM mine) + AND cv.oid NOT IN (SELECT objid FROM member WHERE classid = 'pg_conversion'::regclass) +UNION ALL +SELECT format('ALTER OPERATOR %s OWNER TO %I', o.oid::regoperator, :'app') + FROM pg_operator o + WHERE o.oprowner = (SELECT oid FROM admin) AND o.oprnamespace IN (SELECT oid FROM mine) + AND o.oid NOT IN (SELECT objid FROM member WHERE classid = 'pg_operator'::regclass) +UNION ALL +SELECT format('ALTER TEXT SEARCH CONFIGURATION %s OWNER TO %I', cfg.oid::regconfig, :'app') + FROM pg_ts_config cfg + WHERE cfg.cfgowner = (SELECT oid FROM admin) AND cfg.cfgnamespace IN (SELECT oid FROM mine) + AND cfg.oid NOT IN (SELECT objid FROM member WHERE classid = 'pg_ts_config'::regclass) +UNION ALL +SELECT format('ALTER TEXT SEARCH DICTIONARY %s OWNER TO %I', dict.oid::regdictionary, :'app') + FROM pg_ts_dict dict + WHERE dict.dictowner = (SELECT oid FROM admin) AND dict.dictnamespace IN (SELECT oid FROM mine) + AND dict.oid NOT IN (SELECT objid FROM member WHERE classid = 'pg_ts_dict'::regclass) +\gexec +""" - The database is told first, over its unix socket, which trusts a - local connection without a password: a file that named a password - the database did not have would lock every sign-in out. Returns the - refusal text when the database would not take it, so the caller - sees exactly what psql said, and the new password.""" - password = new_password() - # `psql` variables interpolate with quoting (`:'name'` quotes as a - # literal, `:"name"` as an identifier), so the password never touches - # the SQL text itself. - # - # The statement arrives on STDIN, not through `-c`. Variables are a - # psql feature, and psql only expands them while READING a script: - # text handed to `-c` goes to the server as it stands, so the server - # gets a literal `:"user"` and answers `syntax error at or near ":"`. - # That is what shipped, which is why the graph's reset button never - # worked. - run = subprocess.run( - [ - "psql", "-h", SOCKET_DIR, "-U", ADMIN_USER, "-d", "postgres", - "-v", "ON_ERROR_STOP=1", "-v", f"user={ADMIN_USER}", "-v", f"password={password}", - ], - input="ALTER USER :\"user\" PASSWORD :'password';\n", +# The program's role, made when missing and held to what it must be on +# every start: it signs in and it is no superuser. The program's database +# is its own, with everything in it. +CONVERGE = r""" +SELECT format('CREATE ROLE %I LOGIN', :'app') WHERE NOT EXISTS (SELECT FROM pg_roles WHERE rolname = :'app') +\gexec +ALTER ROLE :"app" WITH LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION NOBYPASSRLS; +ALTER DATABASE :"database" OWNER TO :"app"; +""" + OWNED_BY_ADMIN + +# Postgres closes the role's sessions once they sit idle, which is how a +# connection a client holds and does not use goes back to the pooler +# whoever the client is. Set on a Postgres that has the setting only: an +# older one keeps its idle sessions open, which its card warns of. +CLOSE_IDLE_SESSIONS = """ALTER ROLE :"app" SET idle_session_timeout = :'idle';\n""" + + +def scram_accepts(verifier: str | None, password: str) -> bool: + """Whether Postgres's stored `verifier` (`pg_authid.rolpassword`) is one for `password`. + + A SCRAM-SHA-256 verifier keeps a hash of the key the password + derives, so checking it never needs the database to try a sign-in. + Anything else (no password, an md5 one) answers no.""" + if not verifier or not verifier.startswith("SCRAM-SHA-256$"): + return False + try: + iterations_salt, keys = verifier[len("SCRAM-SHA-256$"):].split("$", 1) + iterations, salt = iterations_salt.split(":", 1) + stored_key = keys.split(":", 1)[0] + salted = hashlib.pbkdf2_hmac("sha256", password.encode("utf-8"), base64.b64decode(salt), int(iterations)) + client_key = hmac.new(salted, b"Client Key", hashlib.sha256).digest() + return hmac.compare_digest(hashlib.sha256(client_key).digest(), base64.b64decode(stored_key)) + except (ValueError, TypeError): + return False + + +def pooler_users(verifier: str) -> str: + """The pooler's file of who may sign in: the program's role with its + verifier (the pooler checks a client's password against it, and signs + in to Postgres with what the client proved), and the superuser with no + password at all, which the pooler's own rules let in only to its admin + console over its unix socket.""" + quoted = lambda value: '"' + value.replace('"', '""') + '"' + return f"{quoted(APP_USER)} {quoted(verifier)}\n{quoted(ADMIN_USER)} \"\"\n" + + +def app_verifier() -> str | None: + """The program's role's stored password verifier, or None when it has none.""" + found = rows(psql("SELECT rolpassword FROM pg_authid WHERE rolname = :'app';\n", {"app": APP_USER})) + if not found: + return None + return found[0]["rolpassword"] or None + + +def set_app_password(password: str) -> None: + psql("ALTER ROLE :\"app\" PASSWORD :'password';\n", {"app": APP_USER, "password": password}) + + +def tell_the_pooler() -> None: + """Write who may sign in from the database's own records, then have + the pooler read it again. Raises `Refused` when the pooler would not.""" + verifier = app_verifier() + if verifier is None: + raise Refused(f"the role {APP_USER} has no password in the database") + write_file(POOLER_USERS_FILE, pooler_users(verifier)) + psql("RELOAD;\n", {}, pooler=True) + + +def wait_until(what: str, not_yet) -> None: + """Wait until `not_yet()` answers None, keeping what it answered instead + for `/health` and saying it in the log every half minute.""" + started = time.monotonic() + said = started + while (why := not_yet()) is not None: + NOT_READY["why"] = f"waiting for {what}: {why}" + if time.monotonic() - said >= 30: + sys.stderr.write(f"credential: still waiting for {what} ({int(time.monotonic() - started)}s): {why}\n") + said = time.monotonic() + time.sleep(1) + + +def postgres_not_serving() -> str | None: + # Over TCP, for the reason the node's readiness check gives: the + # image runs a socket-only server while it initialises. + check = subprocess.run( + ["pg_isready", "-h", "127.0.0.1", "-p", POSTGRES_PORT, "-U", ADMIN_USER, "-d", "postgres"], capture_output=True, text=True, ) - if run.returncode != 0: - detail = (run.stderr or run.stdout).strip() or f"psql exited {run.returncode}" - sys.stderr.write(f"credential: reset refused by the database: {detail}\n") - return detail, password - write_password(password) + return None if check.returncode == 0 else (check.stdout or check.stderr).strip() + + +def pooler_not_told() -> str | None: try: - os.remove(SEALED_FILE) - except FileNotFoundError: - pass + tell_the_pooler() + return None + except Refused as e: + return str(e) + + +def converge() -> None: + """Bring the database to the shape the node declares, then tell the pooler, + trying again every half minute while the database refuses. + + A refusal is often over by itself (Postgres restarting under the + container), and when it is not (an extension this Postgres does not + have) only a change to the node fixes it, which starts the unit + afresh. Meanwhile `/health` answers what was refused, so the unit's + readiness check says it where the person looks, and the program is + never handed a database it cannot sign in to or that lacks an + extension it needs.""" + while (refused := converge_once()) is not None: + NOT_READY["why"] = f"{refused}; trying again every {CONVERGE_RETRY_SECONDS}s" + sys.stderr.write(f"credential: {refused}\n") + time.sleep(CONVERGE_RETRY_SECONDS) + CONVERGED.set() + sys.stderr.write(f"credential: the program signs in as {APP_USER}, through the pooler\n") + + +def converge_once() -> str | None: + """One pass of `converge`: what the database refused, or None once done. + + The password file is the one source of the program's password: the + role is given it when it does not already have it, so a reset that + stopped halfway is settled by the next start. Each extension the + node lists is made by the superuser when it is not there yet.""" + wait_until("Postgres to accept connections", postgres_not_serving) + password = stored_password() + if password is None: + sys.exit("credential: no password on the shared volume; the init container writes it before this one starts") + try: + psql(CONVERGE, {"app": APP_USER, "admin": ADMIN_USER, "database": DATABASE}) + if CLOSES_IDLE_SESSIONS: + psql(CLOSE_IDLE_SESSIONS, {"app": APP_USER, "idle": f"{IDLE_SESSION_SECONDS}s"}) + if not scram_accepts(app_verifier(), password): + set_app_password(password) + except Refused as e: + return f"the database refused to be set up for the program: {e}" + for name in EXTENSIONS: + try: + psql('CREATE EXTENSION IF NOT EXISTS :"name";\n', {"name": name}) + except Refused as e: + return f"the database could not make the extension '{name}' the node lists in `extensions`: {e}" + wait_until("the pooler to take the sign-ins", pooler_not_told) + return None + + +def reset_password() -> str | None: + """Mint a new password, set it on the program's role, tell the pooler, + make it readable again, and tell weft: what went wrong, or None. + + The database is told first, over its unix socket, which trusts a + local connection without a password: a file that named a password + the database did not have would lock every sign-in out. All of it + happens under one lock, weft's part included, so of two presses the + one that changed the database last is the one weft holds.""" + password = new_password() + with CHANGING: + # What the role holds now, exactly: an undo gives it this verifier + # back as it is (Postgres keeps a SCRAM verifier it is handed), not + # the old password hashed again under a new salt, which the pooler, + # holding this one, could no longer sign in with. + held = app_verifier() + if held is None: + return f"the role {APP_USER} has no password in the database, so the reset has nothing to replace; restart the database's infra" + try: + set_app_password(password) + except Refused as e: + sys.stderr.write(f"credential: reset refused by the database: {e}\n") + return f"the database refused the new password: {e}" + try: + write_password(password) + except OSError as e: + # A write that failed after the file was swapped in (its + # directory not synced) still left the new password there. + try: + written = stored_password() == password + except OSError: + written = False + if not written: + sys.stderr.write(f"credential: reset, but the password could not be written: {e}\n") + try: + set_app_password(held) + except Refused as undo: + return ( + f"the database took a new password that could not be written to its disk ({e}), and would not " + f"take the old one back ({undo}), so no client can sign in. Free some room on the database's " + "disk (or whatever the error says) and press Reset password again, or restart the database's infra." + ) + return ( + f"the new password could not be written to the database's disk ({e}), so the reset was undone and " + "the old password still signs in. Free some room on that disk (or whatever the error says) and press Reset password again." + ) + sys.stderr.write(f"credential: the new password is written, but its directory could not be synced: {e}\n") + try: + tell_the_pooler() + except (Refused, OSError) as e: + sys.stderr.write(f"credential: reset, but the pooler was not told: {e}\n") + return ( + f"the database has the new password, but its pooler could not be told ({e}), so no client can " + "sign in until it is. Press Reset password again." + ) + try: + os.remove(SEALED_FILE) + except FileNotFoundError: + pass + except OSError as e: + sys.stderr.write(f"credential: reset, but the password stays sealed: {e}\n") + return ( + f"the database and its pooler have a new password, but it could not be made readable ({e}), so " + "neither the card nor a run can take it. Press Reset password again." + ) + # The connection the node published still holds the old password + # until weft is told the new one. + unheard = push_password(password) + if unheard is not None: + return ( + f"the database has a new password, but weft could not be told ({unheard}), so the " + "program still signs in with the old one. Run `weft infra rebake` on this node: " + "its run reads the new password from the database." + ) sys.stderr.write("credential: password reset; readable again until a run stores it\n") - return None, password + return None def push_password(password: str) -> str | None: @@ -199,7 +577,13 @@ def _send(self, status: int, body: dict) -> None: def do_GET(self) -> None: # noqa: N802 (the stdlib's spelling) if self.path == HEALTH_PATH: - self._send(200, {"ok": True}) + if CONVERGED.is_set(): + # The role the program signs in as: how the node tells + # this unit from one an older weft set up. + # SYNC: signsInAs <-> catalog/postgres/database/mod.rs (signs_in_as_app) + self._send(200, {"ok": True, "signsInAs": APP_USER}) + else: + self._send(503, {"error": NOT_READY["why"]}) return if self.path == LIVE_PATH: self._send(200, {"items": live_items()}) @@ -255,8 +639,12 @@ def do_POST(self) -> None: # noqa: N802 self._send(200, {"stored": False}) return # Idempotent: confirming twice is the same as confirming once. - with open(SEALED_FILE, "w", encoding="utf-8") as f: - f.write("stored") + try: + with open(SEALED_FILE, "w", encoding="utf-8") as f: + f.write("stored") + except OSError as e: + self._send(503, {"error": f"the password could not be retired, its disk refused the write: {e}"}) + return self._send(200, {"stored": True}) def _action(self, body: dict) -> None: @@ -268,19 +656,12 @@ def _action(self, body: dict) -> None: if action != RESET_ACTION: self._send(200, {"result": {"error": f"no such action: {action!r}"}}) return - refused, password = reset_password() - if refused is not None: - self._send(200, {"result": {"error": f"the database refused the new password: {refused}"}}) + if not CONVERGED.is_set(): + self._send(200, {"result": {"error": "the database is still being set up for the program; press it again in a moment"}}) return - # The connection the node published still holds the old password - # until weft is told the new one. - unheard = push_password(password) - if unheard is not None: - self._send(200, {"result": {"error": ( - f"the database has a new password, but weft could not be told ({unheard}), so the " - "program still signs in with the old one. Run `weft infra rebake` on this node: " - "its run reads the new password from the database." - )}}) + refused = reset_password() + if refused is not None: + self._send(200, {"result": {"error": refused}}) return self._send(200, {"result": {"reset": True}}) @@ -290,9 +671,9 @@ def log_message(self, fmt: str, *args: object) -> None: def live_items() -> list: - """What the node's card shows: the password itself while it is still - readable, or the state it has moved on to, plus the one button that - makes it readable again. + """What the node's card shows: the names to sign in with, the password + itself while it is still readable or the state it has moved on to, with + the one button that makes it readable again, then who is connected. The card is a legitimate first taker. The password is readable exactly once, by whoever asks first, and then sealed; a person @@ -326,30 +707,107 @@ def live_items() -> list: "data": password, } # The database and user a client signs in with, beside the - # password. Postgres answers under other database names too (its own - # `postgres` among them), so a client pointed at a guess connects - # fine and then never sees the program's tables. + # password. The pooler lets clients into this database only, under + # this name. names = [ - {"type": "text", "label": "Database", "data": DATABASE}, - {"type": "text", "label": "User", "data": ADMIN_USER}, + {"type": "text", "label": "Database", "data": POOLER_DATABASE}, + {"type": "text", "label": "User", "data": APP_USER}, ] item["action"] = { "label": "Reset password", "actionKind": RESET_ACTION, "confirm": ( - "Give the database a new password? Every connection holding the " - "old one stops working. The program's own next run picks the new " - "one up by itself; anything outside it (a frontend, a psql " - "session) takes it from this card." + "Give the database a new password? A client already connected stays " + "connected; every new sign-in with the old one is refused. The program's " + "own next run picks the new one up by itself; anything outside it (a " + "frontend, a psql session) takes it from this card." ), } - return names + [item] + warning = [{"type": "text", "label": "Warning", "data": POSTGRES_NOTE}] if POSTGRES_NOTE else [] + return warning + names + [item] + connection_items() + + +def connection_items() -> list: + """Who is connected, read now: what Postgres serves, what the pooler + does with its connections, and the clients behind it. Each part says + why it could not be read, in its own place, rather than taking the + rest down.""" + items = [] + try: + limits = rows(psql( + "SELECT current_setting('max_connections') AS most, " + "current_setting('superuser_reserved_connections') AS reserved;\n", + {}, + database="postgres", + ))[0] + activity = rows(psql( + "SELECT usename AS user, coalesce(state, 'starting') AS state, count(*) AS n FROM pg_stat_activity " + "WHERE backend_type = 'client backend' AND pid <> pg_backend_pid() GROUP BY 1, 2 ORDER BY 1, 2;\n", + {}, + database="postgres", + )) + items.append({"type": "text", "label": "Postgres connections", "data": postgres_connections(limits, activity)}) + except Refused as e: + items.append({"type": "text", "label": "Postgres connections", "data": f"could not be read: {e}"}) + try: + databases = rows(psql("SHOW DATABASES;\n", {}, pooler=True)) + pools = rows(psql("SHOW POOLS;\n", {}, pooler=True)) + clients = rows(psql("SHOW CLIENTS;\n", {}, pooler=True)) + items.append({"type": "text", "label": "Pooler", "data": pooler_lanes(databases, pools)}) + items.append({"type": "text", "label": "Clients", "data": pooler_clients(clients)}) + except Refused as e: + items.append({"type": "text", "label": "Pooler", "data": f"could not be read: {e}"}) + return items + + +def postgres_connections(limits: dict, activity: list[dict]) -> str: + """Postgres's own count: in use of the most it takes, by user and state.""" + in_use = sum(int(row["n"]) for row in activity) + lines = [f"{in_use} of {limits['most']} in use ({limits['reserved']} of them kept for the superuser)"] + for user in sorted({row["user"] for row in activity}): + parts = [f"{row['n']} {row['state']}" for row in activity if row["user"] == user] + lines.append(f"{user}: {', '.join(parts)}") + return "\n".join(lines) + + +def pooler_lanes(databases: list[dict], pools: list[dict]) -> str: + """The pooler's two doors: clients served and waiting, connections open of its share.""" + lines = [] + for name, lane in ((POOLER_DATABASE, "queries"), (WATCH_DATABASE, "watchers")): + db = next((row for row in databases if row.get("name") == name), {}) + pool = next((row for row in pools if row.get("database") == name and row.get("user") == APP_USER), {}) + served, waiting = int(pool.get("cl_active", 0)), int(pool.get("cl_waiting", 0)) + line = f"{lane} ({name}): {served} client(s) served, {waiting} waiting" + if waiting: + line += f" (the longest for {pool.get('maxwait', '0')}s)" + line += f"; {db.get('current_connections', 0)} of {db.get('pool_size', '?')} connections open" + lines.append(line) + return "\n".join(lines) + + +def pooler_clients(clients: list[dict]) -> str: + """The pooler's clients by where they come from, most first: which + workers and frontends hold or wait for this database's connections.""" + counts: dict[str, int] = {} + for client in clients: + if client.get("database") not in (POOLER_DATABASE, WATCH_DATABASE): + continue + name = client.get("application_name") or "" + where = client.get("addr", "?") + (f" ({name})" if name else "") + counts[where] = counts.get(where, 0) + 1 + if not counts: + return "none" + return "\n".join(f"{where}: {n}" for where, n in sorted(counts.items(), key=lambda kv: (-kv[1], kv[0]))) def serve() -> None: server = http.server.ThreadingHTTPServer(("0.0.0.0", PORT), Handler) server.daemon_threads = True - server.serve_forever() + # Answering from the start, so the card and the readiness check hear + # "still being set up" rather than nothing while `converge` runs. + threading.Thread(target=server.serve_forever, daemon=True).start() + converge() + threading.Event().wait() if __name__ == "__main__": diff --git a/catalog/postgres/database/images/credential/test_bootstrap.py b/catalog/postgres/database/images/credential/test_bootstrap.py new file mode 100644 index 000000000..26de19196 --- /dev/null +++ b/catalog/postgres/database/images/credential/test_bootstrap.py @@ -0,0 +1,208 @@ +"""The credential container's pure parts: what it reads, what it writes +for the pooler, and what the card shows. Run with the standard library +alone (`python3 -m unittest`), so it needs no install.""" + +import os +import sys +import unittest + +# The settings the node hands the container, so the module imports here +# as it does in the unit. +os.environ.update({ + "WEFT_SECRET_DIR": "/store/secret", "WEFT_PASSWORD_FILE": "/store/secret/password", "WEFT_CREDENTIAL_PORT": "8099", + "WEFT_SOCKET_DIR": "/var/run/postgresql", "WEFT_POSTGRES_PORT": "5433", "WEFT_ADMIN_USER": "weft", + "WEFT_APP_USER": "app", "WEFT_DATABASE": "my shop", "WEFT_POOLER_DATABASE": "shop", "WEFT_WATCH_DATABASE": "shop_watch", + "WEFT_EXTENSIONS": '["pg_trgm", "nope"]', "WEFT_POSTGRES_MAJOR": "17", "WEFT_CLOSES_IDLE_SESSIONS_FROM": "14", + "WEFT_POSTGRES_NOTE": "", "WEFT_DATA_DIR": "/store/pgdata", + "WEFT_IDLE_SESSION_SECONDS": "60", "WEFT_POOLER_DIR": "/run/pooler", "WEFT_POOLER_PORT": "5432", + "WEFT_POOLER_USERS_FILE": "/run/pooler/users", "WEFT_VALUES_URL": "http://agent/values", +}) +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +import bootstrap # noqa: E402 + +# What Postgres 17 stored for the password `pw` (`pg_authid.rolpassword`). +STORED = ( + "SCRAM-SHA-256$4096:8pKmXNvoZw8v9qPgnGSNpg==$fKVddU1mC39C81y4v0D/H8hDtGTvdFJ1nMIQ+xRN6dY=" + ":oM1QsL/ZwfZ+Jt23YpdHrl8YTrU3x6hDncF3Ff/1p00=" +) + + +class ScramTest(unittest.TestCase): + def test_a_verifier_accepts_its_own_password_only(self): + self.assertTrue(bootstrap.scram_accepts(STORED, "pw")) + self.assertFalse(bootstrap.scram_accepts(STORED, "pW")) + + def test_no_verifier_or_another_kind_or_a_broken_one_is_a_no(self): + for stored in (None, "", "md5" + "0" * 32, "SCRAM-SHA-256$broken", "SCRAM-SHA-256$x:!!$y:z"): + self.assertFalse(bootstrap.scram_accepts(stored, "pw"), stored) + + +class PoolerUsersTest(unittest.TestCase): + def test_the_program_signs_in_with_its_verifier_and_the_superuser_with_none(self): + self.assertEqual(bootstrap.pooler_users(STORED), f'"app" "{STORED}"\n"weft" ""\n') + + +class RowsTest(unittest.TestCase): + def test_csv_reads_as_one_dict_per_row(self): + printed = 'user,state,n\napp,active,3\n"we,ft",idle,1\n' + self.assertEqual( + bootstrap.rows(printed), + [{"user": "app", "state": "active", "n": "3"}, {"user": "we,ft", "state": "idle", "n": "1"}], + ) + self.assertEqual(bootstrap.rows(""), []) + # A role with no password, as psql prints it. + self.assertEqual(bootstrap.rows("rolpassword\n\n"), [{"rolpassword": ""}]) + + +class CardTest(unittest.TestCase): + def test_postgres_counts_every_connection_by_user_and_state(self): + shown = bootstrap.postgres_connections( + {"most": "23", "reserved": "3"}, + [{"user": "app", "state": "active", "n": "4"}, {"user": "app", "state": "idle", "n": "9"}, + {"user": "weft", "state": "active", "n": "1"}], + ) + self.assertEqual(shown, "14 of 23 in use (3 of them kept for the superuser)\napp: 4 active, 9 idle\nweft: 1 active") + + def test_the_pooler_shows_each_door_with_who_waits(self): + databases = [ + {"name": "shop", "pool_size": "15", "current_connections": "15"}, + {"name": "shop_watch", "pool_size": "4", "current_connections": "1"}, + {"name": "pgbouncer", "pool_size": "2", "current_connections": "0"}, + ] + pools = [ + {"database": "shop", "user": "app", "cl_active": "15", "cl_waiting": "6", "maxwait": "3"}, + {"database": "shop_watch", "user": "app", "cl_active": "1", "cl_waiting": "0", "maxwait": "0"}, + ] + self.assertEqual( + bootstrap.pooler_lanes(databases, pools), + "queries (shop): 15 client(s) served, 6 waiting (the longest for 3s); 15 of 15 connections open\n" + "watchers (shop_watch): 1 client(s) served, 0 waiting; 1 of 4 connections open", + ) + + def test_a_door_nobody_used_yet_reads_as_empty(self): + shown = bootstrap.pooler_lanes([{"name": "shop", "pool_size": "15", "current_connections": "0"}], []) + self.assertTrue(shown.startswith("queries (shop): 0 client(s) served, 0 waiting; 0 of 15 connections open"), shown) + + def test_clients_are_counted_by_where_they_come_from_and_the_console_is_left_out(self): + clients = [ + {"database": "shop", "addr": "10.0.0.9", "application_name": ""}, + {"database": "shop", "addr": "10.0.0.5", "application_name": ""}, + {"database": "shop_watch", "addr": "10.0.0.5", "application_name": ""}, + {"database": "shop", "addr": "10.0.0.7", "application_name": "web"}, + {"database": "pgbouncer", "addr": "unix", "application_name": "psql"}, + ] + self.assertEqual(bootstrap.pooler_clients(clients), "10.0.0.5: 2\n10.0.0.7 (web): 1\n10.0.0.9: 1") + self.assertEqual(bootstrap.pooler_clients([]), "none") + + +class ConvergeTest(unittest.TestCase): + """One pass of `converge` against a database that answers as a fake: + every script it sends is kept, and the extension `nope` is refused.""" + + def setUp(self): + self.sent = [] + self.kept = {name: getattr(bootstrap, name) for name in ("psql", "wait_until", "stored_password", "app_verifier")} + + def psql(script, variables, database=None, pooler=False): + self.sent.append((script, dict(variables))) + if variables.get("name") == "nope": + raise bootstrap.Refused('ERROR: extension "nope" is not available') + return "" + + bootstrap.psql = psql + bootstrap.wait_until = lambda what, not_yet: None + bootstrap.stored_password = lambda: "pw" + bootstrap.app_verifier = lambda: STORED + + def tearDown(self): + for name, value in self.kept.items(): + setattr(bootstrap, name, value) + + def test_each_listed_extension_is_made_and_one_postgres_lacks_is_refused_by_name(self): + refused = bootstrap.converge_once() + made = [variables["name"] for script, variables in self.sent if "CREATE EXTENSION IF NOT EXISTS" in script] + self.assertEqual(made, ["pg_trgm", "nope"]) + self.assertEqual(self.sent[0][1]["database"], "my shop", "the role and its ownership are set on the database's own name") + self.assertIn("'nope'", refused) + self.assertIn("is not available", refused) + + def test_a_pass_the_database_takes_answers_nothing(self): + bootstrap.EXTENSIONS, kept = ["pg_trgm"], bootstrap.EXTENSIONS + try: + self.assertIsNone(bootstrap.converge_once()) + finally: + bootstrap.EXTENSIONS = kept + + +class OldPostgresTest(unittest.TestCase): + def test_a_new_database_is_never_made_on_a_postgres_that_keeps_idle_sessions(self): + self.assertIsNone(bootstrap.too_old_for_a_new_database(True, False)) + self.assertIsNone(bootstrap.too_old_for_a_new_database(True, True)) + self.assertIsNone(bootstrap.too_old_for_a_new_database(False, True), "an existing database keeps running") + self.assertIn("new database", bootstrap.too_old_for_a_new_database(False, False)) + + def test_the_idle_limit_is_set_only_where_postgres_has_it(self): + sent = [] + kept = {name: getattr(bootstrap, name) for name in ("psql", "wait_until", "stored_password", "app_verifier", "CLOSES_IDLE_SESSIONS", "EXTENSIONS")} + bootstrap.psql = lambda script, variables, database=None, pooler=False: sent.append(script) or "" + bootstrap.wait_until = lambda what, not_yet: None + bootstrap.stored_password = lambda: "pw" + bootstrap.app_verifier = lambda: STORED + bootstrap.EXTENSIONS = [] + try: + for closes, expected in ((True, True), (False, False)): + sent.clear() + bootstrap.CLOSES_IDLE_SESSIONS = closes + self.assertIsNone(bootstrap.converge_once()) + self.assertEqual(any("idle_session_timeout" in script for script in sent), expected) + finally: + for name, value in kept.items(): + setattr(bootstrap, name, value) + + def test_the_card_warns_of_an_old_postgres_first(self): + kept = {name: getattr(bootstrap, name) for name in ("POSTGRES_NOTE", "connection_items", "stored_password")} + bootstrap.POSTGRES_NOTE = "Postgres 13 does not close a connection a client left idle" + bootstrap.connection_items = lambda: [] + bootstrap.stored_password = lambda: None + try: + items = bootstrap.live_items() + self.assertEqual(items[0], {"type": "text", "label": "Warning", "data": bootstrap.POSTGRES_NOTE}) + bootstrap.POSTGRES_NOTE = "" + self.assertNotEqual(bootstrap.live_items()[0]["label"], "Warning") + finally: + for name, value in kept.items(): + setattr(bootstrap, name, value) + + +class ResetUndoTest(unittest.TestCase): + """A reset whose password cannot be written gives the role back the + exact verifier it held, which the pooler still signs in with.""" + + def setUp(self): + self.names = ("app_verifier", "set_app_password", "write_password", "stored_password", "tell_the_pooler", "push_password") + self.kept = {name: getattr(bootstrap, name) for name in self.names} + self.set = [] + bootstrap.app_verifier = lambda: STORED + bootstrap.set_app_password = lambda password: self.set.append(password) + bootstrap.stored_password = lambda: "pw" + + def full(password): + raise OSError(28, "No space left on device") + + bootstrap.write_password = full + bootstrap.tell_the_pooler = lambda: self.fail("the pooler is not told of a password that was undone") + bootstrap.push_password = lambda password: self.fail("weft is not told of a password that was undone") + + def tearDown(self): + for name, value in self.kept.items(): + setattr(bootstrap, name, value) + + def test_the_old_verifier_comes_back_as_it_was(self): + refused = bootstrap.reset_password() + self.assertEqual(len(self.set), 2) + self.assertEqual(self.set[1], STORED, "the verifier itself, never the password hashed again") + self.assertIn("undone", refused) + + +if __name__ == "__main__": + unittest.main() diff --git a/catalog/postgres/database/images/pooler/Dockerfile b/catalog/postgres/database/images/pooler/Dockerfile new file mode 100644 index 000000000..d804fc569 --- /dev/null +++ b/catalog/postgres/database/images/pooler/Dockerfile @@ -0,0 +1,23 @@ +# PgBouncer, the one door every client goes through to reach the +# database. Its settings come from the node that declares this +# container, written out by the entrypoint; nothing is baked here. +FROM alpine:3.24 + +# The release the node's settings were written and checked against. +# Alpine's package decides the version, so a base that brought an older +# one stops the build here rather than starting a pooler that reads the +# settings differently. +RUN apk add --no-cache pgbouncer \ + && version="$(pgbouncer --version | head -n 1 | cut -d ' ' -f 2)" \ + && major="${version%%.*}" && rest="${version#*.}" && minor="${rest%%.*}" \ + && if [ "$major" -lt 1 ] || { [ "$major" -eq 1 ] && [ "$minor" -lt 25 ]; }; then \ + echo "pgbouncer $version is older than 1.25, the release this pooler is set up for" >&2; exit 1; \ + fi + +COPY entrypoint.sh /entrypoint.sh + +# Not root (PgBouncer refuses to run as root), and the same user as the +# credential container: both only need the infra unit's shared group to +# reach the directory they share. +USER 999:999 +ENTRYPOINT ["/bin/sh", "/entrypoint.sh"] diff --git a/catalog/postgres/database/images/pooler/entrypoint.sh b/catalog/postgres/database/images/pooler/entrypoint.sh new file mode 100644 index 000000000..21b12020c --- /dev/null +++ b/catalog/postgres/database/images/pooler/entrypoint.sh @@ -0,0 +1,52 @@ +# Start PgBouncer with the settings the node wrote. +# +# Every setting, and every path, comes from the node that declares this +# container, so this script and the node cannot disagree on any of them. +# The settings and the sign-in rules arrive as the arguments, one line +# each: `--config --hba `. +# SYNC: the pooler's settings <-> catalog/postgres/database/mod.rs (pooler_env, pooler_args) +set -eu + +: "${WEFT_POOLER_CONFIG_FILE:?set by the node}" "${WEFT_POOLER_HBA_FILE:?set by the node}" +: "${WEFT_POOLER_USERS_FILE:?set by the node}" "${WEFT_POOLER_OPEN_FILES:?set by the node}" + +if [ "${1:-}" != "--config" ]; then + echo "pooler: expected the arguments --config --hba from the node" >&2 + exit 1 +fi +shift +: > "$WEFT_POOLER_CONFIG_FILE" +: > "$WEFT_POOLER_HBA_FILE" +into="$WEFT_POOLER_CONFIG_FILE" +for line in "$@"; do + if [ "$line" = "--hba" ]; then + into="$WEFT_POOLER_HBA_FILE" + continue + fi + printf '%s\n' "$line" >> "$into" +done +if [ "$into" != "$WEFT_POOLER_HBA_FILE" ]; then + echo "pooler: no --hba among the arguments, so nobody could sign in" >&2 + exit 1 +fi + +# Every client and every connection to Postgres is an open file, and a +# container often starts allowed far fewer than it may raise itself to. +# A machine that does not allow enough is said here, at the start, rather +# than as clients refused under load. +ulimit -n "$(ulimit -Hn)" +allowed="$(ulimit -n)" +if [ "$allowed" != "unlimited" ] && [ "$allowed" -lt "$WEFT_POOLER_OPEN_FILES" ]; then + echo "pooler: this machine lets the pooler open $allowed files, fewer than the $WEFT_POOLER_OPEN_FILES its clients need" >&2 + exit 1 +fi + +# The credential container writes who may sign in once the database has +# the program's role (`bootstrap.py`, `converge`). Started before that, +# PgBouncer would refuse every client as unknown. +if [ ! -s "$WEFT_POOLER_USERS_FILE" ]; then + echo "pooler: waiting for the database to be set up before taking clients" >&2 + until [ -s "$WEFT_POOLER_USERS_FILE" ]; do sleep 1; done +fi + +exec pgbouncer "$WEFT_POOLER_CONFIG_FILE" diff --git a/catalog/postgres/database/metadata.json b/catalog/postgres/database/metadata.json index f8aed7b17..d11b28976 100644 --- a/catalog/postgres/database/metadata.json +++ b/catalog/postgres/database/metadata.json @@ -1,7 +1,7 @@ { "type": "PostgresDatabase", "label": "Postgres: Database", - "description": "Run a Postgres database inside the project, with a disk that survives restarts; wire its access output into the query and write nodes. Its generated password is handed out exactly ONCE, to whoever asks first: until then the node's card shows it masked with a copy button (how an outside client, such as a frontend with its own tables, gets in); afterwards the card's Reset password button mints and shows a new one. `weft infra env --into .env --set DATABASE_USER=User --set DATABASE_PASSWORD=Password` writes both into an env file without anyone reading the password; once handed over, `weft infra press reset_password` then `weft infra env` writes a fresh one (`weft infra show ` lists the buttons). Terminating its infra deletes the disk and all data.", + "description": "Run a Postgres database inside the project, with a disk that survives restarts; wire its access output into the query and write nodes. Its generated password is handed out exactly ONCE, to whoever asks first: until then the node's card shows it masked with a copy button (how an outside client, such as a frontend with its own tables, gets in); afterwards the card's Reset password button mints and shows a new one. `weft infra env --into .env --set DATABASE_USER=User --set DATABASE_PASSWORD=Password` writes both into an env file without anyone reading the password; once handed over, `weft infra press reset_password` then `weft infra env` writes a fresh one (`weft infra show ` lists the buttons). Every client (the program's workers, a frontend) signs in as `app`, which owns the database and is no superuser, through the node's pooler: Postgres serves a fixed number of connections (see `memory`), and a client past them waits its turn instead of being refused, however many workers run. A connection that sits idle for a minute is closed by the database, so an outside client's own pool must expect an idle connection to be gone (most close theirs sooner). Watching queries (PostgresWatchQuery) have their own share of the connections, a quarter of them. Through the pooler a client reaches this one database only, never Postgres's own `postgres` database or another one made beside it. An extension the program needs, trusted by Postgres or not, goes in `extensions`: the superuser `weft` makes it on every start, and `app` uses it. The superuser signs in only over the unix socket inside the database's own container. Postgres note: a function called in a `cross join lateral (select f(x) as v)` subquery and read several times through `v` can run once per read, because Postgres folds the subquery into the outer query; `offset 0` inside the subquery keeps it to one call. Terminating its infra deletes the disk and all data.", "tags": [ "postgres", "database", @@ -15,7 +15,8 @@ "liveEndpoint": "credential" }, "images": [ - "images/credential" + "images/credential", + "images/pooler" ], "publishes": "postgres", "inputs": [ @@ -24,7 +25,7 @@ "type": "String", "default": "app", "label": "Database name", - "description": "Database created on first start. Changing it later creates nothing, and the connection then names a database that does not exist." + "description": "Database created on first start, any name. Changing it later creates nothing, and the connection then names a database that does not exist. A client signs in by the name the card shows under Database: this one when it is only letters, digits, `_` and `-`, at most 57 characters and not `pgbouncer`, otherwise (or when it is `all`, `sameuser`, `samerole`, `samegroup` or `replication`) one the pooler makes from it (`db_my_app_1a2b3c4d`), which reaches the database under its own name." }, { "name": "storage", @@ -38,7 +39,14 @@ "type": "String", "default": "17", "label": "Postgres version", - "description": "Postgres major version. Changing it on an existing database does not migrate: Postgres refuses a data directory from another major version." + "description": "Postgres major version. A new database needs 14 or later; an existing one on an older Postgres keeps running, without closing connections a client left idle, and its card and `weft infra status` say how to move it. Changing it on an existing database does not migrate: Postgres refuses a data directory from another major version." + }, + { + "name": "extensions", + "type": "List[String]", + "default": [], + "label": "Extensions", + "description": "Extensions the program's database needs, as Postgres names them (`[\"pg_trgm\", \"postgres_fdw\"]`). On every start the superuser makes each one that is not there yet, so the program, signed in as `app`, uses them, including those Postgres does not let a non-superuser make. One this Postgres does not have fails the start, and `weft infra status` names it. Taking one off the list leaves it in the database. Changing the list restarts the database." }, { "name": "reachable", @@ -59,7 +67,7 @@ "type": "String", "default": "1Gi", "label": "Memory", - "description": "Memory, a number followed by Gi, Mi, G or M (`1Gi`). It takes at least 886Mi, the least that holds the 20 connections one worker of the program opens. Sized with `cpu`. Postgres is set up for it: 320Mi stays with the machine and the rest is Postgres's, so at 1Gi it takes 24 connections, at 2Gi 60, at 4Gi 132 (200 at most). Each worker of the program opens up to 20, so a second worker or an outside client (a frontend) beside one wants 2Gi. If the database is killed for running out of memory, or refuses a connection with `too many clients`, raise `memory`. A local install uses the same settings and caps the container the same way." + "description": "Memory, a number followed by Gi, Mi, G or M (`1Gi`). It takes at least 565Mi. Sized with `cpu`. Postgres is set up for it: 352Mi stays with the machine (the pooler and the helper beside Postgres included) and the rest is Postgres's, so at 1Gi it serves 23 connections, at 2Gi 59, at 4Gi 131 (200 at most). Three of them are kept for the superuser and two spare; of the rest a quarter is for watching queries and the others for every other query. When every connection is busy, a client waits for one rather than being refused. If runs often wait for the database, or it is killed for running out of memory, raise `memory`. A local install uses the same settings and caps the container the same way." }, { "name": "machineType", @@ -78,6 +86,66 @@ } ], "validate": [ + { + "when": { + "kind": "all", + "of": [ + { + "kind": "config_present", + "field": "database" + }, + { + "kind": "config_matches", + "field": "database", + "regex": "^$|\\p{Cc}" + } + ] + }, + "then": { + "message": "PostgresDatabase '{id}' has a `database` name that is empty or holds a line break or another control character; name it without (app).", + "level": "structural", + "severity": "error", + "port": "database" + } + }, + { + "when": { + "kind": "all", + "of": [ + { + "kind": "config_present", + "field": "version" + }, + { + "kind": "not", + "of": { + "kind": "config_matches", + "field": "version", + "regex": "^[1-9][0-9]{0,3}(\\.(0|[1-9][0-9]*))?$" + } + } + ] + }, + "then": { + "message": "PostgresDatabase '{id}' has a `version` that is not a Postgres version. Write its major version (17).", + "level": "structural", + "severity": "error", + "port": "version" + } + }, + { + "when": { + "kind": "config_matches", + "field": "version", + "regex": "^([1-9]|1[0-3])(\\.(0|[1-9][0-9]*))?$" + }, + "then": { + "message": "PostgresDatabase '{id}' runs a Postgres older than 14, which does not close connections a client left idle: an existing database keeps running without that, and a new one is refused when it starts. To move to 14 or later: Postgres cannot open an older version's data, so the data moves by dump and restore while the old database still runs. Dump it: `weft infra press {id} reset_password`, then `weft infra env {id} --into old.env --set PGUSER=User --set PGDATABASE=Database --as PGPASSWORD`, and with those exported, `pg_dump -Fc -h -p -f data.dump` at the address `weft infra list-doors` prints for it (turn `reachable` on first if it is off). Restore it: add a PostgresDatabase on 17 with `reachable` on, start it, write its env the same way, `pg_restore --no-owner -h -p -d \"$PGDATABASE\" data.dump`, and wire the new node where the old one was.", + "level": "structural", + "severity": "warning", + "port": "version" + } + }, { "when": { "kind": "all", diff --git a/catalog/postgres/database/mod.rs b/catalog/postgres/database/mod.rs index 4e8522a50..c94d99fdc 100644 --- a/catalog/postgres/database/mod.rs +++ b/catalog/postgres/database/mod.rs @@ -1,12 +1,22 @@ //! PostgresDatabase: a Postgres the project runs itself, handed out as //! an ordinary connection. //! -//! Three pieces make one unit: +//! Four pieces make one unit: //! - an init container that writes a password to the shared volume //! the first time and never again, so the spec carries no //! credential and stays identical on every start; -//! - Postgres itself, reading that password file; -//! - a small server beside it that hands the password over once. +//! - Postgres itself, reading that password file, listening only +//! inside the unit; +//! - a pooler (PgBouncer) in front of it, the one door every client +//! comes through. Workers scale, a rollout runs old and new copies +//! side by side, frontends connect too: the clients are many and +//! change, while Postgres serves a fixed number of connections. The +//! pooler holds at most that number and makes every client past it +//! wait its turn, so the database is never asked for more than it +//! serves, however many clients there are; +//! - a small server beside them that hands the password over once, +//! gives the program its own role (never the superuser), and shows +//! the card. //! //! `run` reads its own published connection when it has one, and asks //! the database for the password when it has none or when the one it @@ -20,8 +30,8 @@ //! and any later run that reads only that output uses the saved value //! without running this node, so it asks the database nothing. //! -//! The graph's `Reset password` button resets it in the database, and -//! the credential container then pushes the new password to weft +//! The graph's `Reset password` button resets it in the database and +//! tells the pooler, and the credential container then pushes the new password to weft //! (`weft_core::infra::VALUES_URL_ENV`), which writes it into the //! connection this node published: no run of this node is involved. //! @@ -45,9 +55,14 @@ use weft::{ WeftResult, }; -use super::postgres::{mistake, MOST_AT_ONCE}; +use super::postgres::{mistake, IDLE_CLOSE}; +/// The port clients connect to: the pooler's. Postgres's own default, so +/// a connection string reads like any other Postgres one. const SQL_PORT: u16 = 5432; +/// Postgres's own port, on the unit's loopback only: nothing but the +/// pooler and the unit's own checks ever dials it. +const POSTGRES_PORT: u16 = 5433; const CREDENTIAL_PORT: u16 = 8099; /// The one disk and its two halves, ordinary subdirectories rather /// than separate volumes, so a restore can never bring back one @@ -58,8 +73,58 @@ const SECRET_PATH: &str = "/store/secret"; /// Where the Postgres image puts its unix socket; the credential /// container mounts the same directory. const SOCKET_PATH: &str = "/var/run/postgresql"; -/// The role Postgres creates on first boot, and the only one weft uses. +/// The role Postgres creates on first boot: the superuser. It signs in +/// only over the unit's unix socket (the card, a reset, a person +/// operating the database), never through the pooler, so the slots +/// Postgres keeps for a superuser are always there for it. const ADMIN_USER: &str = "weft"; +/// The role the program and every other client sign in as, through the +/// pooler. It owns the program's database and is no superuser. +const APP_USER: &str = "app"; +/// How long a session of [`APP_USER`] may sit idle before Postgres closes +/// it, in seconds. This is what frees a connection nobody is using +/// whoever holds it: a worker that lost its CPU between requests, a copy +/// left behind by a rollout, a frontend's pool. It runs inside Postgres, +/// which always has a CPU. A client that means to sit idle (a watching +/// query) says so for its own session. +const IDLE_SESSION_SECS: u64 = 60; +// A worker hands out only connections idle for less than `IDLE_CLOSE`, +// so one the database closed for idling is never handed to a run. +const _: () = assert!(IDLE_CLOSE.as_secs() * 4 <= IDLE_SESSION_SECS); +/// The directory the pooler shares with the credential container: the +/// pooler's unix socket (its admin console, for the card and for telling +/// it a new password) and the file of who may sign in. Scratch space, +/// written afresh on every start. +const POOLER_PATH: &str = "/run/pooler"; +/// Where the pooler's entrypoint writes the settings this node hands it. +/// Inside the pooler's own container, so nothing else reads them. +const POOLER_CONFIG_FILE: &str = "/tmp/pgbouncer.ini"; +const POOLER_HBA_FILE: &str = "/tmp/pgbouncer.hba"; +/// How many clients the pooler takes at once, waiting or served: twenty +/// for each of a hundred workers. Measured on PgBouncer 1.25, two +/// thousand clients waiting to sign in take about 5MB. +const MOST_CLIENTS: u64 = 2000; +/// The pooler's memory limit, in MiB, ample for [`MOST_CLIENTS`]. +const POOLER_MIB: u64 = 32; +/// How long the pooler keeps a connection to Postgres that no client +/// holds, in seconds: under [`IDLE_SESSION_SECS`], so it closes its own +/// rather than finding them closed by Postgres. +const POOLER_SERVER_IDLE_SECS: u64 = 30; +/// The connections Postgres keeps for the superuser +/// (`superuser_reserved_connections`, Postgres's own default, set so it +/// is visible): the operator's way in on a database every client is +/// waiting for. +const SUPERUSER_RESERVED: u64 = 3; +/// Connections left out of the pooler's on top of those: a connection +/// the pooler just closed still counts in Postgres for the moment its +/// process takes to exit, and a new one opened in that moment must not +/// find Postgres full. +const POOLER_MARGIN: u64 = 2; +/// The share of the pooler's connections kept for watching queries, +/// which each hold one for as long as their run watches: a quarter, at +/// least one. Kept apart so watchers can never take every connection +/// from queries, nor queries from watchers. +const WATCH_SHARE: u64 = 4; /// The group the shared disk belongs to, so the container that mints /// the password can write it and Postgres, running as its own user, /// can read it without the file being readable to everything. 70 is @@ -102,20 +167,156 @@ fn secret_mount() -> Mount { } /// Where the credential container reads its settings from, so the -/// paths and the port are declared once, here, and the container -/// cannot drift from them. The database and user names are what its -/// card shows beside the password: the three a client signs in with. -fn credential_env(database: &str) -> Vec { +/// paths, the ports and the names are declared once, here, and the +/// container cannot drift from them. The pooler's name for the database +/// and the program's user are what its card shows beside the password: +/// the three a client signs in with. The extensions are the ones it makes +/// on every start, as JSON, since a container's settings hold no line +/// break. +// SYNC: credential_env <-> catalog/postgres/database/images/credential/bootstrap.py (the container's settings) +fn credential_env(database: &str, extensions: &[String], version: &PostgresVersion) -> Vec { vec![ + EnvEntry::new("WEFT_POSTGRES_MAJOR", version.major.to_string()), + EnvEntry::new("WEFT_CLOSES_IDLE_SESSIONS_FROM", PostgresVersion::CLOSES_IDLE_SESSIONS.to_string()), + // Empty on a Postgres that closes idle sessions. + EnvEntry::new("WEFT_POSTGRES_NOTE", version.note().unwrap_or_default()), + EnvEntry::new("WEFT_DATA_DIR", DATA_PATH), EnvEntry::new("WEFT_DATABASE", database), + EnvEntry::new("WEFT_POOLER_DATABASE", pooler_database(database)), + EnvEntry::new("WEFT_WATCH_DATABASE", watch_database(database)), + EnvEntry::new("WEFT_EXTENSIONS", serde_json::to_string(extensions).expect("a list of names serializes")), EnvEntry::new("WEFT_SECRET_DIR", SECRET_PATH), EnvEntry::new("WEFT_PASSWORD_FILE", password_file()), EnvEntry::new("WEFT_CREDENTIAL_PORT", CREDENTIAL_PORT.to_string()), EnvEntry::new("WEFT_SOCKET_DIR", SOCKET_PATH), + EnvEntry::new("WEFT_POSTGRES_PORT", POSTGRES_PORT.to_string()), EnvEntry::new("WEFT_ADMIN_USER", ADMIN_USER), + EnvEntry::new("WEFT_APP_USER", APP_USER), + EnvEntry::new("WEFT_IDLE_SESSION_SECONDS", IDLE_SESSION_SECS.to_string()), + EnvEntry::new("WEFT_POOLER_DIR", POOLER_PATH), + EnvEntry::new("WEFT_POOLER_PORT", SQL_PORT.to_string()), + EnvEntry::new("WEFT_POOLER_USERS_FILE", pooler_users_file()), ] } +/// What the pooler's entrypoint reads beside its arguments +/// ([`pooler_args`]): where to write its settings and its sign-in rules, +/// the file of who may sign in that it waits for, and how many open files +/// its clients need. +// SYNC: pooler_env, pooler_args <-> catalog/postgres/database/images/pooler/entrypoint.sh +fn pooler_env(tuning: &Tuning) -> Vec { + vec![ + EnvEntry::new("WEFT_POOLER_CONFIG_FILE", POOLER_CONFIG_FILE), + EnvEntry::new("WEFT_POOLER_HBA_FILE", POOLER_HBA_FILE), + EnvEntry::new("WEFT_POOLER_USERS_FILE", pooler_users_file()), + // One per client and one per connection to Postgres, and room + // for the pooler's own (its log, its sockets, the files above). + EnvEntry::new("WEFT_POOLER_OPEN_FILES", (MOST_CLIENTS + tuning.query_pool + tuning.watch_pool + 64).to_string()), + ] +} + +/// The pooler's settings and sign-in rules, as its container's arguments, +/// one line each: `--config` and the lines of [`pooler_config`], then +/// `--hba` and the lines of [`pooler_hba`]. Arguments rather than +/// environment, which cannot carry a line break. +pub fn pooler_args(database: &str, tuning: &Tuning) -> Vec { + std::iter::once("--config".to_string()) + .chain(pooler_config(database, tuning)) + .chain(std::iter::once("--hba".to_string())) + .chain(pooler_hba(database)) + .collect() +} + +/// The name watching queries connect to `database` by: the same +/// database, through the pooler's second door with its own connections +/// ([`Tuning::watch_pool`]). Published on the connection as +/// `watchDatabase`, which the watching node reads. +pub fn watch_database(database: &str) -> String { + format!("{}_watch", pooler_database(database)) +} + +/// PgBouncer's settings. Session pooling: a client keeps the one +/// Postgres connection it was handed until it lets go, so `SET`, a +/// temporary table, an advisory lock, `LISTEN` and a prepared +/// statement all hold across one firing's statements exactly as they +/// would on a connection of its own. The pooler's part is only that it +/// opens at most [`Tuning::query_pool`] connections for queries and +/// [`Tuning::watch_pool`] for watchers, and that a client past them +/// waits, however long, for one to come free: a run waits for the +/// database the way it waits for anything else, and stops when the run +/// does. +/// +/// No password is in it: who may sign in is the file the credential +/// container writes from the database itself. +pub fn pooler_config(database: &str, tuning: &Tuning) -> Vec { + let (door, watch) = (pooler_database(database), watch_database(database)); + // The database's own name, quoted the way PgBouncer reads a value + // (a quote doubled), so any name Postgres has reaches it. + let backend = format!("host=127.0.0.1 port={POSTGRES_PORT} dbname='{}'", database.replace('\'', "''")); + [ + "[databases]".to_string(), + format!("{door} = {backend} pool_size={}", tuning.query_pool), + format!("{watch} = {backend} pool_size={}", tuning.watch_pool), + String::new(), + "[pgbouncer]".to_string(), + "listen_addr = *".to_string(), + format!("listen_port = {SQL_PORT}"), + format!("unix_socket_dir = {POOLER_PATH}"), + "auth_type = hba".to_string(), + format!("auth_hba_file = {POOLER_HBA_FILE}"), + format!("auth_file = {}", pooler_users_file()), + format!("admin_users = {ADMIN_USER}"), + "pool_mode = session".to_string(), + // What a client left on a connection goes before the next one + // gets it. + "server_reset_query = DISCARD ALL".to_string(), + format!("max_client_conn = {MOST_CLIENTS}"), + // A client waits for a connection as long as it takes. + "query_wait_timeout = 0".to_string(), + format!("server_idle_timeout = {POOLER_SERVER_IDLE_SECS}"), + // A client whose machine vanished without closing (a worker + // instance gone mid-request) is found within about a minute and + // its connection given back, rather than after the two hours a + // kernel waits by default. + "tcp_keepalive = 1".to_string(), + "tcp_keepidle = 30".to_string(), + "tcp_keepintvl = 10".to_string(), + "tcp_keepcnt = 3".to_string(), + // Sent by JDBC and other drivers on every connect, and harmless + // to leave out; anything else a client asks for at sign-in that + // the pooler cannot carry is refused, never dropped. + "ignore_startup_parameters = extra_float_digits".to_string(), + ] + .into() +} + +/// Who may sign in to the pooler, and how. The program's role, with its +/// password, to the program's database and its watching door, from +/// anywhere the endpoint lets in; the superuser only to the pooler's +/// admin console, only over the pooler's unix socket, which only the +/// credential container mounts. Nobody reaches the superuser through +/// the pooler. +pub fn pooler_hba(database: &str) -> Vec { + let (door, watch) = (pooler_database(database), watch_database(database)); + [ + format!("local pgbouncer {ADMIN_USER} trust"), + format!("host {door},{watch} {APP_USER} 0.0.0.0/0 scram-sha-256"), + format!("host {door},{watch} {APP_USER} ::/0 scram-sha-256"), + ] + .into() +} + +/// The file of who may sign in to the pooler, written by the credential +/// container from the database's own records. +fn pooler_users_file() -> String { + format!("{POOLER_PATH}/users") +} + +/// The pooler's shared directory ([`POOLER_PATH`]). +fn pooler_mount() -> Mount { + Mount::new("pooler", POOLER_PATH) +} + /// Postgres's unix socket, shared between it and the credential /// container. It is the one door into the database that needs no /// password (initdb trusts local socket connections; only TCP asks @@ -136,15 +337,15 @@ fn password_file() -> String { /// What the machine keeps for everything that is not Postgres, in MiB: /// the operating system, Docker and the agent that runs the unit (about /// 256 together on the smallest cloud machine), plus the credential -/// container's 64Mi limit. -const BESIDE_POSTGRES_MIB: u64 = 256 + 64; +/// container's 64Mi limit and the pooler's ([`POOLER_MIB`]). +const BESIDE_POSTGRES_MIB: u64 = 256 + 64 + POOLER_MIB; /// What one connection costs Postgres before it sorts or hashes /// anything, in MiB: the server process each connection gets, with its /// caches of the tables and plans it has used. const PER_CONNECTION_MIB: u64 = 8; /// The most connections Postgres takes, however big the machine. Past a -/// few hundred, Postgres slows down from managing its connections alone, -/// and the answer to more clients is a pooler in front of it. +/// few hundred, Postgres slows down from managing its connections alone; +/// more clients than that wait at the pooler instead. const MOST_CONNECTIONS: u64 = 200; /// The background cleaners (`autovacuum_max_workers`, Postgres's /// default), each of which may use `maintenance_work_mem` at once. @@ -165,6 +366,11 @@ pub struct Tuning { pub work_mem_kib: u64, pub maintenance_work_mem_mib: u64, pub effective_cache_size_mib: u64, + /// The connections the pooler opens for queries, and for watching + /// queries: together, everything Postgres serves but the + /// superuser's [`SUPERUSER_RESERVED`] and the [`POOLER_MARGIN`]. + pub query_pool: u64, + pub watch_pool: u64, } impl Tuning { @@ -189,17 +395,21 @@ impl Tuning { /// - `effective_cache_size`, only a hint to the planner of how /// much the disk cache holds: three quarters of the share. /// + /// The pooler's connections are what `max_connections` leaves, split + /// between queries and watchers ([`WATCH_SHARE`]). + /// /// `memory` is what the node asks for, which a cloud rounds UP to /// the machine it picks, so the settings never assume more memory - /// than there is. Refused when the share holds fewer connections - /// than one worker opens ([`MOST_AT_ONCE`]), because that worker's - /// own runs would then be refused by the database. + /// than there is. Refused when the share leaves the pooler no + /// connection for queries or none for watchers. pub fn for_machine(memory: &str) -> WeftResult { - let Split { share, shared_buffers_mib, maintenance_work_mem_mib, connections_mib, max_connections } = split(memory_mib(memory)?); - if max_connections < MOST_AT_ONCE as u64 { + let Split { share, shared_buffers_mib, maintenance_work_mem_mib, connections_mib, max_connections, query_pool, watch_pool } = + split(memory_mib(memory)?); + if query_pool == 0 || watch_pool == 0 { mistake!( - "memory {memory} leaves Postgres room for {max_connections} connections, fewer than \ - the {MOST_AT_ONCE} one worker of the program opens; raise `memory` to {LEAST_MEMORY_MIB}Mi or more" + "memory {memory} leaves Postgres room for {max_connections} connections, too few to keep \ + {SUPERUSER_RESERVED} for the superuser and still give queries and watchers one each; raise \ + `memory` to {LEAST_MEMORY_MIB}Mi or more" ); } let each_kib = connections_mib * 1024 / max_connections; @@ -210,6 +420,8 @@ impl Tuning { work_mem_kib: (each_kib - PER_CONNECTION_MIB * 1024) / 2, maintenance_work_mem_mib, effective_cache_size_mib: share * 3 / 4, + query_pool, + watch_pool, }) } @@ -218,6 +430,7 @@ impl Tuning { pub fn args(&self) -> Vec { [ format!("max_connections={}", self.max_connections), + format!("superuser_reserved_connections={SUPERUSER_RESERVED}"), format!("shared_buffers={}MB", self.shared_buffers_mib), format!("work_mem={}kB", self.work_mem_kib), format!("maintenance_work_mem={}MB", self.maintenance_work_mem_mib), @@ -238,6 +451,8 @@ struct Split { maintenance_work_mem_mib: u64, connections_mib: u64, max_connections: u64, + query_pool: u64, + watch_pool: u64, } const fn split(machine_mib: u64) -> Split { @@ -251,21 +466,161 @@ const fn split(machine_mib: u64) -> Split { let connections_mib = share.saturating_sub(shared_buffers_mib + CLEANERS * maintenance_work_mem_mib); let fit = connections_mib / (PER_CONNECTION_MIB + 2 * 4); let max_connections = if fit < MOST_CONNECTIONS { fit } else { MOST_CONNECTIONS }; - Split { share, shared_buffers_mib, maintenance_work_mem_mib, connections_mib, max_connections } + let pooled = max_connections.saturating_sub(SUPERUSER_RESERVED + POOLER_MARGIN); + let watch_pool = match pooled / WATCH_SHARE { + 0 if pooled >= 2 => 1, + share => share, + }; + let query_pool = pooled - watch_pool; + Split { share, shared_buffers_mib, maintenance_work_mem_mib, connections_mib, max_connections, query_pool, watch_pool } } /// The least `memory` the node takes, in MiB: the smallest machine whose -/// share holds the [`MOST_AT_ONCE`] connections one worker opens. Read -/// off [`split`], so it moves with the formula; the `memory` input's -/// description states it too, and a test holds the two together. +/// share gives the pooler a connection for queries and one for +/// watchers. Read off [`split`], so it moves with the formula; the +/// `memory` input's description states it too, and a test holds the two +/// together. pub const LEAST_MEMORY_MIB: u64 = { let mut mib = BESIDE_POSTGRES_MIB; - while split(mib).max_connections < MOST_AT_ONCE as u64 { + while split(mib).query_pool == 0 || split(mib).watch_pool == 0 { mib += 1; } mib }; +/// The `database` input as Postgres has the database: exactly as +/// written, whatever it holds, because a database made under a name is +/// found only under that name again, and a node pointing at another +/// name points at a new, empty database. Refused only when nothing +/// could carry it: empty, or with a line break or another control +/// character, which no container's settings can hold. +pub fn database_name(raw: &str) -> WeftResult { + if raw.is_empty() { + mistake!("the `database` input is empty; name the database (app)"); + } + if raw.chars().any(char::is_control) { + mistake!("database {raw:?} has a line break or another control character in it; name it without (app)"); + } + Ok(raw.to_string()) +} + +/// The longest name the pooler can let clients in by as written: its +/// watching door ([`watch_database`]) adds `_watch`, and Postgres and +/// the pooler both keep 63 characters of a name. +const MOST_DOOR_CHARS: usize = 63 - "_watch".len(); + +/// The name a client signs in to `database` by, through the pooler, and +/// so the one the connection carries. PgBouncer's settings take a +/// database's name unquoted, which holds letters, digits, `_` and `-` +/// (a space, a dot or a quote stops it reading its settings at all); +/// `pgbouncer` is its own console, and its sign-in rules read `all`, +/// `sameuser`, `samerole`, `samegroup` and `replication` as words of their +/// own ([`HBA_WORDS`]). So any other name like that, short enough for its +/// watching door, is the door itself; the rest are each given one made +/// from it, `db_`, the characters of it that fit, and 8 hex of a hash of +/// the whole name so two names never share a door, while the pooler +/// still hands every client to the database under its own name +/// ([`pooler_config`]). +pub fn pooler_database(database: &str) -> String { + let plain = database.chars().all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-'); + if plain && database.len() <= MOST_DOOR_CHARS && database != "pgbouncer" && !HBA_WORDS.iter().any(|word| word.eq_ignore_ascii_case(database)) { + return database.to_string(); + } + let kept: String = database + .chars() + .map(|c| if c.is_ascii_alphanumeric() || c == '_' || c == '-' { c } else { '_' }) + .take(40) + .collect(); + format!("db_{kept}_{:08x}", name_hash(database) as u32) +} + +/// The database names PgBouncer's sign-in rules (`auth_hba_file`, read +/// the way Postgres reads `pg_hba.conf`) take as keywords rather than as a +/// database: a rule naming `sameuser` matches only a database named after +/// the user signing in. Compared ignoring case, which costs a name like +/// `All` nothing but its own door. +const HBA_WORDS: [&str; 5] = ["all", "sameuser", "samerole", "samegroup", "replication"]; + +/// A hash of `name` that every build of weft computes the same (64-bit +/// FNV-1a): the door [`pooler_database`] makes is published on the +/// connection, so it must not move between versions. +fn name_hash(name: &str) -> u64 { + name.bytes().fold(0xcbf2_9ce4_8422_2325, |hash, byte| (hash ^ u64::from(byte)).wrapping_mul(0x0100_0000_01b3)) +} + +/// The `extensions` input: each extension the program's database needs, +/// as Postgres names it, made by the superuser on every start +/// (`bootstrap.py`, `converge`). An empty name, or one with a control +/// character, is refused here; one Postgres does not know fails the start +/// naming it. +pub fn extension_names(raw: Vec) -> WeftResult> { + for name in &raw { + if name.is_empty() || name.chars().any(char::is_control) { + mistake!("extension {name:?} in `extensions` is not a name; write each as Postgres names it (pg_trgm)"); + } + } + Ok(raw) +} + +/// How a database on a Postgres older than 14 moves to 14 or later: dump +/// and restore, while it still runs (its card answers, and when it was set +/// up with `reachable` on its address is a door on this machine). +// SYNC: MOVE_TO_14 <-> catalog/postgres/database/metadata.json (the old `version` rule's message) +const MOVE_TO_14: &str = "Postgres cannot open an older version's data, so the data moves by dump and restore while \ + the old database still runs. Dump it: `weft infra press reset_password`, then `weft infra env \ + --into old.env --set PGUSER=User --set PGDATABASE=Database --as PGPASSWORD`, and with those exported, `pg_dump \ + -Fc -h -p -f data.dump` at the address `weft infra list-doors` prints for it (turn `reachable` on \ + first if it is off). Restore it: add a PostgresDatabase on 17 with `reachable` on, start it, write its env the \ + same way, `pg_restore --no-owner -h -p -d \"$PGDATABASE\" data.dump`, and wire the new node where \ + the old one was"; + +/// The Postgres a `version` input names: the image tag, as written, and +/// its major version. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct PostgresVersion { + pub tag: String, + pub major: u32, +} + +impl PostgresVersion { + /// The first major that closes idle sessions (`idle_session_timeout`), + /// which this node sets on the program's role. A NEW database is made + /// on this or later only (the init container refuses an older one on an + /// empty disk, the one place that can tell); an existing one on an older + /// Postgres keeps running without the limit, said by [`Self::note`]. + pub const CLOSES_IDLE_SESSIONS: u32 = 14; + + /// What `weft infra status` and the card warn of a database on a + /// Postgres that does not close idle sessions, or `None` on one that does. + pub fn note(&self) -> Option { + (self.major < Self::CLOSES_IDLE_SESSIONS).then(|| { + format!( + "Postgres {} does not close a connection a client left idle, so connections an idle worker or an \ + old revision holds stay taken until the pooler's run out; a new database needs 14 or later. To \ + move to 14 or later: {MOVE_TO_14}", + self.tag + ) + }) + } +} + +/// The `version` input as a Postgres this node can run, written as an +/// image tag names one: a major version of at most four digits and at +/// most one minor (`17`, `16.4`), with no leading zero, kept as written +/// for the tag. +// SYNC: postgres_version <-> catalog/postgres/database/metadata.json (the two `version` rules) +pub fn postgres_version(raw: &str) -> WeftResult { + let number = |part: &str| !part.is_empty() && part.bytes().all(|b| b.is_ascii_digit()) && (part == "0" || !part.starts_with('0')); + let (major, minor) = raw.split_once('.').map_or((raw, None), |(major, minor)| (major, Some(minor))); + if !number(major) || major == "0" || major.len() > 4 || minor.is_some_and(|minor| !number(minor)) { + mistake!("version '{raw}' is not a Postgres version; write its major version (17)"); + } + match major.parse::() { + Ok(major) => Ok(PostgresVersion { tag: raw.to_string(), major }), + Err(_) => mistake!("version '{raw}' is not a Postgres version; write its major version (17)"), + } +} + /// A memory size as the node's `memory` input writes it (`1Gi`, `512Mi`, /// `2G`, `1500M`), in whole MiB, rounded down so nothing is planned for /// memory that is not there. @@ -288,9 +643,9 @@ impl Node for PostgresDatabaseNode { _ctx: InfraProvisionContext, input: ValueBag, ) -> WeftResult { - let database: String = input.get("database")?; + let database = database_name(&input.get::("database")?)?; let storage: String = input.get("storage")?; - let version: String = input.get("version")?; + let version = postgres_version(&input.get::("version")?)?; // Whether anything outside this program may connect. The // author's decision, read here because the spec IS the answer: // a person reading this node sees what is reachable, and @@ -301,6 +656,7 @@ impl Node for PostgresDatabaseNode { let memory: String = input.get("memory")?; let machine = MachineShape { cpu: input.opt("cpu")?, memory: Some(memory.clone()), kind: input.opt::("machineType")?.filter(|t| !t.trim().is_empty()), gpu: None }; let tuning = Tuning::for_machine(&memory)?; + let extensions = extension_names(input.opt::>("extensions")?.unwrap_or_default())?; let credential = Image::Local { name: "credential".into() }; Ok(InfraSpec { @@ -311,13 +667,16 @@ impl Node for PostgresDatabaseNode { name: "db".into(), init_containers: vec![Container::new("mint", credential.clone()) .with_args(vec!["mint".into()]) - .with_env(credential_env(&database)) + .with_env(credential_env(&database, &extensions, &version)) .with_mounts(vec![store_mount()])], containers: vec![ Container::new("postgres", Image::Upstream { - reference: format!("postgres:{version}-alpine"), + reference: format!("postgres:{}-alpine", version.tag), }) - .with_args(tuning.args()) + // Only the unit's own loopback: every client comes + // through the pooler, which is what keeps the + // connections it is asked for to what it serves. + .with_args(tuning.args().into_iter().chain(["-c".into(), "listen_addresses=127.0.0.1".into()]).collect()) // Postgres's share of the machine, so going past it // is an out-of-memory kill of this container that // the platform can see and name, rather than the @@ -333,12 +692,11 @@ impl Node for PostgresDatabaseNode { // mount point with lost+found in it, so it // gets a subdirectory of the mount. EnvEntry::new("PGDATA", DATA_PATH), + // The port as the image's entrypoint reads it, + // for the server and for the setup it runs on + // first boot alike; the pooler takes 5432. + EnvEntry::new("PGPORT", POSTGRES_PORT.to_string()), ]) - .with_ports(vec![ContainerPort { - name: "sql".into(), - port: SQL_PORT, - protocol: Protocol::Tcp, - }]) .with_mounts(vec![store_mount(), socket_mount()]) // Postgres accepts TCP while it is still starting up // and refuses every connection, so the port being @@ -348,7 +706,7 @@ impl Node for PostgresDatabaseNode { // Over TCP, not the unix socket: the entrypoint // runs a temporary socket-only server while it // initialises, so a socket check reports ready - // while port 5432 is still closed. `-d postgres` + // while the port is still closed. `-d postgres` // because without one the check signs in to a // database named after the user, which does not // exist, and the server logs that on every check. @@ -356,6 +714,8 @@ impl Node for PostgresDatabaseNode { "pg_isready".into(), "-h".into(), "127.0.0.1".into(), + "-p".into(), + POSTGRES_PORT.to_string(), "-U".into(), ADMIN_USER.into(), "-d".into(), @@ -363,19 +723,40 @@ impl Node for PostgresDatabaseNode { ]) .with_initial_delay(5), ), + // Before the pooler: a unit says why it is not ready by its + // first container that is not, and the pooler only waits + // on this one, which knows why (`/health`). Container::new("credential", credential) .with_args(vec!["serve".into()]) - .with_env(credential_env(&database)) + .with_env(credential_env(&database, &extensions, &version)) .with_ports(vec![ContainerPort { name: "http".into(), port: CREDENTIAL_PORT, protocol: Protocol::Tcp, }]) .with_limits(Limits { cpu: Some("0.1".into()), memory: Some("64Mi".into()) }) - .with_mounts(vec![secret_mount(), socket_mount()]) + .with_mounts(vec![secret_mount(), socket_mount(), pooler_mount()]) .with_readiness( Probe::http(HEALTH_PATH, CREDENTIAL_PORT).with_initial_delay(2), ), + Container::new("pooler", Image::Local { name: "pooler".into() }) + .with_args(pooler_args(&database, &tuning)) + .with_env(pooler_env(&tuning)) + .with_ports(vec![ContainerPort { + name: "sql".into(), + port: SQL_PORT, + protocol: Protocol::Tcp, + }]) + // No CPU limit: every byte a client and the + // database exchange goes through it. + .with_limits(Limits { cpu: None, memory: Some(format!("{POOLER_MIB}Mi")) }) + .with_mounts(vec![pooler_mount()]) + // Its entrypoint starts PgBouncer only once the + // database's sign-ins are written, so a bound + // port is a pooler that can let clients in. The + // credential container's own readiness waits for + // the same thing to be current. + .with_readiness(Probe::tcp(SQL_PORT).with_initial_delay(2)), ], fs_group: Some(POSTGRES_GID), machine, @@ -385,9 +766,12 @@ impl Node for PostgresDatabaseNode { Volume { name: "store".into(), kind: VolumeKind::Disk { size: storage, class: None } }, // The socket lives and dies with the unit. Volume { name: "socket".into(), kind: VolumeKind::Scratch { size_limit: None } }, + // So does the pooler's, with who may sign in, which the + // credential container writes afresh on every start. + Volume { name: "pooler".into(), kind: VolumeKind::Scratch { size_limit: None } }, ], // `sql` is the only endpoint that may ever be reachable, - // and only when the author asked: reaching Postgres there + // and only when the author asked: reaching the pooler there // still costs a password, so letting a client at it gives // nobody anything they did not already have. That is what // lets a frontend on the same machine use this database @@ -402,7 +786,7 @@ impl Node for PostgresDatabaseNode { endpoints: vec![ Endpoint { name: "sql".into(), - target: EndpointTarget::Unit { unit: "db".into(), container: "postgres".into(), port: "sql".into() }, + target: EndpointTarget::Unit { unit: "db".into(), container: "pooler".into(), port: "sql".into() }, expose: if reachable { Expose::SameNetwork } else { Expose::Project }, }, Endpoint { @@ -413,16 +797,18 @@ impl Node for PostgresDatabaseNode { ], keep_on_terminate: Vec::new(), health_rules: None, + notes: version.note().into_iter().collect(), }) } async fn run(&self, ctx: RunContext) -> WeftResult<()> { - let database: String = ctx.inputs.get("database")?; + let database = database_name(&ctx.inputs.get::("database")?)?; // Three asks that need nothing from each other, so they go out // together. let (sql, credential, published) = tokio::try_join!(ctx.endpoint("sql"), ctx.endpoint("credential"), ctx.published_access())?; let (host, port) = sql.host_and_port()?; + signs_in_as_app(&credential).await?; let published = match published { Some(access) => Some(ctx.open(&access).await?), None => None, @@ -455,8 +841,11 @@ impl Node for PostgresDatabaseNode { // answers on a new one, and the connection must follow it. values.insert("host".to_string(), host); values.insert("port".to_string(), port.to_string()); - values.insert("database".to_string(), database); - values.insert("user".to_string(), ADMIN_USER.to_string()); + // The program signs in as its own role, through the pooler; the + // superuser is never on a connection. + values.insert("watchDatabase".to_string(), watch_database(&database)); + values.insert("database".to_string(), pooler_database(&database)); + values.insert("user".to_string(), APP_USER.to_string()); // SYNC: the published `password` <-> images/credential/bootstrap.py (push_values) values.insert("password".to_string(), password.clone()); // Inside the project's own network; the database serves no TLS. @@ -483,6 +872,25 @@ impl Node for PostgresDatabaseNode { } } +/// Refuse to publish a connection for a unit from before the program had +/// its own role: it has no `app`, and every sign-in would be refused with +/// nothing saying why. Such a unit is one this node's infra was not +/// upgraded for yet, and its credential server's health answer names no +/// role. +// SYNC: signs_in_as_app <-> catalog/postgres/database/images/credential/bootstrap.py (signsInAs) +async fn signs_in_as_app(credential: &weft::EndpointHandle) -> WeftResult<()> { + let answer = credential.call(EndpointMethod::Get, HEALTH_PATH, None).await?; + match answer.get("signsInAs").and_then(|v| v.as_str()) { + Some(user) if user == APP_USER => Ok(()), + other => weft::node_bail!( + "this database's infra was set up by an older weft, which signs the program in as {}, not as \ + `{APP_USER}`. Run `weft infra upgrade `: the database keeps its data and gives the \ + program its own role, and the upgrade runs this node again.", + other.map_or("its superuser".to_string(), |u| format!("`{u}`")) + ), + } +} + /// Hand `password` back to the database and answer whether it is /// really the one the database made. A match also retires it, so it /// stops being readable, which is why this is only ever sent for a diff --git a/catalog/postgres/database/tests.rs b/catalog/postgres/database/tests.rs index 960902885..2d064f6a1 100644 --- a/catalog/postgres/database/tests.rs +++ b/catalog/postgres/database/tests.rs @@ -6,7 +6,11 @@ use serde_json::json; use weft::{EndpointMethod, FakeRig, NodeTest, WeftResult}; -use super::{memory_mib, PostgresDatabaseNode, Tuning, CREDENTIAL_PATH, CREDENTIAL_STORED_PATH, LEAST_MEMORY_MIB, MOST_AT_ONCE}; +use super::{ + database_name, extension_names, memory_mib, PostgresVersion, pooler_config, pooler_database, pooler_hba, postgres_version, watch_database, + PostgresDatabaseNode, Tuning, + CREDENTIAL_PATH, CREDENTIAL_STORED_PATH, HEALTH_PATH, LEAST_MEMORY_MIB, +}; /// The service this node declares it publishes, spelled here as the /// test's own expectation rather than read from the node, so a change @@ -20,6 +24,17 @@ pub fn tests() -> Vec { NodeTest::fake("its_machine_is_what_its_settings_say", sized), NodeTest::fake("keeps_the_credential_endpoint_to_the_project", internal), NodeTest::fake("waits_for_postgres_to_answer_for_itself", ready), + NodeTest::fake("every_client_comes_in_through_the_pooler", one_door), + NodeTest::basic("the_pooler_keeps_sessions_and_its_connections_to_what_postgres_serves", pooler_settings), + NodeTest::basic("only_the_program_s_role_signs_in_through_the_pooler", pooler_sign_ins), + NodeTest::basic("any_database_name_is_kept_as_postgres_has_it", database_names), + NodeTest::basic("a_name_the_pooler_cannot_read_gets_a_door_of_its_own", doors), + NodeTest::fake("a_database_named_with_a_dash_is_reached_by_that_name", dashed), + NodeTest::fake("a_database_the_pooler_cannot_name_is_published_by_its_door", aliased), + NodeTest::fake("each_listed_extension_is_handed_to_the_credential_server", extensions), + NodeTest::basic("a_version_reads_as_its_tag_and_major_and_an_old_one_says_how_to_move", versions), + NodeTest::fake("an_existing_database_on_an_old_postgres_runs_with_a_warning", old_postgres), + NodeTest::basic("a_pooled_connection_idle_too_long_is_never_handed_out", idle_close), NodeTest::basic("memory_reads_in_every_unit_the_input_takes", memory_units), NodeTest::basic("postgres_is_tuned_to_the_memory_it_has", tuned), NodeTest::basic("a_machine_too_small_for_one_worker_is_refused", too_small), @@ -31,6 +46,7 @@ pub fn tests() -> Vec { NodeTest::fake("a_replaced_disk_makes_the_node_ask_again", replaced_disk), NodeTest::fake("an_answer_that_is_neither_yes_nor_no_fails", garbled), NodeTest::fake("a_refusal_from_the_credential_server_surfaces", refused), + NodeTest::fake("a_unit_an_older_weft_set_up_is_refused_naming_the_upgrade", older_unit), ] } @@ -49,6 +65,7 @@ fn reachable_config() -> serde_json::Value { fn endpoints(rig: &FakeRig) { rig.declare_endpoint("sql", "http://wi-db-sql:5432"); rig.declare_endpoint("credential", "http://wi-db-credential:8099"); + rig.answer_endpoint("credential", EndpointMethod::Get, HEALTH_PATH, json!({ "ok": true, "signsInAs": "app" })); } /// The password the node ends up publishing, whichever way it got it. @@ -77,7 +94,7 @@ async fn declares(rig: FakeRig) -> WeftResult<()> { assert_eq!(spec.units.len(), 1); let unit = &spec.units[0]; assert_eq!(unit.init_containers.len(), 1, "the password is minted before Postgres boots"); - assert_eq!(unit.containers.len(), 2, "Postgres plus the server that hands the password over"); + assert_eq!(unit.containers.len(), 3, "Postgres, its pooler, and the server that hands the password over"); let postgres = &unit.containers[0]; assert!( postgres.env.iter().any(|e| matches!(e, @@ -89,21 +106,21 @@ async fn declares(rig: FakeRig) -> WeftResult<()> { weft::infra::EnvEntry { name, .. } if name == "POSTGRES_PASSWORD")), "and never from a value carried in the spec" ); - assert_eq!(spec.volumes.len(), 2, "one disk for data and password, one scratch dir for the socket"); + assert_eq!(spec.volumes.len(), 3, "one disk for data and password, scratch dirs for the two sockets"); assert!( - matches!(spec.volumes[1].kind, weft::infra::VolumeKind::Scratch { .. }), - "the socket never lands on the disk" + spec.volumes[1..].iter().all(|v| matches!(v.kind, weft::infra::VolumeKind::Scratch { .. })), + "a socket never lands on the disk" ); let socket_dir = "/var/run/postgresql"; - for container in &unit.containers { - assert!( - container.mounts.iter().any(|m| m.path == socket_dir && m.volume == "socket"), - "{} shares the socket: it is how a password nobody holds gets replaced", - container.name - ); - } - let credential = &unit.containers[1]; - for name in ["WEFT_SOCKET_DIR", "WEFT_ADMIN_USER"] { + let shares_socket = |name: &str| { + unit.containers.iter().find(|c| c.name == name).expect("declared").mounts.iter().any(|m| m.path == socket_dir && m.volume == "socket") + }; + assert!(shares_socket("postgres") && shares_socket("credential"), "the socket is how a password nobody holds gets replaced"); + // The socket signs in as the superuser with no password, and the + // pooler has no business with that. + assert!(!shares_socket("pooler"), "the pooler never reaches the superuser's door"); + let credential = unit.containers.iter().find(|c| c.name == "credential").expect("declared"); + for name in ["WEFT_SOCKET_DIR", "WEFT_ADMIN_USER", "WEFT_APP_USER", "WEFT_POOLER_USERS_FILE"] { assert!( credential.env.iter().any(|e| matches!(e, weft::infra::EnvEntry { name: n, .. } if n == name)), @@ -196,6 +213,7 @@ async fn ready(rig: FakeRig) -> WeftResult<()> { // socket-only server while it initialises, so a socket check // reports ready while the port is still closed. assert!(command.iter().any(|a| a == "-h"), "the check must go over TCP: {command:?}"); + assert!(command.windows(2).any(|w| w == ["-p", "5433"]), "to Postgres's own port, not the pooler's: {command:?}"); // A database that always exists: without `-d` the check signs in to // one named after the user, and the server logs that it is missing // on every check. @@ -217,25 +235,29 @@ fn memory_units() -> WeftResult<()> { } /// The default machine, worked through by hand from the formula: 1024 -/// MiB less 320 for the rest of the machine leaves 704; a quarter of -/// that is the shared cache, a sixteenth each cleaner's, and the 396 -/// left over holds 24 connections at 16 MiB each. +/// MiB less 352 for the rest of the machine leaves 672; a quarter of +/// that is the shared cache, a sixteenth each cleaner's, and the 378 +/// left over holds 23 connections at 16 MiB each. Three are the +/// superuser's and two the margin; of the 18 left, a quarter (4) are +/// the watchers' and 14 the queries'. fn tuned() -> WeftResult<()> { assert_eq!( Tuning::for_machine("1Gi")?, Tuning { - limit_mib: 704, - max_connections: 24, - shared_buffers_mib: 176, - work_mem_kib: 4352, - maintenance_work_mem_mib: 44, - effective_cache_size_mib: 528, + limit_mib: 672, + max_connections: 23, + shared_buffers_mib: 168, + work_mem_kib: 4318, + maintenance_work_mem_mib: 42, + effective_cache_size_mib: 504, + query_pool: 14, + watch_pool: 4, } ); // More memory, more connections, each still given about Postgres's // own 4MB to sort in. let two = Tuning::for_machine("2Gi")?; - assert_eq!(two.max_connections, 60); + assert_eq!(two.max_connections, 59); assert!((4096..4400).contains(&two.work_mem_kib), "{two:?}"); // Past the most connections Postgres takes, the memory goes to each // connection's sorting instead. @@ -245,8 +267,12 @@ fn tuned() -> WeftResult<()> { assert!(big.work_mem_kib > 100 * 1024, "{big:?}"); // Whatever the machine, everything the settings allow at once fits // in Postgres's share. - for memory in ["1Gi", "1.5Gi", "2Gi", "4Gi", "8Gi", "16Gi", "64Gi"] { + for memory in [&format!("{LEAST_MEMORY_MIB}Mi"), "1Gi", "1.5Gi", "2Gi", "4Gi", "8Gi", "16Gi", "64Gi"] { let t = Tuning::for_machine(memory)?; + // The pooler never opens more than Postgres serves to anybody + // but the superuser, with the margin to spare. + assert_eq!(t.query_pool + t.watch_pool + 3 + 2, t.max_connections, "{memory}: {t:?}"); + assert!(t.query_pool >= 1 && t.watch_pool >= 1 && t.query_pool >= t.watch_pool, "{memory}: {t:?}"); let at_once_kib = t.shared_buffers_mib * 1024 + 3 * t.maintenance_work_mem_mib * 1024 + t.max_connections * (8 * 1024 + 2 * t.work_mem_kib); @@ -256,7 +282,7 @@ fn tuned() -> WeftResult<()> { } fn too_small() -> WeftResult<()> { - let refused = Tuning::for_machine("512Mi").expect_err("512Mi holds fewer connections than one worker opens"); + let refused = Tuning::for_machine("512Mi").expect_err("512Mi leaves the pooler no connection to give"); let floor = format!("raise `memory` to {LEAST_MEMORY_MIB}Mi or more"); assert!(matches!(&refused, weft::WeftError::Input(m) if m.contains(&floor)), "{refused:?}"); Ok(()) @@ -281,9 +307,10 @@ fn full_waits() -> WeftResult<()> { /// The floor the refusal names is the real one (one MiB less is /// refused, it is not), and the `memory` input's description states -/// the same number and connection count. +/// the same number and the connection counts the formula gives. fn least_memory() -> WeftResult<()> { - assert_eq!(Tuning::for_machine(&format!("{LEAST_MEMORY_MIB}Mi"))?.max_connections, MOST_AT_ONCE as u64); + let least = Tuning::for_machine(&format!("{LEAST_MEMORY_MIB}Mi"))?; + assert_eq!((least.query_pool, least.watch_pool), (1, 1), "{least:?}"); assert!(Tuning::for_machine(&format!("{}Mi", LEAST_MEMORY_MIB - 1)).is_err()); let metadata: serde_json::Value = serde_json::from_str(include_str!("metadata.json")).expect("metadata.json reads"); let memory = metadata["inputs"] @@ -291,7 +318,10 @@ fn least_memory() -> WeftResult<()> { .and_then(|inputs| inputs.iter().find(|i| i["name"] == "memory")) .and_then(|i| i["description"].as_str()) .expect("a memory input with a description"); - let says = format!("at least {LEAST_MEMORY_MIB}Mi, the least that holds the {MOST_AT_ONCE} connections"); + let says = format!("at least {LEAST_MEMORY_MIB}Mi."); + assert!(memory.contains(&says), "the description says '{says}': {memory}"); + let serves = |at: &str| Tuning::for_machine(at).map(|t| t.max_connections); + let says = format!("at 1Gi it serves {} connections, at 2Gi {}, at 4Gi {}", serves("1Gi")?, serves("2Gi")?, serves("4Gi")?); assert!(memory.contains(&says), "the description says '{says}': {memory}"); Ok(()) } @@ -300,9 +330,10 @@ async fn runs_tuned(rig: FakeRig) -> WeftResult<()> { let outcome = rig.run_provision_infra(&PostgresDatabaseNode, config()).await.ok()?; let spec = outcome.infra_spec()?; let postgres = &spec.units[0].containers[0]; - assert_eq!(postgres.args, Tuning::for_machine("1Gi")?.args()); - assert!(postgres.args.windows(2).any(|w| w == ["-c", "max_connections=24"]), "{:?}", postgres.args); - assert_eq!(postgres.limits.memory.as_deref(), Some("704Mi")); + assert_eq!(postgres.args[..postgres.args.len() - 2], Tuning::for_machine("1Gi")?.args()); + assert!(postgres.args.windows(2).any(|w| w == ["-c", "max_connections=23"]), "{:?}", postgres.args); + assert!(postgres.args.windows(2).any(|w| w == ["-c", "superuser_reserved_connections=3"]), "{:?}", postgres.args); + assert_eq!(postgres.limits.memory.as_deref(), Some("672Mi")); Ok(()) } @@ -320,7 +351,8 @@ async fn first_run(rig: FakeRig) -> WeftResult<()> { let values = rig.published_values(SERVICE).expect("a connection was published"); assert_eq!(values.get("host").map(String::as_str), Some("wi-db-sql")); assert_eq!(values.get("port").map(String::as_str), Some("5432")); - assert_eq!(values.get("user").map(String::as_str), Some("weft")); + assert_eq!(values.get("user").map(String::as_str), Some("app"), "the program's own role, never the superuser"); + assert_eq!(values.get("watchDatabase").map(String::as_str), Some("app_watch")); assert_eq!(retired(&rig), vec!["minted".to_string()], "retired once, after publishing"); // Retiring AFTER publishing is the whole safety property: were it @@ -355,6 +387,10 @@ async fn later_run(rig: FakeRig) -> WeftResult<()> { assert!(!asked_for_the_password(&rig), "the connection already holds it"); assert_eq!(published_password(&rig).as_deref(), Some("minted")); + // A connection published before the program had its own role is + // brought to it: the same password, now for `app`. + let values = rig.published_values(SERVICE).expect("published again"); + assert_eq!((values.get("user").map(String::as_str), values.get("watchDatabase").map(String::as_str)), (Some("app"), Some("app_watch"))); Ok(()) } @@ -395,7 +431,14 @@ async fn replaced_disk(rig: FakeRig) -> WeftResult<()> { /// the way to telling the user their data is unreachable. async fn garbled(rig: FakeRig) -> WeftResult<()> { endpoints(&rig); - rig.published_connection(SERVICE, &[("password", "minted")]); + rig.published_connection(SERVICE, &[ + ("host", "wi-db-sql"), + ("port", "5432"), + ("database", "app"), + ("user", "app"), + ("password", "minted"), + ("sslmode", "disable"), + ]); rig.answer_endpoint("credential", EndpointMethod::Post, CREDENTIAL_STORED_PATH, json!({ "error": "busy" })); let failure = rig.run(&PostgresDatabaseNode, config()).await.failure()?; @@ -422,3 +465,252 @@ async fn refused(rig: FakeRig) -> WeftResult<()> { assert!(rig.published_values(SERVICE).is_none(), "nothing was published"); Ok(()) } + +/// `sql`, the endpoint clients use, is the pooler's, on the port clients +/// already use; Postgres itself has no port anybody can reach, and +/// listens only on the unit's loopback, where the pooler dials it. +async fn one_door(rig: FakeRig) -> WeftResult<()> { + let outcome = rig.run_provision_infra(&PostgresDatabaseNode, reachable_config()).await.ok()?; + let spec = outcome.infra_spec()?; + let sql = spec.endpoints.iter().find(|e| e.name == "sql").expect("an sql endpoint"); + assert_eq!( + sql.target, + weft::infra::EndpointTarget::Unit { unit: "db".into(), container: "pooler".into(), port: "sql".into() } + ); + let unit = &spec.units[0]; + let pooler = unit.containers.iter().find(|c| c.name == "pooler").expect("a pooler"); + assert_eq!(pooler.ports.iter().map(|p| p.port).collect::>(), vec![5432]); + assert_eq!(pooler.image, weft::infra::Image::Local { name: "pooler".into() }); + let postgres = &unit.containers[0]; + assert!(postgres.ports.is_empty(), "Postgres takes no client but the pooler: {:?}", postgres.ports); + assert!(postgres.args.windows(2).any(|w| w == ["-c", "listen_addresses=127.0.0.1"]), "{:?}", postgres.args); + // The pooler's settings carry no password: who may sign in is the + // file the credential container writes from the database. + for env in &pooler.env { + assert!(!env.value.to_lowercase().contains("password"), "{env:?}"); + } + assert!(pooler.args.iter().all(|a| !a.to_lowercase().contains("password")), "{:?}", pooler.args); + Ok(()) +} + +/// Session pooling, so a firing's statements share one session as they +/// would on a connection of its own; both doors onto the one database, +/// each with its own share; and a client waits however long it takes. +fn pooler_settings() -> WeftResult<()> { + let tuning = Tuning::for_machine("2Gi")?; + let lines = pooler_config("shop", &tuning); + let config = lines.join("\n"); + let line = |key: &str| lines.iter().find(|l| l.starts_with(&format!("{key} = "))).cloned().unwrap_or_else(|| panic!("{key} in {config}")); + assert_eq!(line("pool_mode"), "pool_mode = session"); + assert_eq!(line("shop"), format!("shop = host=127.0.0.1 port=5433 dbname='shop' pool_size={}", tuning.query_pool)); + assert_eq!( + line(&watch_database("shop")), + format!("shop_watch = host=127.0.0.1 port=5433 dbname='shop' pool_size={}", tuning.watch_pool) + ); + assert_eq!(line("listen_port"), "listen_port = 5432"); + assert_eq!(line("query_wait_timeout"), "query_wait_timeout = 0", "a client waits its turn, never refused for waiting"); + assert_eq!(line("server_reset_query"), "server_reset_query = DISCARD ALL"); + // The pooler lets its own idle connections go before the database + // would close them for idling. + let idle: u64 = line("server_idle_timeout").trim_start_matches("server_idle_timeout = ").parse().expect("seconds"); + assert!(idle < super::IDLE_SESSION_SECS, "{idle}"); + assert!(!config.to_lowercase().contains("password"), "{config}"); + // Handed over as the container's arguments, a line each, since no + // platform carries a line break in an environment variable. + let args = super::pooler_args("shop", &tuning); + assert_eq!(args[0], "--config"); + assert_eq!(args[1..=lines.len()], lines[..]); + assert_eq!(args[lines.len() + 1], "--hba"); + assert_eq!(args[lines.len() + 2..], pooler_hba("shop")[..]); + assert!(args.iter().all(|a| !a.contains('\n'))); + Ok(()) +} + +/// The program's role, by password, to the program's database and its +/// watching door only; the superuser only to the pooler's console, over +/// its unix socket. +fn pooler_sign_ins() -> WeftResult<()> { + assert_eq!( + pooler_hba("shop"), + ["local pgbouncer weft trust", "host shop,shop_watch app 0.0.0.0/0 scram-sha-256", "host shop,shop_watch app ::/0 scram-sha-256"] + ); + Ok(()) +} + +/// Whatever name a database was made under is the one it keeps, since +/// a node pointing at another name points at a new, empty database. Only +/// a name no container's settings can carry is refused. +fn database_names() -> WeftResult<()> { + for kept in ["app", "my-app", " spaced ", "it's \\ odd", "naïve", "2shop", "pgbouncer", &"a".repeat(70)] { + assert_eq!(database_name(kept)?, kept); + } + for refused in ["", "two\nlines", "tab\there"] { + assert!(matches!(database_name(refused), Err(weft::WeftError::Input(_))), "{refused:?}"); + } + Ok(()) +} + +/// A name the pooler reads unquoted is its own door; any other gets one +/// made from it, the same on every build, and the pooler hands its +/// clients to the database under the database's own name, quoted. +fn doors() -> WeftResult<()> { + for plain in ["app", "my-app", "Shop_2", &"a".repeat(57)] { + assert_eq!(pooler_database(plain), plain); + assert_eq!(watch_database(plain), format!("{plain}_watch")); + } + // Pinned: the door is published on the connection, so it must not + // move between versions of weft. + assert_eq!(pooler_database("my app"), "db_my_app_23dfc298"); + for odd in ["my app", "it's", "naïve", "pgbouncer", "sameuser", "all", "All", "replication", "a.b", &"a".repeat(58), &"é".repeat(70)] { + let door = pooler_database(odd); + assert!(door.starts_with("db_") && door.len() <= 57, "{door}"); + assert!(door.chars().all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-'), "{door}"); + assert_eq!(watch_database(odd), format!("{door}_watch")); + } + assert_ne!(pooler_database("a.b"), pooler_database("a b"), "two names never share a door"); + let tuning = Tuning::for_machine("1Gi")?; + let lines = pooler_config("it's odd", &tuning); + let door = pooler_database("it's odd"); + assert!( + lines.contains(&format!("{door} = host=127.0.0.1 port=5433 dbname='it''s odd' pool_size={}", tuning.query_pool)), + "{lines:?}" + ); + assert_eq!(pooler_hba("it's odd")[1], format!("host {door},{door}_watch app 0.0.0.0/0 scram-sha-256")); + Ok(()) +} + +/// The value a key of the credential server's settings has. +fn credential_setting(spec: &weft::infra::InfraSpec, name: &str) -> String { + let credential = spec.units[0].containers.iter().find(|c| c.name == "credential").expect("declared"); + credential.env.iter().find(|e| e.name == name).unwrap_or_else(|| panic!("{name} is set")).value.clone() +} + +/// Run the node's setup and its body against a database named `name`, +/// and hand back the spec it declared and the values it published. +async fn named(rig: &FakeRig, name: &str) -> WeftResult<(weft::infra::InfraSpec, std::collections::BTreeMap)> { + let mut settings = config(); + settings["database"] = json!(name); + let spec = rig.run_provision_infra(&PostgresDatabaseNode, settings.clone()).await.ok()?.infra_spec()?.clone(); + endpoints(rig); + rig.answer_endpoint("credential", EndpointMethod::Get, CREDENTIAL_PATH, json!({ "password": "minted" })); + rig.answer_endpoint("credential", EndpointMethod::Post, CREDENTIAL_STORED_PATH, json!({ "stored": true })); + rig.run(&PostgresDatabaseNode, settings).await.ok()?; + Ok((spec, rig.published_values(SERVICE).expect("published"))) +} + +/// An existing database named with a dash keeps that name everywhere: +/// Postgres's own, the pooler's door, the connection. +async fn dashed(rig: FakeRig) -> WeftResult<()> { + let (spec, values) = named(&rig, "my-app").await?; + let postgres = &spec.units[0].containers[0]; + assert!(postgres.env.iter().any(|e| e.name == "POSTGRES_DB" && e.value == "my-app"), "{:?}", postgres.env); + assert_eq!(credential_setting(&spec, "WEFT_DATABASE"), "my-app"); + assert_eq!(credential_setting(&spec, "WEFT_POOLER_DATABASE"), "my-app"); + assert_eq!(values["database"], "my-app"); + assert_eq!(values["watchDatabase"], "my-app_watch"); + Ok(()) +} + +/// A name the pooler cannot read as written stays Postgres's own, and the +/// connection carries the door the pooler lets clients in by. +async fn aliased(rig: FakeRig) -> WeftResult<()> { + let (spec, values) = named(&rig, "my app").await?; + let door = pooler_database("my app"); + let postgres = &spec.units[0].containers[0]; + assert!(postgres.env.iter().any(|e| e.name == "POSTGRES_DB" && e.value == "my app"), "{:?}", postgres.env); + assert_eq!(credential_setting(&spec, "WEFT_DATABASE"), "my app"); + assert_eq!(credential_setting(&spec, "WEFT_POOLER_DATABASE"), door); + assert_eq!(values["database"], door); + assert_eq!(values["watchDatabase"], format!("{door}_watch")); + let pooler = spec.units[0].containers.iter().find(|c| c.name == "pooler").expect("declared"); + assert!(pooler.args.iter().any(|a| a.starts_with(&format!("{door} = ")) && a.contains("dbname='my app'")), "{:?}", pooler.args); + Ok(()) +} + +/// The extensions the node lists reach the credential server, which makes +/// them as the superuser on every start; none listed is an empty list. +async fn extensions(rig: FakeRig) -> WeftResult<()> { + let mut settings = config(); + settings["extensions"] = json!(["postgres_fdw"]); + let listed = rig.run_provision_infra(&PostgresDatabaseNode, settings).await.ok()?; + assert_eq!(credential_setting(listed.infra_spec()?, "WEFT_EXTENSIONS"), r#"["postgres_fdw"]"#); + let none = rig.run_provision_infra(&PostgresDatabaseNode, config()).await.ok()?; + assert_eq!(credential_setting(none.infra_spec()?, "WEFT_EXTENSIONS"), "[]"); + assert!(extension_names(vec!["".into()]).is_err()); + assert!(extension_names(vec!["a\nb".into()]).is_err()); + Ok(()) +} + +fn versions() -> WeftResult<()> { + for (raw, major) in [("17", 17), ("16.4", 16), ("14", 14), ("13", 13), ("9.6", 9)] { + assert_eq!(postgres_version(raw)?, PostgresVersion { tag: raw.to_string(), major }); + } + for refused in ["latest", " 17", "014", "17.4.1", "17.", ".4", "0", "17.04", "99999"] { + assert!(postgres_version(refused).is_err(), "{refused:?}"); + } + assert_eq!(postgres_version("14")?.note(), None); + let note = postgres_version("13")?.note().expect("13 keeps idle sessions open"); + assert!(note.contains("14 or later") && note.contains("pg_dump") && note.contains("pg_restore"), "{note}"); + // The build warns of it with the same steps. + let metadata: serde_json::Value = serde_json::from_str(include_str!("metadata.json")).expect("metadata reads"); + let rule = metadata["validate"] + .as_array() + .expect("rules") + .iter() + .find(|rule| rule["then"]["message"].as_str().is_some_and(|m| m.contains("older than 14"))) + .expect("a rule for an old version"); + assert_eq!(rule["then"]["severity"], "warning", "an existing database on it keeps running"); + let said = rule["then"]["message"].as_str().unwrap_or_default(); + assert!(said.contains(&super::MOVE_TO_14.replace("", "{id}")), "{said}"); + Ok(()) +} + +/// A database on a Postgres older than 14 is declared, with a note for +/// `weft infra status` and the card saying what it lacks and how to move; +/// the credential container is told the major, so it refuses a new +/// database on it and skips the idle limit on an existing one. On 14 or +/// later there is nothing to say. +async fn old_postgres(rig: FakeRig) -> WeftResult<()> { + let mut settings = config(); + settings["version"] = json!("13"); + let old = rig.run_provision_infra(&PostgresDatabaseNode, settings).await.ok()?; + let spec = old.infra_spec()?; + assert_eq!(spec.notes.len(), 1, "{:?}", spec.notes); + assert!(spec.notes[0].contains("does not close a connection a client left idle"), "{}", spec.notes[0]); + assert_eq!(credential_setting(spec, "WEFT_POSTGRES_MAJOR"), "13"); + assert_eq!(credential_setting(spec, "WEFT_POSTGRES_NOTE"), spec.notes[0]); + let postgres = &spec.units[0].containers[0]; + assert_eq!(postgres.image, weft::infra::Image::Upstream { reference: "postgres:13-alpine".into() }); + let current = rig.run_provision_infra(&PostgresDatabaseNode, config()).await.ok()?; + let spec = current.infra_spec()?; + assert!(spec.notes.is_empty(), "{:?}", spec.notes); + assert_eq!(credential_setting(spec, "WEFT_POSTGRES_MAJOR"), "17"); + assert_eq!(credential_setting(spec, "WEFT_POSTGRES_NOTE"), ""); + Ok(()) +} + +/// A connection idle in the pool for [`IDLE_CLOSE`] or longer is never +/// handed to a run: by then the database may have closed it, and a worker +/// without CPU between requests cannot have closed it itself on time. +fn idle_close() -> WeftResult<()> { + use super::super::postgres::{still_fresh, IDLE_CLOSE}; + let since = std::time::Instant::now(); + assert!(still_fresh(since, since)); + assert!(still_fresh(since, since + IDLE_CLOSE - std::time::Duration::from_millis(1))); + assert!(!still_fresh(since, since + IDLE_CLOSE)); + assert!(still_fresh(since + IDLE_CLOSE, since), "a clock read earlier is not idle at all"); + Ok(()) +} + +/// A unit an older weft set up has no role for the program: publishing +/// `app` for it would fail every sign-in with nothing saying why, so the +/// node refuses before publishing and names the upgrade. +async fn older_unit(rig: FakeRig) -> WeftResult<()> { + rig.declare_endpoint("sql", "http://wi-db-sql:5432"); + rig.declare_endpoint("credential", "http://wi-db-credential:8099"); + rig.answer_endpoint("credential", EndpointMethod::Get, HEALTH_PATH, json!({ "ok": true })); + let failure = rig.run(&PostgresDatabaseNode, config()).await.failure()?; + assert!(failure.contains("weft infra upgrade"), "{failure}"); + assert!(rig.published_values(SERVICE).is_none(), "nothing was published"); + Ok(()) +} diff --git a/catalog/postgres/postgres.rs b/catalog/postgres/postgres.rs index 5fbabbffe..0b0a26d27 100644 --- a/catalog/postgres/postgres.rs +++ b/catalog/postgres/postgres.rs @@ -3,7 +3,9 @@ //! answered rows back into JSON. //! //! The connection stores `host`, `database`, `user`, `password`, and -//! the optional `port` and `sslmode`; this module is the one place +//! the optional `port`, `sslmode` and `watchDatabase` (the database name +//! a watching query connects to, when the database keeps a separate door +//! for those, as a PostgresDatabase does); this module is the one place //! that vocabulary is read, so every Postgres node dials the same way. //! //! Parameter typing: a JSON string binds as the TEXT form of whatever @@ -123,12 +125,35 @@ type Connection = tokio_postgres::Connection< >::Stream, >; -/// Dial the wired database: the one handshake behind `connect` and -/// `connect_listening`, each of which drives the connection its own way. -async fn dial(ctx: &RunContext, conn: &OpenedConnection) -> WeftResult<(tokio_postgres::Client, Connection)> { - dial_with(ctx, &dial_config(conn)?).await +/// Dial the wired database for a run that watches: the one handshake +/// behind `connect_alone` and `connect_listening`, each of which drives +/// the connection its own way. Through the database's door for watchers +/// when the connection names one ([`watch_config`]). +async fn dial_watching(ctx: &RunContext, conn: &OpenedConnection) -> WeftResult<(tokio_postgres::Client, Connection)> { + dial_with(ctx, &watch_config(conn)?).await } +/// [`dial_config`] for a run that watches: the same database under the +/// name its door for watchers goes by (`watchDatabase`), when the +/// connection has one. A PostgresDatabase keeps that door apart, with +/// its own connections, so watchers that hold theirs for hours never +/// take every connection from queries, nor queries from watchers. +pub fn watch_config(conn: &OpenedConnection) -> WeftResult { + let mut config = dial_config(conn)?; + if let Some(watch) = conn.opt_value("watchDatabase").map(str::trim).filter(|w| !w.is_empty()) { + config.dbname(watch); + } + Ok(config) +} + +/// Sent first on a watching connection: it sits idle between wakes on +/// purpose, so it lifts, for its own session, the limit a database may +/// put on idle sessions (`idle_session_timeout`, from Postgres 14 on; a +/// PostgresDatabase sets one so idle clients give their connection +/// back). Read through `pg_settings`, so a server without the setting +/// changes nothing instead of refusing. +const STAYS_IDLE: &str = "SELECT set_config('idle_session_timeout', '0', false) FROM pg_settings WHERE name = 'idle_session_timeout'"; + /// The first wait before asking a full database again, doubled on every /// refusal up to [`FULL_LONGEST_WAIT`]. const FULL_FIRST_WAIT: std::time::Duration = std::time::Duration::from_millis(500); @@ -188,7 +213,8 @@ impl FullWaits { let said = format!( "postgres: the database has no connection free (too many clients); {} run(s) wait for one now, \ {} began waiting since the last notice. Runs slow down until one frees up. If this keeps happening, \ - raise the database's size: on a PostgresDatabase node, raise `memory` (its connection count follows it)", + the database takes fewer connections than the program's workers and its other clients ask for at \ + once: raise its connection limit, or put a pooler in front of it", self.waiting, self.began ); self.said_at = Some(now); @@ -268,17 +294,28 @@ fn connect_failed(e: tokio_postgres::Error) -> WeftError { } /// The most runs of one worker that hold a connection to one database at -/// once. Kept low on purpose: Postgres takes 100 connections by default, -/// and several workers of a project share them, so 20 each lets a few -/// workers run side by side without the database refusing anyone. A -/// PostgresDatabase takes fewer on a small machine (24 at its default -/// 1Gi, see its `memory` input), so there a second worker can find it -/// full and waits for a connection to free up ([`dial_with`]). -/// A run past it waits for one to finish. A proper fix is a pooler in -/// front of the database (TODO.md, "Several workers of a project can open -/// more Postgres connections than the database takes"). +/// once; a run past it waits for one to finish. It keeps one worker from +/// taking every connection from its siblings, and nothing more: how many +/// workers run at once is not this worker's to know (they scale, and a +/// rollout runs old and new copies side by side), so it cannot keep a +/// database under its limit. A PostgresDatabase is kept there by its +/// pooler, where every client past what Postgres serves waits its turn; +/// a database reached through a PostgresAccess that has no pooler can +/// still be found full, and a run then waits for a connection to free up +/// ([`dial_with`]). pub(crate) const MOST_AT_ONCE: usize = 20; +/// How long a connection may sit idle in a worker's pool before it is +/// closed instead of handed to the next run. Long enough to serve a burst +/// of runs with one sign-in; short, because an idle connection held here +/// is one the database cannot give anybody else, and a pooler in session +/// mode keeps a real Postgres connection behind every one of them. A +/// worker that loses its CPU between requests (a Cloud Run worker does) +/// cannot close them on time itself, so a connection idle past this is +/// also never handed out: the database may have closed it meanwhile +/// (a PostgresDatabase does after a minute). +pub(crate) const IDLE_CLOSE: std::time::Duration = std::time::Duration::from_secs(5); + /// How long a returning connection gets to clear its session before it /// is let go instead. Clearing is a few short statements a healthy /// server answers in milliseconds; a connection that sits longer is one @@ -289,16 +326,54 @@ const CLEAR_WITHIN: std::time::Duration = std::time::Duration::from_secs(10); /// Connections to one database, signed in once and held by the worker for /// its runs to share (`ctx.shared`), instead of dialed and signed in again /// by every run. Built from the connection's values, so a changed -/// password builds a fresh pool. +/// password builds a fresh pool. A connection idle for [`IDLE_CLOSE`] is +/// closed. pub struct Pool { config: Config, - idle: std::sync::Mutex>, + /// Oldest first, each with when it went idle. + idle: std::sync::Mutex>, } impl Pool { fn new(conn: &OpenedConnection) -> WeftResult { Ok(Self { config: dial_config(conn)?, idle: std::sync::Mutex::new(Vec::new()) }) } + + /// The most recently returned connection that is still open and was + /// idle for less than [`IDLE_CLOSE`] at `now`, closing every one that + /// was idle longer (dropping a client closes its connection). + fn take(&self, now: std::time::Instant) -> Option { + let mut idle = self.idle.lock().expect("pool poisoned"); + idle.retain(|(_, since)| still_fresh(*since, now)); + while let Some((client, _)) = idle.pop() { + if !client.is_closed() { + return Some(client); + } + } + None + } + + /// Put `client` back, idle from `now`, and close it once it has sat + /// idle for [`IDLE_CLOSE`] if no run took it. The close is a timer + /// here, so a worker without CPU between requests runs it late; the + /// next run's [`Pool::take`] closes it then, before anything uses it. + fn give_back(self: std::sync::Arc, client: tokio_postgres::Client, now: std::time::Instant) { + self.idle.lock().expect("pool poisoned").push((client, now)); + let pool = std::sync::Arc::downgrade(&self); + tokio::spawn(async move { + tokio::time::sleep(IDLE_CLOSE).await; + if let Some(pool) = pool.upgrade() { + let now = std::time::Instant::now(); + pool.idle.lock().expect("pool poisoned").retain(|(_, since)| still_fresh(*since, now)); + } + }); + } +} + +/// Whether a connection idle since `since` may still be handed out at +/// `now`. Pure, so the decision is testable. +pub fn still_fresh(since: std::time::Instant, now: std::time::Instant) -> bool { + now.saturating_duration_since(since) < IDLE_CLOSE } /// A client of the pool, for one run's use. It goes back to the pool when @@ -360,7 +435,7 @@ impl Drop for Pooled { let may_hold_transaction = self.may_hold_transaction; tokio::spawn(async move { match tokio::time::timeout(CLEAR_WITHIN, clear_session(&client, may_hold_transaction)).await { - Ok(Ok(())) => pool.idle.lock().expect("pool poisoned").push(client), + Ok(Ok(())) => pool.arc().give_back(client, std::time::Instant::now()), Ok(Err(e)) => tracing::warn!("postgres: a pooled connection is let go, its session could not be cleared: {e}"), Err(_) => tracing::warn!( "postgres: a pooled connection is let go, its database did not answer within {}s while its session was cleared", @@ -466,16 +541,8 @@ async fn clear_session(client: &tokio_postgres::Client, may_hold_transaction: bo /// does. pub async fn connect(ctx: &RunContext, access: &weft::access::Access) -> WeftResult { let pool = ctx.shared(access, |opened| async move { Pool::new(&opened) }).with_limit(MOST_AT_ONCE).await?; - loop { - let idle = pool.idle.lock().expect("pool poisoned").pop(); - match idle { - Some(client) if !client.is_closed() => { - return Ok(Pooled { client: Some(client), pool: Some(pool), cancel: ctx.cancellation(), may_hold_transaction: false }); - } - // A connection the server or the network closed since. - Some(_) => continue, - None => break, - } + if let Some(client) = pool.take(std::time::Instant::now()) { + return Ok(Pooled { client: Some(client), pool: Some(pool), cancel: ctx.cancellation(), may_hold_transaction: false }); } let (client, connection) = dial_with(ctx, &pool.config).await?; // The connection future must be driven for the client to work; it @@ -491,10 +558,11 @@ pub async fn connect(ctx: &RunContext, access: &weft::access::Access) -> WeftRes /// A connection of its own to the wired database, for a run that holds /// one for as long as it watches (it would hold a pooled one away from -/// every other run). Its task ends when the client drops or the run is +/// every other run), dialed the way a watcher dials ([`dial_watching`], +/// [`STAYS_IDLE`]). Its task ends when the client drops or the run is /// cancelled. pub async fn connect_alone(ctx: &RunContext, conn: &OpenedConnection) -> WeftResult { - let (client, connection) = dial(ctx, conn).await?; + let (client, connection) = dial_watching(ctx, conn).await?; let cancel = ctx.cancellation(); tokio::spawn(async move { tokio::select! { @@ -506,6 +574,10 @@ pub async fn connect_alone(ctx: &RunContext, conn: &OpenedConnection) -> WeftRes _ = cancel.cancelled() => {} } }); + client + .batch_execute(STAYS_IDLE) + .await + .map_err(|e| WeftError::NodeFailed(format!("postgres: keep the watching connection open while idle: {}", pg_detail(&e))))?; Ok(client) } @@ -520,7 +592,8 @@ pub enum ListenEvent { Failed(String), } -/// Dial the wired database and `LISTEN` on `channel`: the client, and +/// Dial the wired database the way a watcher dials ([`dial_watching`], +/// [`STAYS_IDLE`]) and `LISTEN` on `channel`: the client, and /// the stream of notifications the server raises on that channel (a /// `NOTIFY channel` in a trigger, say). The connection task is driven /// here so its asynchronous messages can be read off it; it ends when @@ -532,7 +605,7 @@ pub async fn connect_listening( channel: &str, ) -> WeftResult<(tokio_postgres::Client, tokio::sync::mpsc::UnboundedReceiver)> { use futures::StreamExt; - let (client, mut connection) = dial(ctx, conn).await?; + let (client, mut connection) = dial_watching(ctx, conn).await?; let (tx, rx) = tokio::sync::mpsc::unbounded_channel(); let cancel = ctx.cancellation(); tokio::spawn(async move { @@ -563,9 +636,9 @@ pub async fn connect_listening( } }); client - .batch_execute(&format!("LISTEN {}", quote_ident(channel)?)) + .batch_execute(&format!("{STAYS_IDLE}; LISTEN {}", quote_ident(channel)?)) .await - .map_err(|e| WeftError::NodeFailed(format!("postgres: LISTEN {channel}: {}", pg_detail(&e))))?; + .map_err(|e| WeftError::NodeFailed(format!("postgres: keep the watching connection open while idle and LISTEN {channel}: {}", pg_detail(&e))))?; Ok((client, rx)) } diff --git a/catalog/postgres/watch_query/tests.rs b/catalog/postgres/watch_query/tests.rs index aeb280c8b..3eed154ea 100644 --- a/catalog/postgres/watch_query/tests.rs +++ b/catalog/postgres/watch_query/tests.rs @@ -5,10 +5,12 @@ //! outside, which the live rig cannot do yet; it is proven end to end //! by a program. +use async_trait::async_trait; use serde_json::json; -use weft::{FakeRig, NodeManifest, NodeTest, WeftResult}; +use weft::{Access, FakeRig, Node, NodeManifest, NodeMetadata, NodeTest, RunContext, WeftResult}; +use super::super::postgres::watch_config; use super::super::postgres_execute_query::PostgresExecuteQueryNode; use super::{cadence, changed, first_row, watched, PostgresWatchQueryNode}; @@ -22,6 +24,7 @@ pub fn tests() -> Vec { NodeTest::fake("a_bad_cadence_refuses_before_dialing", bad_cadence), NodeTest::basic("row_carries_the_first_row_or_null", first_row_or_null), NodeTest::basic("the_placeholder_pattern_is_postgres_query_s", same_pattern), + NodeTest::fake("a_watcher_dials_the_database_s_door_for_watchers_when_it_has_one", watching_door), ] } @@ -134,3 +137,38 @@ async fn bad_cadence(rig: FakeRig) -> WeftResult<()> { assert!(err.contains("intervalSecs must be at least 5"), "{err}"); Ok(()) } + +/// The database name a watching run dials on the wired connection, under +/// this node's own manifest: what `watch_config` makes of an opened +/// connection, with no socket behind it. +struct DialsInto(&'static str); + +impl NodeManifest for DialsInto { + fn manifest(&self) -> &'static NodeMetadata { + PostgresWatchQueryNode.manifest() + } +} + +#[async_trait] +impl Node for DialsInto { + async fn run(&self, ctx: RunContext) -> WeftResult<()> { + let account: Access = ctx.inputs.get("account")?; + let config = watch_config(&ctx.open(&account).await?)?; + if config.get_dbname() != Some(self.0) { + weft::node_bail!("a watcher dials {:?}, not {:?}", config.get_dbname(), self.0); + } + Ok(()) + } +} + +/// A PostgresDatabase keeps a door apart for watchers, named on its +/// connection, so they never take the connections queries need; any +/// other connection (a PostgresAccess) is dialed as it is. +async fn watching_door(rig: FakeRig) -> WeftResult<()> { + let inputs = |rig: &FakeRig| json!({ "account": rig.access("postgres"), "query": "SELECT 1", "intervalSecs": 5 }); + connection(&rig); + rig.run(&DialsInto("app"), inputs(&rig)).await.ok()?; + rig.connection_value("postgres", "watchDatabase", "app_watch"); + rig.run(&DialsInto("app_watch"), inputs(&rig)).await.ok()?; + Ok(()) +} diff --git a/crates/weft-access-store/src/flows.rs b/crates/weft-access-store/src/flows.rs index c88628c2d..c6be7b5ab 100644 --- a/crates/weft-access-store/src/flows.rs +++ b/crates/weft-access-store/src/flows.rs @@ -13,7 +13,7 @@ use weft_task_store::db::{self, Client, FromRow, JsonText, Pool, Row}; use weft_core::access::client::{authed_client, base_client, resolve_steps, RefusalKind}; use weft_core::access::spec::{ - lookup_path, Acquisition, AnswerReader, CredentialField, Door, GrantCoexistence, OAuthGrant, TestCall, + lookup_path, storable_values, Acquisition, AnswerReader, CredentialField, Door, GrantCoexistence, OAuthGrant, TestCall, TokenAnswer, TokenCall, TokenExpiry, TokenLife, TokenRead, VerificationRung, }; // The connect wire shapes live in the core (weft-core `access::wire`), @@ -353,14 +353,14 @@ pub async fn connect_direct( let mut life: Option = None; match &spec.acquisition { Acquisition::Static { fields } => { - values = storable_values(spec, fields, values)?; + values = storable_values(spec, fields, values).map_err(AccessError::Invalid)?; } Acquisition::MintJwt { fields, .. } => { // Held to the same rule as any other pasted set: these are // the fields a PERSON filled in, and a missing or blank one // would otherwise fail somewhere inside the mint instead of // naming itself here. - values = storable_values(spec, fields, values)?; + values = storable_values(spec, fields, values).map_err(AccessError::Invalid)?; // Mint + exchange once now, so a bad key/id fails at // connect instead of at first run. The minted token is // stored like any other value and re-minted lazily on @@ -392,7 +392,7 @@ pub async fn connect_direct( if spec.verification.cost.may_auto_run() { run_test_call(spec.test.as_ref(), &spec.auth, &mut values).await?; } - let identity = resolve_identity(spec, &values)?; + let identity = spec.identity_of(&values).map_err(AccessError::Invalid)?; // What the row records as granted: the introspected set when the // service's verification produced one (the `granted_permissions` // capture), else the user's claimed ticks. @@ -538,44 +538,6 @@ fn connection_name(typed: Option, registration: &Option } } -/// `values` reduced to what may actually be STORED for `spec`, or the -/// reason they cannot be: every required field present and non-blank, -/// no name the service does not declare, the service's all-or-nothing -/// groups satisfied, and blanks dropped so a later read sees "missing" -/// rather than an empty secret. -/// -/// The gate for every set of values a PERSON or a NODE supplied: a -/// pasted connection, a minted one, and one a node published all -/// answer the same questions and are held to the same standard. A -/// browser consent is the one that does not come through here, and -/// cannot: its values are the provider's own answer to the token -/// exchange, not fields anybody filled in. -fn storable_values( - spec: &AccessSpec, - fields: &[CredentialField], - mut values: BTreeMap, -) -> anyhow::Result> { - for f in fields { - let empty = values.get(&f.name).is_none_or(|v| v.trim().is_empty()); - if empty && !f.optional { - return Err(AccessError::Invalid(format!("field '{}' is required", f.name)).into()); - } - } - if let Some(unknown) = values.keys().find(|k| !fields.iter().any(|f| &&f.name == k)) { - return Err(AccessError::Invalid(format!( - "the '{}' service declares no value named '{unknown}'", - spec.service - )) - .into()); - } - weft_core::access::spec::capability_shortfall(&spec.capabilities, |name| { - values.get(name).is_some_and(|v| !v.trim().is_empty()) - }) - .map_err(AccessError::Invalid)?; - values.retain(|_, v| !v.trim().is_empty()); - Ok(values) -} - pub(crate) async fn insert_grant(db: &impl Client, tenant: &str, grant: NewGrant<'_>) -> anyhow::Result { write_grant(db, tenant, None, grant) .await? @@ -1589,7 +1551,7 @@ async fn finish_connect( if spec.verification.cost.may_auto_run() { run_test_call(spec.test.as_ref(), &spec.auth, &mut values).await.map_err(sign_in_spent)?; } - let identity = resolve_identity(&spec, &values)?; + let identity = spec.identity_of(&values).map_err(AccessError::Invalid)?; let label = Some(registration.label.clone()); // An instance's connection always lives in its project (whatever the @@ -1945,16 +1907,6 @@ pub(crate) async fn run_test_call( apply_captures(&test.captures, &body, values, AnswerOf::First) } -pub(crate) fn resolve_identity( - spec: &AccessSpec, - values: &BTreeMap, -) -> anyhow::Result> { - match &spec.identity { - None => Ok(None), - Some(t) => Ok(Some(t.resolve(values).map_err(AccessError::Invalid)?)), - } -} - fn head_of(body: &str) -> String { weft_core::truncate_user_string(body, 300) } @@ -2147,20 +2099,6 @@ mod tests { } - fn field(name: &str, optional: bool) -> CredentialField { - CredentialField { - name: name.into(), - label: None, - optional, - secret: false, - placeholder: None, - } - } - - fn values(pairs: &[(&str, &str)]) -> BTreeMap { - pairs.iter().map(|(k, v)| (k.to_string(), v.to_string())).collect() - } - /// A connection left unnamed is named after its app, else its /// service, so no list ever shows a nameless row a person made. #[test] @@ -2182,44 +2120,6 @@ mod tests { assert_eq!(connection_name(None, &None, &spec), "Firecrawl"); } - /// The gate every set of values a person or a node supplied - /// passes. A blank required value is MISSING, not stored: a - /// downstream node would otherwise get an empty password and fail - /// at the provider instead of here. - #[test] - fn the_value_check_holds_a_pasted_and_a_published_connection_to_one_rule() { - // The spec DECLARES the two fields the check is run against, - // so the rule and the recipe cannot be a mismatched pair the - // production callers could never produce. - let spec: AccessSpec = serde_json::from_value(serde_json::json!({ - "service": "pg", - "acquisition": { "kind": "static", "fields": [ - { "name": "host" }, - { "name": "port", "optional": true }, - ]}, - })) - .expect("a static spec"); - let fields = [field("host", false), field("port", true)]; - let check = |vals| storable_values(&spec, &fields, vals); - - check(values(&[("host", "db"), ("port", "5432")])).expect("a complete set passes"); - - let kept = check(values(&[("host", "db"), ("port", " ")])) - .expect("a blank optional passes"); - assert!(!kept.contains_key("port"), "a blank optional stores nothing"); - - let e = check(values(&[("port", "5432")])).expect_err("required missing"); - assert!(e.to_string().contains("'host' is required"), "{e}"); - - let e = check(values(&[("host", " ")])) - .expect_err("a blank required value is missing"); - assert!(e.to_string().contains("'host' is required"), "{e}"); - - let e = check(values(&[("host", "db"), ("hostname", "db")])) - .expect_err("undeclared name"); - assert!(e.to_string().contains("no value named 'hostname'"), "{e}"); - } - /// What lands on the row: only an authoritative rung with a /// provider-produced set records VERIFIED; a non-authoritative /// rung records the claimed ticks even when something echoed, and @@ -2338,11 +2238,8 @@ async fn publish_grant_in( req: PublishAccess, ) -> anyhow::Result { let PublishAccess { spec, project_id, node_id, instance, values, label } = req; - spec.validate().map_err(AccessError::Invalid)?; - let fields = - weft_core::access::spec::publishable_fields(&spec).map_err(AccessError::Invalid)?; - let values = storable_values(&spec, fields, values)?; - let identity = resolve_identity(&spec, &values)?; + let weft_core::access::spec::PublishableConnection { values, identity } = + weft_core::access::spec::publishable_connection(&spec, values).map_err(AccessError::Invalid)?; let current: Option = db .opt( "SELECT id, spec, spec_version, values_sealed, label, identity FROM access_grant diff --git a/crates/weft-cli/src/client.rs b/crates/weft-cli/src/client.rs index 598df8ab0..335f15e0b 100644 --- a/crates/weft-cli/src/client.rs +++ b/crates/weft-cli/src/client.rs @@ -718,8 +718,19 @@ pub(crate) mod fake { } /// The address of a port nothing listens on: a connection there is refused. + /// + /// The port stays bound, never listening, for as long as the test + /// process runs. Released, it is free for any other process to take, + /// and with every test running at once another test's fake install + /// did: a test expecting a refusal then spent that install's canned + /// answers, and both failed for something neither did. pub fn closed() -> String { - format!("http://{}", std::net::TcpListener::bind("127.0.0.1:0").unwrap().local_addr().unwrap()) + static HELD: std::sync::Mutex> = std::sync::Mutex::new(Vec::new()); + let socket = tokio::net::TcpSocket::new_v4().unwrap(); + socket.bind("127.0.0.1:0".parse().unwrap()).unwrap(); + let address = format!("http://{}", socket.local_addr().unwrap()); + HELD.lock().unwrap().push(socket); + address } } diff --git a/crates/weft-compiler/src/codegen.rs b/crates/weft-compiler/src/codegen.rs index 4bfcb28ee..d59a8758b 100644 --- a/crates/weft-compiler/src/codegen.rs +++ b/crates/weft-compiler/src/codegen.rs @@ -1344,6 +1344,28 @@ fn baked_type_decls(catalog: &FsCatalog) -> String { .collect() } +/// The recipes of the services `node_types` publish, as the JSON of a +/// `Vec` a node-test binary bakes in. Each is resolved the +/// way enrich resolves it for a project build +/// ([`weft_core::access::spec::published_recipe`], against the WHOLE +/// catalog), so a package publishing a service another package declares +/// tests against the same recipe production holds it to, and a node +/// publishing a service nothing declares is refused here, at emit time. +pub(crate) fn baked_published_services(catalog: &FsCatalog, node_types: &[String]) -> BuildResult { + use weft_core::MetadataCatalog as _; + let mut services: BTreeMap<&str, &weft_core::AccessSpec> = BTreeMap::new(); + for node_type in node_types { + let Some(service) = catalog.lookup(node_type).and_then(|m| m.publishes.as_deref()) else { + continue; + }; + let spec = weft_core::access::spec::published_recipe(catalog.all(), node_type, service) + .map_err(BuildError::Build)?; + services.insert(&spec.service, spec); + } + let specs: Vec<&weft_core::AccessSpec> = services.into_values().collect(); + serde_json::to_string(&specs).map_err(|e| BuildError::Build(format!("serialize the published services: {e}"))) +} + fn write_main_rs(src_dir: &Path, catalog: &FsCatalog) -> BuildResult<()> { std::fs::write(src_dir.join("main.rs"), worker_main_source(&baked_type_decls(catalog))).map_err(BuildError::Io)?; Ok(()) @@ -1622,6 +1644,9 @@ pub fn emit_test_crate( }; let mut node_types = package.node_types.clone(); node_types.sort(); + // Resolved before anything is written, so a node publishing a + // service no package declares leaves no half-emitted crate behind. + let published_services = baked_published_services(catalog, &node_types)?; let pkg = PackageEmit { package, node_types: node_types.clone(), @@ -1717,7 +1742,8 @@ path = "src/main.rs" let main_rs = format!( r#"//! Per-package node-test binary. All runner logic lives in //! `weft_engine::test_runner`; this file only bakes the project's -//! type declarations and the package's node registry. +//! type declarations, the recipes of the services the package's nodes +//! publish, and the package's node registry. mod registry; @@ -1731,7 +1757,9 @@ fn main() -> std::process::ExitCode {{ ) .expect("first registry install in this process"); - weft_engine::test_runner::main(®istry::PROJECT_CATALOG) + // The recipe each node of this package publishes against, resolved + // at emit time against the whole catalog, as a project build does. + weft_engine::test_runner::main(®istry::PROJECT_CATALOG, {published_services:?}) }} "#, ); diff --git a/crates/weft-compiler/src/enrich.rs b/crates/weft-compiler/src/enrich.rs index f206128bc..6c3bb4f99 100644 --- a/crates/weft-compiler/src/enrich.rs +++ b/crates/weft-compiler/src/enrich.rs @@ -738,25 +738,14 @@ pub fn enrich_collecting( // ships only the node types this project uses and so usually // does not include the service's own access node. if let Some(service) = &meta.publishes { - match weft_core::access::spec::spec_for_service(catalog.all(), service) { - // Refused HERE too, not only where the connection is - // finally written: a recipe shape a published - // connection cannot have is an authoring mistake, and - // it belongs on the author's screen at build time. - Some(spec) => match weft_core::access::spec::publishable_fields(spec) { - Ok(_) => node.published_service = Some(spec.clone()), - Err(message) => errors.push(EnrichError { code: "enrich", span: node_span, file: node.source_file.clone(), subjects: vec![node.id.clone()], message }), - }, - None => errors.push(EnrichError { + match weft_core::access::spec::published_recipe(catalog.all(), &node.id, service) { + Ok(spec) => node.published_service = Some(spec.clone()), + Err(message) => errors.push(EnrichError { code: "enrich", span: node_span, file: node.source_file.clone(), subjects: vec![node.id.clone()], - message: format!( - "node '{}' publishes a '{service}' connection, but no node in the \ - catalog declares that service", - node.id - ), + message, }), } } diff --git a/crates/weft-compiler/src/hash.rs b/crates/weft-compiler/src/hash.rs index 083186800..feea1392f 100644 --- a/crates/weft-compiler/src/hash.rs +++ b/crates/weft-compiler/src/hash.rs @@ -510,6 +510,23 @@ mod fs_hashes { } /// The package whose root is `root`, alone, or an error naming it. + /// The recipes of the services the package at `root` publishes, as its + /// node-test binary bakes them (`codegen::baked_published_services`). + /// They may come from another package's metadata, so a change there + /// changes what this package's tests are held to. A recipe that cannot + /// be resolved is folded as its refusal: the emit refuses it the same. + fn hash_published_services(hasher: &mut Sha256, catalog: &FsCatalog, root: &Path) -> Result<()> { + let package = catalog + .packages() + .find(|p| p.root == root) + .ok_or_else(|| anyhow::anyhow!("no package of this catalog is at {}", root.display()))?; + let baked = crate::codegen::baked_published_services(catalog, &package.node_types).unwrap_or_else(|e| format!("refused: {e}")); + hasher.update(b"published_services:\n"); + hasher.update(baked.as_bytes()); + hasher.update(b"\n"); + Ok(()) + } + fn package_at(catalog: &FsCatalog, root: &Path) -> Result { let package = catalog .packages() @@ -543,8 +560,8 @@ mod fs_hashes { } /// Content hash of everything that can change a package's node-test - /// OUTCOME: the package's own sources plus the catalog's nominal - /// type registry (the test binary bakes EVERY package's type + /// OUTCOME: the package's own sources, the recipes of the services it + /// publishes, plus the catalog's nominal type registry (the test binary bakes EVERY package's type /// declarations and resolves this package's port types through /// them, so a sibling's type edit changes this package's results). /// Excludes the worker build environment on purpose: it rebuilds the @@ -552,9 +569,10 @@ mod fs_hashes { /// real provider money. pub fn compute_node_test_outcome_hash(package_root: &Path, catalog: &FsCatalog) -> Result { let mut hasher = Sha256::new(); - hasher.update(b"weft-node-test-outcome-v2\n"); + hasher.update(b"weft-node-test-outcome-v3\n"); hash_packages(&mut hasher, &package_at(catalog, package_root)?, &DISK)?; hash_type_registry(&mut hasher, catalog); + hash_published_services(&mut hasher, catalog, package_root)?; Ok(hex(&hasher.finalize())) } @@ -568,16 +586,19 @@ mod fs_hashes { /// a compiled server is current. pub fn compute_node_test_hash(package_root: &Path, weft_root: &Path, catalog: &FsCatalog) -> Result { let mut hasher = Sha256::new(); - // v4: a test server is a program compiled on the machine, so no - // image recipe or builder base (v3: its package is named by its + // v5: it bakes the recipes of the services its package publishes + // (v4: a test server is a program compiled on the machine, so no + // image recipe or builder base; v3: its package is named by its // name, never its path). - hasher.update(b"weft-node-test-v4\n"); + hasher.update(b"weft-node-test-v5\n"); hash_packages(&mut hasher, &package_at(catalog, package_root)?, &DISK)?; // The test crate's main.rs bakes the FULL catalog type // registry (see the binary hash's identical step): a type // edit in a SIBLING package changes this package's test // binary, so it must flip this digest too. hash_type_registry(&mut hasher, catalog); + // So does a recipe the package publishes, which the binary bakes. + hash_published_services(&mut hasher, catalog, package_root)?; hash_worker_build_env(&mut hasher, weft_root, false)?; Ok(hex(&hasher.finalize())) } diff --git a/crates/weft-compiler/tests/integration/test_crate_emit.rs b/crates/weft-compiler/tests/integration/test_crate_emit.rs index 91ac4e8a5..6a6d910a9 100644 --- a/crates/weft-compiler/tests/integration/test_crate_emit.rs +++ b/crates/weft-compiler/tests/integration/test_crate_emit.rs @@ -80,7 +80,10 @@ fn emitted_test_crate_has_the_expected_shape() { ); let main_rs = std::fs::read_to_string(dir.path().join("src/main.rs")).unwrap(); - assert!(main_rs.contains("weft_engine::test_runner::main"), "{main_rs}"); + assert!( + main_rs.contains(r#"weft_engine::test_runner::main(®istry::PROJECT_CATALOG, "[]")"#), + "no slack node publishes, so no recipe is baked:\n{main_rs}" + ); assert!(main_rs.contains("TypeRegistry::install"), "baked types installed first"); let registry = std::fs::read_to_string(dir.path().join("src/registry.rs")).unwrap(); @@ -103,6 +106,38 @@ fn emitted_test_crate_has_the_expected_shape() { ); } +/// A package whose nodes publish a connection bakes the recipe of each +/// service they publish into the runner's main, resolved against the +/// whole catalog the way a project build resolves it, so the fake rig +/// holds a publish to the recipe production holds it to. +#[test] +fn emitted_test_crate_bakes_the_services_its_nodes_publish() { + let catalog = stdlib(); + let weft_root = weft_catalog::weft_repo_root().expect("weft root"); + let dir = tempfile::tempdir().expect("temp dir"); + emit_test_crate(dir.path(), &catalog, "postgres", &EmitPaths::Local { weft_root }) + .expect("emit"); + + let main_rs = std::fs::read_to_string(dir.path().join("src/main.rs")).unwrap(); + let baked = main_rs + .lines() + .find(|l| l.contains("weft_engine::test_runner::main(®istry::PROJECT_CATALOG, ")) + .expect("the runner is called with the baked services"); + let literal = baked + .trim() + .trim_start_matches("weft_engine::test_runner::main(®istry::PROJECT_CATALOG, ") + .trim_end_matches(')'); + // The literal is the Debug form of the JSON string: unescape it the + // way the compiler will, then read the recipes back. + let json: String = serde_json::from_str(literal).expect("a string literal JSON reads as well"); + let specs: Vec = serde_json::from_str(&json).expect("the baked recipes read"); + assert_eq!( + specs.iter().map(|s| s.service.as_str()).collect::>(), + vec!["postgres"], + "the database node publishes the postgres service:\n{main_rs}" + ); +} + /// Stage a minimal project holding just the slack package, so /// container-mode emission and the hash have a real project root. fn staged_slack_project() -> tempfile::TempDir { diff --git a/crates/weft-core/src/access/spec.rs b/crates/weft-core/src/access/spec.rs index 90981167a..dbfba8903 100644 --- a/crates/weft-core/src/access/spec.rs +++ b/crates/weft-core/src/access/spec.rs @@ -1992,8 +1992,81 @@ pub fn publishable_fields(spec: &AccessSpec) -> Result<&[CredentialField], Strin Ok(fields) } -/// The recipe for `service`, from whichever node's metadata declares -/// it. +/// `values` reduced to what may actually be STORED for `spec`, or the +/// reason they cannot be: every required field present and non-blank, +/// no name the service does not declare, the service's all-or-nothing +/// groups satisfied, and blanks dropped so a later read sees "missing" +/// rather than an empty secret. +/// +/// The gate for every set of values a PERSON or a NODE supplied: a +/// pasted connection, a minted one, and one a node published all +/// answer the same questions and are held to the same standard. A +/// browser consent is the one that does not come through here, and +/// cannot: its values are the provider's own answer to the token +/// exchange, not fields anybody filled in. +/// +/// A node's publish reaches it through [`publishable_connection`], +/// which holds the recipe and the values together. +pub fn storable_values( + spec: &AccessSpec, + fields: &[CredentialField], + mut values: BTreeMap, +) -> Result, String> { + for f in fields { + let empty = values.get(&f.name).is_none_or(|v| v.trim().is_empty()); + if empty && !f.optional { + return Err(format!("field '{}' is required", f.name)); + } + } + if let Some(unknown) = values.keys().find(|k| !fields.iter().any(|f| &&f.name == k)) { + return Err(format!( + "the '{}' service declares no value named '{unknown}'", + spec.service + )); + } + capability_shortfall(&spec.capabilities, |name| { + values.get(name).is_some_and(|v| !v.trim().is_empty()) + })?; + values.retain(|_, v| !v.trim().is_empty()); + Ok(values) +} + +/// What a node's publish stores: the values as the store keeps them +/// (blanks dropped), and the connection's display identity resolved +/// from them, which the published connection carries back to the node. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct PublishableConnection { + pub values: BTreeMap, + pub identity: Option, +} + +/// The whole check a node's publish passes, or the reason it cannot: +/// the recipe valid ([`AccessSpec::validate`]) and of the shape a +/// published connection has ([`publishable_fields`]), the values +/// storable ([`storable_values`]), and the identity resolvable from +/// what is stored (an identity naming an optional value left blank is +/// refused here, not shown as a broken label). +/// +/// ONE definition, because two places hold a publish to it and they +/// must not be able to disagree: the access store, on every real +/// write, and the node-test fake rig, so a test of a node that +/// publishes refuses exactly what production refuses and answers the +/// identity production answers. +pub fn publishable_connection( + spec: &AccessSpec, + values: BTreeMap, +) -> Result { + spec.validate()?; + let fields = publishable_fields(spec)?; + let values = storable_values(spec, fields, values)?; + let identity = spec.identity_of(&values)?; + Ok(PublishableConnection { values, identity }) +} + +/// The recipe a node publishing `service` carries, from whichever +/// node's metadata declares the service, held to the shape a published +/// connection can have ([`publishable_fields`]). `publisher` names the +/// publishing node in the refusal. /// /// Discovery refuses a second node claiming one service, so a catalog /// built from a node tree has exactly one answer. That is a guarantee @@ -2001,17 +2074,33 @@ pub fn publishable_fields(spec: &AccessSpec) -> Result<&[CredentialField], Strin /// another way and the first match wins, so give it the metadata a /// discovered catalog produced. /// -/// ONE definition, because two places need it and they must not be -/// able to disagree: the compiler resolves it against the whole -/// catalog to stamp a publishing node's definition, and a node-test -/// rig resolves it against the package under test. -pub fn spec_for_service<'a>( +/// `publisher` is what the refusal names the publishing node by: a +/// program's node id when a program is built, its node type when a +/// package's tests are. +/// +/// ONE definition, because two places resolve it and they must not be +/// able to disagree, both against the WHOLE catalog: the compiler, to +/// stamp a publishing node's definition, and the node-test crate's +/// emitter, to bake the recipes a package's fake tests are held to. +/// Refused at build time because a recipe a published connection +/// cannot have is an authoring mistake, and it belongs on the author's +/// screen. +pub fn published_recipe<'a>( metadata: impl IntoIterator, + publisher: &str, service: &str, -) -> Option<&'a AccessSpec> { - metadata +) -> Result<&'a AccessSpec, String> { + let spec = metadata .into_iter() .find_map(|m| m.service.as_ref().filter(|spec| spec.service == service)) + .ok_or_else(|| { + format!( + "node '{publisher}' publishes a '{service}' connection, but no node in the \ + catalog declares that service" + ) + })?; + publishable_fields(spec)?; + Ok(spec) } impl AccessSpec { @@ -2067,6 +2156,13 @@ impl AccessSpec { worker_value_names_of(&self.auth) } + /// The display identity of a connection holding `values`, from the + /// recipe's `identity` template; `None` when it declares none. A + /// name the values do not hold is refused, never shown blank. + pub fn identity_of(&self, values: &BTreeMap) -> Result, String> { + self.identity.as_ref().map(|t| t.resolve(values)).transpose() + } + /// Serde-inexpressible rules; run wherever a spec enters the /// system (metadata load, store connect). pub fn validate(&self) -> Result<(), String> { @@ -3137,6 +3233,74 @@ mod tests { ); } + /// The gate every set of values a person or a node supplied + /// passes. A blank required value is MISSING, not stored: a + /// downstream node would otherwise get an empty password and fail + /// at the provider instead of here. + #[test] + fn the_value_check_holds_a_pasted_and_a_published_connection_to_one_rule() { + // The fields come from the spec itself, so the rule and the + // recipe cannot be a mismatched pair the production callers + // could never produce. + let spec: AccessSpec = serde_json::from_value(serde_json::json!({ + "service": "pg", + "acquisition": { "kind": "static", "fields": [ + { "name": "host" }, + { "name": "port", "optional": true }, + ]}, + })) + .expect("a static spec"); + let fields = publishable_fields(&spec).expect("a static spec publishes"); + let values = |pairs: &[(&str, &str)]| -> BTreeMap { + pairs.iter().map(|(k, v)| (k.to_string(), v.to_string())).collect() + }; + let check = |vals| storable_values(&spec, fields, vals); + + check(values(&[("host", "db"), ("port", "5432")])).expect("a complete set passes"); + + let kept = check(values(&[("host", "db"), ("port", " ")])) + .expect("a blank optional passes"); + assert!(!kept.contains_key("port"), "a blank optional stores nothing"); + + let e = check(values(&[("port", "5432")])).expect_err("required missing"); + assert!(e.contains("'host' is required"), "{e}"); + + let e = check(values(&[("host", " ")])) + .expect_err("a blank required value is missing"); + assert!(e.contains("'host' is required"), "{e}"); + + let e = check(values(&[("host", "db"), ("hostname", "db")])) + .expect_err("undeclared name"); + assert!(e.contains("no value named 'hostname'"), "{e}"); + } + + /// A publish answers the identity the store would record, and an + /// identity naming an optional value left blank is refused, since + /// the stored values no longer hold it. + #[test] + fn a_publish_resolves_its_identity_from_what_is_stored() { + let spec: AccessSpec = serde_json::from_value(serde_json::json!({ + "service": "pg", + "acquisition": { "kind": "static", "fields": [ + { "name": "host", "secret": false }, + { "name": "database", "optional": true, "secret": false }, + ]}, + "identity": "{host}/{database}", + })) + .expect("a static spec"); + let values = |pairs: &[(&str, &str)]| -> BTreeMap { + pairs.iter().map(|(k, v)| (k.to_string(), v.to_string())).collect() + }; + + let published = publishable_connection(&spec, values(&[("host", "db"), ("database", "app")])) + .expect("a complete set publishes"); + assert_eq!(published.identity.as_deref(), Some("db/app")); + assert_eq!(published.values, values(&[("host", "db"), ("database", "app")])); + + publishable_connection(&spec, values(&[("host", "db"), ("database", " ")])) + .expect_err("the identity names a value the blank optional left out"); + } + /// The capability rule, all four ways: a complete group passes, a /// half-filled one names what is missing, filling none names the /// choices, and a service declaring no groups is unaffected. diff --git a/crates/weft-core/src/context/program_calls/infra.rs b/crates/weft-core/src/context/program_calls/infra.rs index f06149568..666057311 100644 --- a/crates/weft-core/src/context/program_calls/infra.rs +++ b/crates/weft-core/src/context/program_calls/infra.rs @@ -253,17 +253,13 @@ impl InfraCalls<'_> { }; loop { let end: SetupEnd = self.at.call(ProgramCall::Infra(InfraCall::Setup { run }), StopSelf::Keep).await?; - let node = &self.node; match end { SetupEnd::Completed => return Ok(()), SetupEnd::Going => { self.at.ctx.await_signal(LOOK_AGAIN).await?; } - SetupEnd::Failed | SetupEnd::Cancelled => { - return Err(WeftError::NodeFailed(format!( - "the rebake of infra '{node}' {end}: its setup run {run} did not complete; `weft runs` shows it" - ))) - } + SetupEnd::Failed { reason } => return Err(rebake_ended(&self.node, "failed", run, reason)), + SetupEnd::Cancelled { reason } => return Err(rebake_ended(&self.node, "cancelled", run, reason)), } } } @@ -281,6 +277,16 @@ impl InfraCalls<'_> { } } +/// The error a rebake whose setup run `run` ended `how` (failed, +/// cancelled) fails with: the reason the run recorded, or, when it +/// recorded none, a pointer to `weft runs`. +fn rebake_ended(node: &str, how: &str, run: crate::RunId, reason: Option) -> WeftError { + WeftError::NodeFailed(match reason { + Some(reason) => format!("the rebake of infra '{node}' {how}: {reason}"), + None => format!("the rebake of infra '{node}' {how}: its setup run {run} did not complete; `weft runs` shows it"), + }) +} + /// What reaches every infra copy of a project at once. See /// [`ProjectCalls::all_infra`]. pub struct AllInfraCalls<'a> { @@ -345,3 +351,17 @@ impl AllInfraCalls<'_> { } } } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn a_rebake_that_ended_says_its_reason_or_points_at_weft_runs() { + let run = crate::RunId::from_u128(3); + let failed = rebake_ended("db", "failed", run, Some("the extension is missing".into())); + assert_eq!(failed.to_string(), WeftError::NodeFailed("the rebake of infra 'db' failed: the extension is missing".into()).to_string()); + let cancelled = rebake_ended("db", "cancelled", run, None).to_string(); + assert!(cancelled.contains("the rebake of infra 'db' cancelled: its setup run") && cancelled.contains("`weft runs`"), "{cancelled}"); + } +} diff --git a/crates/weft-core/src/context/program_calls/mod.rs b/crates/weft-core/src/context/program_calls/mod.rs index 837ee95bf..0bf48c6de 100644 --- a/crates/weft-core/src/context/program_calls/mod.rs +++ b/crates/weft-core/src/context/program_calls/mod.rs @@ -165,7 +165,12 @@ impl<'a> ProjectCalls<'a> { } }; serde_json::from_value(value) - .map_err(|e| WeftError::NodeFailed(format!("{name}: the runtime answered a shape this node cannot read: {e}"))) + .map_err(|e| { + WeftError::NodeFailed(format!( + "{name}: the runtime answered a shape this node cannot read ({e}), which happens when the program \ + was built by an older weft, or this run was started by one: run `weft resync` and start the run again" + )) + }) } } diff --git a/crates/weft-core/src/infra/resolve.rs b/crates/weft-core/src/infra/resolve.rs index 3199df6c0..e19e579eb 100644 --- a/crates/weft-core/src/infra/resolve.rs +++ b/crates/weft-core/src/infra/resolve.rs @@ -151,6 +151,10 @@ pub enum ResolveError { #[error("infra node '{node}': container '{container}' sets {name}, which weft sets itself on \ every container (where to push the values that changed); read it instead of setting it")] ReservedEnv { node: String, container: String, name: &'static str }, + #[error("infra node '{node}': container '{container}' sets {name} to a value with a line break, \ + which a platform cannot pass in an environment variable; hand text of several lines over \ + as the container's arguments instead (one line each)")] + EnvLineBreak { node: String, container: String, name: String }, #[error("infra node '{node}': endpoint '{endpoint}' is public at path '{path}', which is not \ a plain path. It must start with '/', and every segment between slashes must be \ non-empty, not '.' or '..', and use only letters, digits, '-', '_', '.' and '~'")] @@ -286,6 +290,11 @@ pub fn resolve(spec: &InfraSpec, node: &NodeRef, tags: &BTreeMap if let Some(c) = u.containers.iter().chain(u.init_containers.iter()).find(|c| c.env.iter().any(|e| e.name == VALUES_URL_ENV)) { return Err(ResolveError::ReservedEnv { node: plain(id), container: c.name.clone(), name: VALUES_URL_ENV }); } + for c in u.containers.iter().chain(u.init_containers.iter()) { + if let Some(e) = c.env.iter().find(|e| e.value.contains(['\n', '\r'])) { + return Err(ResolveError::EnvLineBreak { node: plain(id), container: c.name.clone(), name: e.name.clone() }); + } + } } for v in &spec.volumes { check_name(id, "volume", &v.name, MAX_VOLUME)?; @@ -440,6 +449,7 @@ mod tests { }], keep_on_terminate: Vec::new(), health_rules: None, + notes: Vec::new(), } } @@ -456,6 +466,17 @@ mod tests { assert!(matches!(e, ResolveError::MissingLocalImage { .. }), "{e}"); } + /// No platform passes a line break in an environment variable, so a + /// spec that sets one is refused here, naming it, rather than when a + /// platform gets to start the container. + #[test] + fn an_environment_value_with_a_line_break_is_refused() { + let mut broken = spec(); + broken.units[0].containers[1].env.push(EnvEntry::new("SETTINGS", "a = 1\nb = 2")); + let e = resolve(&broken, &node(), &tags()).unwrap_err(); + assert!(matches!(&e, ResolveError::EnvLineBreak { container, name, .. } if container == "cred" && name == "SETTINGS"), "{e}"); + } + #[test] fn the_unit_hash_follows_what_the_unit_runs_with() { let a = resolve(&spec(), &node(), &tags()).unwrap().units[0].hash.clone(); diff --git a/crates/weft-core/src/infra/types.rs b/crates/weft-core/src/infra/types.rs index 3df48d784..7a51284e6 100644 --- a/crates/weft-core/src/infra/types.rs +++ b/crates/weft-core/src/infra/types.rs @@ -47,6 +47,14 @@ pub struct InfraSpec { /// empty list: it runs none. #[serde(default, rename = "healthRules", skip_serializing_if = "Option::is_none")] pub health_rules: Option>, + + /// What the node says a person should know about the copy it declares + /// while it runs (it runs short of something it would normally have), + /// one plain sentence each. `weft infra status` prints them as + /// warnings, beside what the host itself runs differently from what + /// was asked. Usually empty. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub notes: Vec, } /// What a terminate does with the disks the node lists in @@ -640,6 +648,7 @@ mod tests { }], keep_on_terminate: Vec::new(), health_rules: None, + notes: Vec::new(), } } diff --git a/crates/weft-core/src/node_test.rs b/crates/weft-core/src/node_test.rs index f97e772ce..3e02568f6 100644 --- a/crates/weft-core/src/node_test.rs +++ b/crates/weft-core/src/node_test.rs @@ -325,26 +325,33 @@ impl NodeTest { } } - /// Run a fake test against a fresh rig. Panics fail this one test, - /// and so does hanging: a test that never returns fails by NAME - /// after [`FAKE_TEST_DEADLINE_SECS`] instead of stalling the suite. + /// Run a fake test against a fresh rig that knows `services`, the + /// recipes of the services the package's nodes publish, so a node that + /// publishes is refused what the store would refuse. Panics fail + /// this one test, and so does hanging: a test that never returns + /// fails by NAME after [`FAKE_TEST_DEADLINE_SECS`] instead of + /// stalling the suite. /// /// The bound is here and not in each test because the shapes that /// hang are the rig's own (a cursor parked on a bus nothing closes, /// a caller nobody attaches), and without it the runner is killed /// from outside with no test named at all, which leaves bisecting /// by hand as the only way to find the culprit. - pub async fn run_fake(&self) -> WeftResult<()> { - self.run_fake_within(std::time::Duration::from_secs(FAKE_TEST_DEADLINE_SECS)).await + pub async fn run_fake(&self, services: Arc) -> WeftResult<()> { + self.run_fake_within(services, std::time::Duration::from_secs(FAKE_TEST_DEADLINE_SECS)).await } /// [`Self::run_fake`] with the bound named, so a test of the bound /// itself does not have to wait it out. - pub async fn run_fake_within(&self, deadline: std::time::Duration) -> WeftResult<()> { + pub async fn run_fake_within( + &self, + services: Arc, + deadline: std::time::Duration, + ) -> WeftResult<()> { match &self.run { TestFn::Fake(f) => { let body = futures::FutureExt::catch_unwind(std::panic::AssertUnwindSafe( - f(FakeRig::new()), + f(FakeRig::for_package(services)), )); match tokio::time::timeout(deadline, body).await { Ok(Ok(result)) => result, @@ -792,6 +799,38 @@ impl FakeState { } } +/// The recipes of the services a node package's nodes publish, by +/// service name, as the build resolves them: against the WHOLE catalog, +/// the way the compiler stamps a publishing node's definition +/// ([`crate::access::spec::published_recipe`]), baked into the node-test +/// binary by its emitter. A publish in a test is held to the recipe +/// found here, the way a real one is held to the recipe its definition +/// carries, so a package publishing a service another package declares +/// tests the same as one declaring its own. +#[derive(Debug, Clone, Default)] +pub struct PublishedServices(BTreeMap); + +impl PublishedServices { + /// These recipes, by their service. Handed two of one service, the + /// last wins; the emitter resolves one per service. + pub fn new(specs: impl IntoIterator) -> Self { + Self(specs.into_iter().map(|spec| (spec.service.clone(), spec)).collect()) + } + + /// The recipes the emitter baked, as the JSON of a `Vec`. + pub fn from_json(json: &str) -> Result { + let specs: Vec = serde_json::from_str(json) + .map_err(|e| format!("the published services baked into this test binary do not read: {e}"))?; + Ok(Self::new(specs)) + } + + /// The recipe a node publishing `service` carries, if the build + /// resolved one. + pub fn get(&self, service: &str) -> Option<&crate::access::spec::AccessSpec> { + self.0.get(service) + } +} + /// Shared state behind the fake rig and its handle. Dumb by rule: /// plain maps, append-only logs, zero provider logic. struct FakeState { @@ -825,10 +864,14 @@ struct FakeState { /// Stored values the fake's opened connections answer /// (`conn.value(name)`), keyed by service then value name. connection_values: Mutex>>, - /// Services this run published a connection for, so - /// `published_access` answers "already published" exactly when the - /// node did publish, and answers None on the first run. - published: Mutex>, + /// Services this run published a connection for, each with the + /// identity the publish resolved, so `published_access` answers + /// "already published" exactly when the node did publish (None on + /// the first run), carrying the identity a real one carries. + published: Mutex>>, + /// The recipes of the services the package's nodes publish, which + /// a publish is held to the way the store holds a real one. + services: Arc, /// Declared granted permissions by service. A service with no /// declaration skips the required-permissions check entirely (a /// real granted set can also be unknowable); a declared set is @@ -980,7 +1023,33 @@ impl FakeState { .build() } - fn new() -> Arc { + /// A publish of `service`'s connection, held to the store's own + /// check against the recipe the build resolved for it + /// ([`crate::access::spec::publishable_connection`]) and stored as + /// the store stores it: the values the next open answers, and the + /// identity a later `published_access` carries, which is returned. + /// The ONE place both the node's publish and a test's declared + /// earlier publish ([`FakeRig::published_connection`]) go through. + fn publish(&self, service: &str, values: BTreeMap) -> WeftResult> { + let spec = self.services.get(service).ok_or_else(|| { + WeftError::Config(format!( + "this node publishes a '{service}' connection, but the build resolved no recipe \ + for that service; the fake rig knows only the services the package's nodes \ + publish (a rig built with `FakeRig::new` knows none: run the node on the rig \ + the test was handed, or on `rig.fresh()`)" + )) + })?; + // The store's check (`publish_grant`), and its refusal reaches + // the node the way a real one does. + let crate::access::spec::PublishableConnection { values, identity } = + crate::access::spec::publishable_connection(spec, values) + .map_err(|e| WeftError::NodeFailed(format!("publish the '{service}' connection: {e}")))?; + self.connection_values.lock().unwrap().insert(service.to_string(), values); + self.published.lock().unwrap().insert(service.to_string(), identity.clone()); + Ok(identity) + } + + fn new(services: Arc) -> Arc { Arc::new(Self { routes: Mutex::new(HashMap::new()), requests: Mutex::new(Vec::new()), @@ -991,7 +1060,8 @@ impl FakeState { registered_signals: Mutex::new(Vec::new()), awaited_signals: Mutex::new(Vec::new()), connection_values: Mutex::new(BTreeMap::new()), - published: Mutex::new(std::collections::BTreeSet::new()), + published: Mutex::new(BTreeMap::new()), + services, connection_permissions: Mutex::new(BTreeMap::new()), output_types: Mutex::new(HashMap::new()), wired_outputs: Mutex::new(HashSet::new()), @@ -1061,10 +1131,24 @@ pub struct FakeRig { } impl FakeRig { - /// A fresh, empty rig. Built by the runner ([`NodeTest::run_fake`]); - /// tests receive it, they don't construct it. + /// A fresh, empty rig that knows no service, so a node run on it + /// cannot publish a connection. A test that needs a second rig for + /// a node that publishes takes [`Self::fresh`] off the one it was + /// handed. pub fn new() -> Self { - Self { state: FakeState::new() } + Self::for_package(Arc::default()) + } + + /// A fresh, empty rig that knows `services`, the recipes of the + /// services the package's nodes publish. Built by the runner + /// ([`NodeTest::run_fake`]); tests receive it. + pub fn for_package(services: Arc) -> Self { + Self { state: FakeState::new(services) } + } + + /// A fresh, empty rig knowing the same services as this one. + pub fn fresh(&self) -> Self { + Self::for_package(self.state.services.clone()) } /// Declare a canned 200 response: any request the node sends on a @@ -1316,7 +1400,7 @@ impl FakeRig { /// it has not published one. For a test asserting on a node that /// opens something it runs itself. pub fn published_values(&self, service: &str) -> Option> { - if !self.state.published.lock().unwrap().contains(service) { + if !self.state.published.lock().unwrap().contains_key(service) { return None; } Some( @@ -1333,12 +1417,16 @@ impl FakeRig { /// Declare that this node published `service`'s connection on an /// earlier run, holding these values. The second-run state: the /// node finds its own connection instead of asking the thing it - /// runs for credentials again. + /// runs for credentials again. Held to the check a publish is + /// ([`crate::access::spec::publishable_connection`]), so a test never + /// starts from a connection no real run could have published; a set + /// it refuses is a mistake in the test, and panics. pub fn published_connection(&self, service: &str, values: &[(&str, &str)]) { let values: BTreeMap = values.iter().map(|(k, v)| (k.to_string(), v.to_string())).collect(); - self.state.connection_values.lock().unwrap().insert(service.to_string(), values); - self.state.published.lock().unwrap().insert(service.to_string()); + self.state + .publish(service, values) + .unwrap_or_else(|e| panic!("no run could have published this '{service}' connection: {e}")); } /// Declare that this node's own infrastructure answers on `name` @@ -2535,27 +2623,19 @@ impl ContextHandle for TestHandle { /// what it published behaves as it does in production. The service /// comes from the node's own `publishes` declaration, exactly as it /// does in a real run, so a node that declares nothing fails here - /// the same way. + /// the same way. The publish passes the store's own check + /// ([`FakeState::publish`]), refused the way a real one is, stored as + /// it stores it, and answered with the identity it resolves. async fn publish_access(&self, values: BTreeMap) -> WeftResult { let service = self.published_service()?; - self.state - .connection_values - .lock() - .unwrap() - .insert(service.clone(), values); - self.state.published.lock().unwrap().insert(service.clone()); - Ok(Access::new("fake-connection", service, None)) + let identity = self.state.publish(&service, values)?; + Ok(Access::new("fake-connection", service, identity)) } async fn published_access(&self) -> WeftResult> { let service = self.published_service()?; - Ok(self - .state - .published - .lock() - .unwrap() - .contains(&service) - .then(|| Access::new("fake-connection", service, None))) + let identity = self.state.published.lock().unwrap().get(&service).cloned(); + Ok(identity.map(|identity| Access::new("fake-connection", service, identity))) } async fn open_connection( @@ -4301,6 +4381,153 @@ mod tests { ); } + /// A node that publishes a connection to something it runs itself, + /// holding the values it was built with. Its manifest publishes + /// `probedb`, which the access node below declares. + struct OwnConnectionNode { + values: &'static [(&'static str, &'static str)], + } + fn publisher_manifest() -> &'static NodeMetadata { + static MANIFEST: std::sync::OnceLock = std::sync::OnceLock::new(); + MANIFEST.get_or_init(|| { + serde_json::from_value(json!({ + "type": "RigPublisher", + "label": "Rig publisher", + "description": "test-only node", + "inputs": [], + "outputs": [{"name": "access", "type": "Access"}], + "publishes": "probedb" + })) + .expect("publisher manifest") + }) + } + fn probedb_access_manifest() -> &'static NodeMetadata { + static MANIFEST: std::sync::OnceLock = std::sync::OnceLock::new(); + MANIFEST.get_or_init(|| { + serde_json::from_value(json!({ + "type": "RigProbeDbAccess", + "label": "Rig probe database access", + "description": "test-only node", + "inputs": [], + "outputs": [{"name": "access", "type": "Access"}], + "service": { + "service": "probedb", + "acquisition": { "kind": "static", "fields": [ + { "name": "host", "secret": false }, + { "name": "port", "optional": true, "secret": false }, + ]}, + } + })) + .expect("access manifest") + }) + } + impl crate::node::NodeManifest for OwnConnectionNode { + fn manifest(&self) -> &'static NodeMetadata { + publisher_manifest() + } + } + #[async_trait::async_trait] + impl Node for OwnConnectionNode { + async fn run(&self, ctx: RunContext) -> WeftResult<()> { + let values = + self.values.iter().map(|(k, v)| (k.to_string(), v.to_string())).collect(); + ctx.publish_access(values).await?; + Ok(()) + } + } + + /// The services the publisher's package publishes, as the build + /// resolves them: the recipe the access node declares. + fn probedb_package() -> Arc { + Arc::new(PublishedServices::new([probedb_access_manifest() + .service + .clone() + .expect("the access node declares the service")])) + } + + /// A fake publish is held to the service's recipe exactly as the + /// store holds a real one: a value the service does not declare is + /// refused by name, and a declared set is stored as the store stores + /// it, blanks dropped. Before this, the rig stored anything, so a + /// node test passed on a publish production refused. + #[tokio::test] + async fn a_fake_publish_is_held_to_the_values_its_service_declares() { + let rig = FakeRig::for_package(probedb_package()); + let err = rig + .run(&OwnConnectionNode { values: &[("host", "db"), ("watchDatabase", "db_watch")] }, json!({})) + .await + .result + .expect_err("an undeclared value is refused") + .to_string(); + assert!(err.contains("'probedb' service declares no value named 'watchDatabase'"), "{err}"); + assert!(rig.published_values("probedb").is_none(), "a refused publish stores nothing"); + + let rig = rig.fresh(); + rig.run(&OwnConnectionNode { values: &[("host", "db"), ("port", " ")] }, json!({})) + .await + .ok() + .expect("a declared set publishes"); + assert_eq!( + rig.published_values("probedb"), + Some(BTreeMap::from([("host".to_string(), "db".to_string())])), + "stored as the store stores it, the blank optional dropped" + ); + } + + /// A node that publishes and hands on what the publish answered. + struct EmitsItsPublish; + impl crate::node::NodeManifest for EmitsItsPublish { + fn manifest(&self) -> &'static NodeMetadata { + publisher_manifest() + } + } + #[async_trait::async_trait] + impl Node for EmitsItsPublish { + async fn run(&self, ctx: RunContext) -> WeftResult<()> { + let access = ctx.publish_access(BTreeMap::from([("host".to_string(), "db".to_string())])).await?; + ctx.pulse_downstream(NodeOutput::new().set("access", access.to_value())).await + } + } + + /// A fake publish answers the connection's identity the way the + /// store does, resolved from the values it stored, so a node that + /// shows or passes on its own connection sees in a test what it + /// sees on an install. + #[tokio::test] + async fn a_fake_publish_answers_the_identity_a_real_one_does() { + let spec: crate::access::spec::AccessSpec = serde_json::from_value(json!({ + "service": "probedb", + "acquisition": { "kind": "static", "fields": [ + { "name": "host", "secret": false }, + ]}, + "identity": "{host}", + })) + .expect("a static spec"); + let rig = FakeRig::for_package(Arc::new(PublishedServices::new([spec]))); + let outcome = rig.run(&EmitsItsPublish, json!({})).await.ok().expect("the publish passes"); + let access = Access::from_value(&outcome.outputs["access"]).expect("the node emitted its access"); + assert_eq!(access.identity(), Some("db")); + } + + /// A node publishing a service the build resolved no recipe for has + /// nothing to be held to, so the rig refuses loudly instead of + /// storing whatever it was handed. + #[tokio::test] + async fn a_fake_publish_to_a_service_without_a_resolved_recipe_fails_loudly() { + let rig = FakeRig::new(); + let err = rig + .run(&OwnConnectionNode { values: &[("host", "db")] }, json!({})) + .await + .result + .expect_err("no recipe, no publish"); + assert!(matches!(err, WeftError::Config(_)), "{err}"); + let err = err.to_string(); + assert!( + err.contains("'probedb'") && err.contains("services the package's nodes publish"), + "{err}" + ); + } + /// A node that publishes through `ctx.run`, the exactly-once step: /// the provider call is the step's work, and the emitted id is /// whatever the step answered. @@ -4932,7 +5159,7 @@ mod tests { assert_eq!(1, 2, "fake body assertion"); Ok(()) }); - let err = fake.run_fake().await.expect_err("async panic too").to_string(); + let err = fake.run_fake(Arc::default()).await.expect_err("async panic too").to_string(); assert!(err.contains("panicked"), "{err}"); } @@ -4947,7 +5174,7 @@ mod tests { Ok(()) }); let err = hangs - .run_fake_within(std::time::Duration::from_millis(20)) + .run_fake_within(Arc::default(), std::time::Duration::from_millis(20)) .await .expect_err("a hang is a failure") .to_string(); @@ -4964,7 +5191,7 @@ mod tests { assert!(err.contains("fake-tier, not basic"), "{err}"); let basic = NodeTest::basic("b", || Ok(())); - let err = basic.run_fake().await.expect_err("basic is not fake").to_string(); + let err = basic.run_fake(Arc::default()).await.expect_err("basic is not fake").to_string(); assert!(err.contains("basic-tier, not fake"), "{err}"); let err = basic .run_live(LiveRig::new( diff --git a/crates/weft-core/src/program/infra.rs b/crates/weft-core/src/program/infra.rs index 6453d0f44..97b17944b 100644 --- a/crates/weft-core/src/program/infra.rs +++ b/crates/weft-core/src/program/infra.rs @@ -163,14 +163,22 @@ pub enum RebakeAnswer { NothingToSetUp, } -crate::wire_enum! { - /// Where an infra setup run stands ([`InfraCall::Setup`]). - pub enum SetupEnd { - Going = "going", - Completed = "completed", - Failed = "failed", - Cancelled = "cancelled", - } +/// Where an infra setup run stands ([`InfraCall::Setup`]): still going, or +/// how it ended, with the reason its run recorded for a failure or a +/// cancel (none when the run recorded none). +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(tag = "end", rename_all = "snake_case")] +pub enum SetupEnd { + Going, + Completed, + Failed { + #[serde(default, skip_serializing_if = "Option::is_none")] + reason: Option, + }, + Cancelled { + #[serde(default, skip_serializing_if = "Option::is_none")] + reason: Option, + }, } #[cfg(test)] @@ -217,5 +225,19 @@ mod tests { assert_eq!(serde_json::to_value(InfraCancelAnswer { cancelled: true }).unwrap(), serde_json::json!({ "cancelled": true })); } - crate::wire_enum_roundtrip_tests!(SetupEnd); + #[test] + fn a_setup_end_carries_its_reason_under_its_tag() { + let cases = [ + (SetupEnd::Going, serde_json::json!({ "end": "going" })), + (SetupEnd::Completed, serde_json::json!({ "end": "completed" })), + (SetupEnd::Failed { reason: Some("the extension is missing".into()) }, serde_json::json!({ "end": "failed", "reason": "the extension is missing" })), + (SetupEnd::Failed { reason: None }, serde_json::json!({ "end": "failed" })), + (SetupEnd::Cancelled { reason: Some("Cancelled by user".into()) }, serde_json::json!({ "end": "cancelled", "reason": "Cancelled by user" })), + (SetupEnd::Cancelled { reason: None }, serde_json::json!({ "end": "cancelled" })), + ]; + for (end, wire) in cases { + assert_eq!(serde_json::to_value(&end).unwrap(), wire); + assert_eq!(serde_json::from_value::(wire).unwrap(), end); + } + } } diff --git a/crates/weft-core/src/shared.rs b/crates/weft-core/src/shared.rs index 8678fa2b7..504c3e866 100644 --- a/crates/weft-core/src/shared.rs +++ b/crates/weft-core/src/shared.rs @@ -2,8 +2,9 @@ //! (`RunContext::shared`): an open pool of database connections, a //! set of ready interpreter processes. Each is built the first time a run //! asks for it, handed to every run after that, let go once nothing used -//! it for the project's `shared_idle_seconds` worker lever, and gone with -//! the worker. A worker serves one project, so nothing here is ever +//! it for the project's `shared_idle_seconds` worker lever, and let go +//! when the worker stops ([`Shared::let_go_all`]), once its last run +//! ended, rather than held until the process is gone. A worker serves one project, so nothing here is ever //! another project's, and a project that scales to zero holds nothing. //! //! A run asks by a key ([`SharedKey`]), and the KIND of key decides @@ -251,6 +252,17 @@ impl Shared { }); } + /// Let go of every thing, used or not: the worker is stopping and its + /// last run ended, so nothing will ask again, and what a thing holds + /// (a pool's connections to a database, interpreter processes) is + /// given back now rather than whenever the process goes. A thing a + /// handle still holds goes when that handle does. + pub fn let_go_all(&self) { + let all = std::mem::take(&mut *self.things.lock().expect("shared poisoned")); + // Dropped outside the lock: letting a thing go may take its time. + drop(all); + } + /// How many things are held. pub fn len(&self) -> usize { self.things.lock().expect("shared poisoned").len() @@ -388,6 +400,31 @@ mod tests { assert!(shared.get_or_build::("pool", 0, None, || async { Ok(1) }).await.is_err(), "none is another limit too"); } + /// Everything goes at once, used recently or not, and a thing a + /// handle still holds goes when the handle does. + #[tokio::test] + async fn letting_go_of_all_drops_every_thing_and_a_held_one_with_its_handle() { + struct Dropped(Arc); + impl Drop for Dropped { + fn drop(&mut self) { + self.0.fetch_add(1, Ordering::SeqCst); + } + } + let shared = Shared::new(Duration::from_secs(60)); + let dropped = Arc::new(AtomicUsize::new(0)); + let build = |slot: &'static str| { + let dropped = dropped.clone(); + shared.get_or_build::(slot, 0, None, move || async move { Ok(Dropped(dropped)) }) + }; + drop(build("idle").await.unwrap()); + let held = build("held").await.unwrap(); + shared.let_go_all(); + assert!(shared.is_empty()); + assert_eq!(dropped.load(Ordering::SeqCst), 1, "the one nothing holds, at once"); + drop(held); + assert_eq!(dropped.load(Ordering::SeqCst), 2, "the held one, with its handle"); + } + #[test] fn the_same_values_give_the_same_fingerprint_and_a_changed_one_another() { let mut values = std::collections::BTreeMap::from([("password".to_string(), "a".to_string())]); diff --git a/crates/weft-dispatcher/src/actions/infra.rs b/crates/weft-dispatcher/src/actions/infra.rs index 42086735c..c7f800f66 100644 --- a/crates/weft-dispatcher/src/actions/infra.rs +++ b/crates/weft-dispatcher/src/actions/infra.rs @@ -800,8 +800,8 @@ async fn setup(state: &DispatcherState, actor: &Actor, project_id: uuid::Uuid, r Ok(match crate::api::run::terminal_outcome(&state.pg_pool, run).await.map_err(failed("read how the setup ended"))? { None => SetupEnd::Going, Some(TerminalOutcome::Completed) => SetupEnd::Completed, - Some(TerminalOutcome::Failed) => SetupEnd::Failed, - Some(TerminalOutcome::Cancelled) => SetupEnd::Cancelled, + Some(TerminalOutcome::Failed(reason)) => SetupEnd::Failed { reason }, + Some(TerminalOutcome::Cancelled(reason)) => SetupEnd::Cancelled { reason }, }) } diff --git a/crates/weft-dispatcher/src/actions/settings.rs b/crates/weft-dispatcher/src/actions/settings.rs index 16e198731..7812b48b5 100644 --- a/crates/weft-dispatcher/src/actions/settings.rs +++ b/crates/weft-dispatcher/src/actions/settings.rs @@ -222,14 +222,19 @@ pub async fn workers_put( state.worker_defaults.with(levers).validate().map_err(|e| (StatusCode::BAD_REQUEST, e))?; state.projects.set_worker_overrides(project, levers).await.map_err(failed("save the worker levers"))?; if let Some(program) = state.projects.running_program_identity(project).await.map_err(failed("the running program"))? { - let target = crate::delivery::worker_target(state, actor.tenant().as_str(), project, &program.binary_hash) + // Read from the rows just written: this process's held copy of the + // levers hears the change a moment after it commits. + let target = crate::delivery::current_worker_target(state, actor.tenant().as_str(), project, &program.binary_hash) .await .map_err(failed("the worker target"))?; state.runner.prepare(&target).await.map_err(|e| { (StatusCode::BAD_GATEWAY, format!("the levers are saved, and the platform refused them for the running workers: {e:#}")) })?; // A project that takes calls has its front started again with the - // new levers (a container's or a revision's are fixed at its start). + // new levers (a container's or a revision's are fixed at its start), + // which settles the front's program on them. The running program is + // settled here too: it is another one when the triggers that are on + // still fire an older build. if crate::front::address(state, project).await.map_err(failed("the project's front"))?.is_some() { let unserved = |e: &anyhow::Error| { (StatusCode::BAD_GATEWAY, format!("the levers are saved, and the project's front could not start with them: {e:#}")) @@ -239,6 +244,7 @@ pub async fn workers_put( return Err(unserved(e)); } } + crate::front::settle(state, actor.tenant().as_str(), project, &program.binary_hash).await; // A queued event a failing hand-over stopped may run on the new // levers. crate::parked_drain::retry_stopped(state, project).await; diff --git a/crates/weft-dispatcher/src/api/project.rs b/crates/weft-dispatcher/src/api/project.rs index a3d1c122d..314d050b4 100644 --- a/crates/weft-dispatcher/src/api/project.rs +++ b/crates/weft-dispatcher/src/api/project.rs @@ -867,10 +867,12 @@ async fn setup_ended( run_id: weft_core::RunId, ) -> anyhow::Result>> { use crate::api::run::TerminalOutcome; - Ok(crate::api::run::terminal_outcome(&state.pg_pool, run_id).await?.map(|outcome| match outcome { - TerminalOutcome::Completed => Ok(()), - TerminalOutcome::Cancelled => Err(SyncNotLanded::Cancelled("infra setup cancelled".into())), - _ => Err(SyncNotLanded::Failed(StatusCode::INTERNAL_SERVER_ERROR, "infra setup failed".into())), + Ok(crate::api::run::terminal_outcome(&state.pg_pool, run_id).await?.map(|outcome| { + let Some(message) = outcome.not_landed("infra setup") else { return Ok(()) }; + Err(match outcome { + TerminalOutcome::Cancelled(_) => SyncNotLanded::Cancelled(message), + _ => SyncNotLanded::Failed(StatusCode::INTERNAL_SERVER_ERROR, message), + }) })) } @@ -940,14 +942,14 @@ pub(crate) async fn await_infra_setup( if c == run_id => { return Err(SyncNotLanded::Failed( StatusCode::INTERNAL_SERVER_ERROR, - format!("infra setup failed: {error}"), + crate::api::run::not_landed("infra setup", "failed", Some(&error)), )); } Ok(crate::events::DispatcherEvent::RunCancelled { run_id: c, reason, .. }) if c == run_id => { // The user's infra-cancel (or a per-run stop) // interrupted the provisioning run. - return Err(SyncNotLanded::Cancelled(format!("infra setup cancelled: {reason}"))); + return Err(SyncNotLanded::Cancelled(crate::api::run::not_landed("infra setup", "cancelled", Some(&reason)))); } Ok(_) => {} Err(tokio::sync::broadcast::error::RecvError::Lagged(_)) => { @@ -3820,8 +3822,11 @@ async fn move_paused_runs( } /// Tell the platform to have the workers of the image `binary_hash` -/// ready (`Runner::prepare`). Workers of other images are left as they -/// are: what runs there finishes there, and they stop once idle. +/// ready (`Runner::prepare`) with the settings the project has stored, +/// then settle them on those (`crate::front::settle`): what the image +/// still runs under other settings is let go. Workers of other images are +/// left as they are: what runs there finishes there, and they stop once +/// idle. async fn prepare_running_image( state: &DispatcherState, project_id: uuid::Uuid, @@ -3829,8 +3834,10 @@ async fn prepare_running_image( ) -> Result<(), (StatusCode, String)> { let internal = |e: anyhow::Error| (StatusCode::INTERNAL_SERVER_ERROR, format!("prepare the workers: {e:#}")); let tenant = state.tenant_router.tenant_for_project(project_id).await.map_err(internal)?; - let target = crate::delivery::worker_target(state, tenant.as_str(), project_id, binary_hash).await.map_err(internal)?; - state.runner.prepare(&target).await.map_err(internal) + let target = crate::delivery::current_worker_target(state, tenant.as_str(), project_id, binary_hash).await.map_err(internal)?; + state.runner.prepare(&target).await.map_err(internal)?; + crate::front::settle(state, tenant.as_str(), project_id, binary_hash).await; + Ok(()) } @@ -4129,14 +4136,10 @@ async fn wait_for_setup( run_id: uuid::Uuid, rest: &mut SetupRest<'_>, ) -> Result<(), (StatusCode, String)> { - let ended = |outcome: crate::api::run::TerminalOutcome, why: Option| match outcome { - crate::api::run::TerminalOutcome::Completed => Ok(()), - crate::api::run::TerminalOutcome::Failed => { - Err((StatusCode::INTERNAL_SERVER_ERROR, why.map_or("trigger setup failed".into(), |error| format!("trigger setup failed: {error}")))) - } - crate::api::run::TerminalOutcome::Cancelled => { - Err((StatusCode::INTERNAL_SERVER_ERROR, why.map_or("trigger setup cancelled".into(), |reason| format!("trigger setup cancelled: {reason}")))) - } + // What the run's row recorded, or what its ending event carried. + let ended = |outcome: crate::api::run::TerminalOutcome| match outcome.not_landed("trigger setup") { + None => Ok(()), + Some(message) => Err((StatusCode::INTERNAL_SERVER_ERROR, message)), }; // The journal is authoritative: what the row says is read whenever // what was heard may have missed it (before the first event, and after @@ -4148,7 +4151,7 @@ async fn wait_for_setup( let outcome = crate::api::run::terminal_outcome(&pool, run_id) .await .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("trigger setup terminal lookup: {e}")))?; - let driven = match (outcome, resting) { + let driven = match (&outcome, resting) { (None, false) => weft_task_store::runs::in_flight(&pool, run_id) .await .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("trigger setup lookup: {e:#}")))?, @@ -4159,7 +4162,7 @@ async fn wait_for_setup( }; let (outcome, driven) = look(rest).await?; if let Some(outcome) = outcome { - return ended(outcome, None); + return ended(outcome); } if driven { rest.rest().await; @@ -4168,10 +4171,10 @@ async fn wait_for_setup( match events.recv().await.map(|record| record.event) { Ok(crate::events::DispatcherEvent::RunCompleted { run_id: c, .. }) if c == run_id => return Ok(()), Ok(crate::events::DispatcherEvent::RunFailed { run_id: c, error, .. }) if c == run_id => { - return ended(crate::api::run::TerminalOutcome::Failed, Some(error)); + return ended(crate::api::run::TerminalOutcome::Failed(Some(error))); } Ok(crate::events::DispatcherEvent::RunCancelled { run_id: c, reason, .. }) if c == run_id => { - return ended(crate::api::run::TerminalOutcome::Cancelled, Some(reason)); + return ended(crate::api::run::TerminalOutcome::Cancelled(Some(reason))); } // Its first event after its birth: a worker took it. Ok(event) if event.run_id() == Some(run_id) && !matches!(event, crate::events::DispatcherEvent::RunStarted { .. }) => { @@ -4182,7 +4185,7 @@ async fn wait_for_setup( Err(tokio::sync::broadcast::error::RecvError::Lagged(_)) => { let (outcome, driven) = look(rest).await?; if let Some(outcome) = outcome { - return ended(outcome, None); + return ended(outcome); } if driven { rest.rest().await; diff --git a/crates/weft-dispatcher/src/api/run.rs b/crates/weft-dispatcher/src/api/run.rs index ed141f3c0..e14c846b4 100644 --- a/crates/weft-dispatcher/src/api/run.rs +++ b/crates/weft-dispatcher/src/api/run.rs @@ -464,26 +464,69 @@ pub async fn program_for_cancel( } /// The terminal outcome recorded for a run, if any: its run row's -/// `outcome`, stamped by the write that records its ending. `None` means the run is -/// still in flight. Used both for cancel-dedup and as the source of -/// truth when the in-RAM event bus drops events (broadcast `Lagged`). +/// `outcome`, with the `error` it failed with or the `cancel_cause` it +/// was cancelled for, all stamped by the write that records its ending. +/// `None` means the run is still in flight. Used both for cancel-dedup +/// and as the source of truth when the in-RAM event bus drops events +/// (broadcast `Lagged`). pub(crate) async fn terminal_outcome( pool: &weft_task_store::db::Pool, run_id: RunId, ) -> anyhow::Result> { - let outcome: Option> = pool.scalar_opt("SELECT outcome FROM run WHERE run_id = $1", &[&run_id]).await?; - Ok(outcome.flatten().map(|outcome| match outcome.as_str() { - "completed" => TerminalOutcome::Completed, - "cancelled" => TerminalOutcome::Cancelled, - _ => TerminalOutcome::Failed, + type Ending = (Option, Option, Option>); + let row: Option = pool + .opt("SELECT outcome, error, cancel_cause FROM run WHERE run_id = $1", &[&run_id]) + .await?; + Ok(row.and_then(|(outcome, error, cause)| { + outcome.map(|outcome| TerminalOutcome::of(&outcome, error, cause.map(|cause| cause.0))) })) } -#[derive(Debug, Clone, Copy, PartialEq, Eq)] +/// How a run ended, with the reason its row recorded for a failure or a +/// cancel (`None` when the row holds none). +#[derive(Debug, Clone, PartialEq, Eq)] pub(crate) enum TerminalOutcome { Completed, - Failed, - Cancelled, + Failed(Option), + Cancelled(Option), +} + +impl TerminalOutcome { + /// The outcome a run row's `outcome`, `error` and `cancel_cause` + /// columns say. Any outcome other than `completed` and `cancelled` + /// reads as a failure. + pub(crate) fn of(outcome: &str, error: Option, cause: Option) -> Self { + match outcome { + "completed" => Self::Completed, + "cancelled" => Self::Cancelled(cause.map(|cause| cause.to_string())), + _ => Self::Failed(error), + } + } + + /// What a person reads when `what` (a run doing some job, like + /// "infra setup") did not complete: ` failed: ` or + /// ` cancelled: `, the bare ` failed` or + /// ` cancelled` when the row recorded no reason. `None` when it + /// completed. + pub(crate) fn not_landed(&self, what: &str) -> Option { + let (ended, reason) = match self { + Self::Completed => return None, + Self::Failed(error) => ("failed", error), + Self::Cancelled(cause) => ("cancelled", cause), + }; + Some(not_landed(what, ended, reason.as_deref())) + } +} + +/// What a person reads when `what` (a run doing some job, like "infra +/// setup") `ended` ("failed", "cancelled") for `reason`, or bare when it +/// gave none: one wording, whether the ending was heard on the bus or read +/// from the run's row ([`TerminalOutcome::not_landed`]). +pub(crate) fn not_landed(what: &str, ended: &str, reason: Option<&str>) -> String { + match reason { + Some(reason) => format!("{what} {ended}: {reason}"), + None => format!("{what} {ended}"), + } } /// THE reading of a runs listing as a run store query, shared by every @@ -642,3 +685,30 @@ mod cancel_tests { assert!(matches!(&writes[1], RunEvent::CostReported { charge: Some(charge), amount_usd: None, .. } if charge == "job-1")); } } + +#[cfg(test)] +mod outcome_tests { + use super::*; + + /// A run row's ending columns read as the outcome they say, and a + /// run that did not complete is told with the reason its row kept, + /// or bare when it kept none. + #[test] + fn a_run_row_ending_reads_with_its_reason() { + let completed = TerminalOutcome::of("completed", None, None); + assert_eq!(completed, TerminalOutcome::Completed); + assert_eq!(completed.not_landed("infra setup"), None); + + let failed = TerminalOutcome::of("failed", Some("the extension is missing".into()), None); + assert_eq!(failed, TerminalOutcome::Failed(Some("the extension is missing".into()))); + assert_eq!(failed.not_landed("infra setup").as_deref(), Some("infra setup failed: the extension is missing")); + assert_eq!(TerminalOutcome::of("failed", None, None).not_landed("infra setup").as_deref(), Some("infra setup failed")); + + let cancelled = TerminalOutcome::of("cancelled", None, Some(CancelCause::User)); + assert_eq!(cancelled.not_landed("trigger setup").as_deref(), Some("trigger setup cancelled: Cancelled by user")); + assert_eq!( + TerminalOutcome::of("cancelled", None, None).not_landed("infra setup").as_deref(), + Some("infra setup cancelled") + ); + } +} diff --git a/crates/weft-dispatcher/src/delivery.rs b/crates/weft-dispatcher/src/delivery.rs index c6b04ae68..470ce78b5 100644 --- a/crates/weft-dispatcher/src/delivery.rs +++ b/crates/weft-dispatcher/src/delivery.rs @@ -337,7 +337,9 @@ impl std::error::Error for NotRegistered {} /// the install's. [`NotRegistered`] when the project is gone; a /// [`Refusal`] naming the lever when the settings together are a shape no /// platform could run (the install's changed under a project's own), a -/// definite no that changing them answers. +/// definite no that changing them answers. Read from this process's held +/// copy of the levers, which a change reaches a moment after it commits: +/// right for a call to the workers, whose next one follows the change. pub async fn worker_target( state: &DispatcherState, tenant: &str, @@ -348,9 +350,45 @@ pub async fn worker_target( // absence is read from the rows themselves. let overrides = match state.held.worker_overrides.held(&project).filter(|held| held.is_some()) { Some(held) => held, - None => state.held.worker_overrides.load_fresh(project, || state.projects.worker_overrides(project)).await?, + None => stored_overrides(state, project).await?, }; - let overrides = overrides.as_ref().as_ref().ok_or(NotRegistered(project))?; + target_with(state, tenant, project, binary_hash, &overrides) +} + +/// [`worker_target`] with the levers as the project has them stored now, +/// read from the rows rather than the held copy: for what decides which +/// settings the project's workers run (its front, `Runner::prepare`, +/// `Runner::settle`), where a copy from before a change would bring the +/// old settings back. +pub async fn current_worker_target( + state: &DispatcherState, + tenant: &str, + project: uuid::Uuid, + binary_hash: &str, +) -> anyhow::Result { + let overrides = stored_overrides(state, project).await?; + target_with(state, tenant, project, binary_hash, &overrides) +} + +/// `project`'s levers as its rows hold them now (kept in the held copy for +/// the calls after), `None` when the project is gone. +async fn stored_overrides( + state: &DispatcherState, + project: uuid::Uuid, +) -> anyhow::Result>> { + state.held.worker_overrides.load_fresh(project, || state.projects.worker_overrides(project)).await +} + +/// The target of `project`'s workers at the image of `binary_hash` with +/// `overrides` (its levers, `None` when the project is gone). +fn target_with( + state: &DispatcherState, + tenant: &str, + project: uuid::Uuid, + binary_hash: &str, + overrides: &Option, +) -> anyhow::Result { + let overrides = overrides.as_ref().ok_or(NotRegistered(project))?; Ok(weft_platform_traits::WorkerTarget { tenant: tenant.to_string(), project, diff --git a/crates/weft-dispatcher/src/front.rs b/crates/weft-dispatcher/src/front.rs index ac1e18006..f0a5eca82 100644 --- a/crates/weft-dispatcher/src/front.rs +++ b/crates/weft-dispatcher/src/front.rs @@ -67,7 +67,11 @@ pub(crate) async fn serve(state: &DispatcherState, project: uuid::Uuid, port: Op .await } -/// [`serve`]'s work, under its lock. +/// [`serve`]'s work, under its lock. The front runs the settings the +/// project has stored now ([`crate::delivery::current_worker_target`]), +/// never a held copy from before a change. It settles nothing: the reaper +/// comes through here every half minute, and the workers are settled +/// where the settings or the build change ([`settle`]). async fn put_in_place(state: &DispatcherState, project: uuid::Uuid, port: Option) -> anyhow::Result> { if !takes_work(state, project).await? { return Ok(None); @@ -76,10 +80,21 @@ async fn put_in_place(state: &DispatcherState, project: uuid::Uuid, port: Option return Ok(None); }; let tenant = state.projects.tenant_for(project).await?.ok_or_else(|| anyhow::anyhow!("project {project} is not registered"))?; - let target = crate::delivery::worker_target(state, &tenant, project, &binary).await?; + let target = crate::delivery::current_worker_target(state, &tenant, project, &binary).await?; + Ok(Some(open(state, project, port, &target).await?)) +} + +/// Make `target` the project's front (`Runner::front`), on the project's +/// port when the install gives projects one. +async fn open( + state: &DispatcherState, + project: uuid::Uuid, + port: Option, + target: &weft_platform_traits::WorkerTarget, +) -> anyhow::Result { let Some(ports) = &state.project_ports else { anyhow::ensure!(port.is_none(), "this install gives projects no port: a project's own address is the platform's"); - return Ok(Some(state.runner.front(&target, None).await?)); + return state.runner.front(target, None).await; }; let mut at = ports.port_of(&state.pg_pool, project, port).await?; // A port another program took moves the project to a free one, except @@ -89,8 +104,8 @@ async fn put_in_place(state: &DispatcherState, project: uuid::Uuid, port: Option // in between makes another. const MOVES: usize = 3; for _ in 0..MOVES { - match state.runner.front(&target, Some(at)).await { - Ok(address) => return Ok(Some(address)), + match state.runner.front(target, Some(at)).await { + Ok(address) => return Ok(address), Err(e) => match weft_platform_traits::PortTaken::of(&e) { Some(taken) if port.is_some() => { anyhow::bail!("port {taken}, which `weft activate --port` asked for, was taken by another program as the project activated") @@ -106,6 +121,30 @@ async fn put_in_place(state: &DispatcherState, project: uuid::Uuid, port: Option anyhow::bail!("every free port tried for project {project} was taken by another program before its front could open it") } +/// Settle the project's workers at the image of `binary_hash` on the +/// settings it has stored (`Runner::settle`): what the platform keeps +/// running of that image under other settings is let go. Under the +/// project's front lock, the one every change of its front takes, so a +/// settling never reads settings older than a front a sibling process +/// just put in place. Logged, never raised: the stored settings already +/// serve, and what lingers takes no calls. +pub(crate) async fn settle(state: &DispatcherState, tenant: &str, project: uuid::Uuid, binary_hash: &str) { + let settled = locked(state, project, || async { + let target = crate::delivery::current_worker_target(state, tenant, project, binary_hash).await?; + state.runner.settle(&target).await + }) + .await; + if let Err(e) = settled { + tracing::error!( + target: "weft_dispatcher::front", + %project, + error = %format!("{e:#}"), + "could not let go of all the project's workers running under settings it no longer has; they take no \ + calls and scale to zero once nothing calls them, and the next settings change or activation tries again" + ); + } +} + /// The binary that answers `project`'s routes: the one its triggers that /// are on fire, else the registered build's; `None` when it runs no /// program yet. diff --git a/crates/weft-e2e/fixtures/postgres_db/src/main.weft b/crates/weft-e2e/fixtures/postgres_db/src/main.weft index b4aa134b8..e00675449 100644 --- a/crates/weft-e2e/fixtures/postgres_db/src/main.weft +++ b/crates/weft-e2e/fixtures/postgres_db/src/main.weft @@ -13,7 +13,9 @@ # ordering wire, any value on it means go); `read` waits on `write` # through the port it reads. -db = PostgresDatabase +# Named with a dash, which the pooler reads as written, so clients sign +# in by the database's own name. +db = PostgresDatabase { database: "notes-db" } create = PostgresExecuteQuery { query: "CREATE TABLE IF NOT EXISTS notes (id serial primary key, body text, tag text)" @@ -57,3 +59,15 @@ optional.photo = maybe.photo absent = Debug absent.data = optional.rows + +# One firing is one session, whatever stands between the program and +# Postgres: a setting made by the script's first statement is still +# there for its last. And the program signs in as its own role, which is +# no superuser. +session = PostgresExecuteQuery { + query: "SET application_name = 'one session'; SELECT current_setting('application_name') AS name, current_user AS signed_in, (SELECT rolsuper FROM pg_roles WHERE rolname = current_user) AS superuser" +} +session.account = db.access + +same = Debug +same.data = session.rows diff --git a/crates/weft-e2e/fixtures/postgres_watch/src/main.weft b/crates/weft-e2e/fixtures/postgres_watch/src/main.weft new file mode 100644 index 000000000..603905c54 --- /dev/null +++ b/crates/weft-e2e/fixtures/postgres_watch/src/main.weft @@ -0,0 +1,24 @@ +# A query kept live through the database's pooler. A caller of the route +# gets the count of notes as ndjson, a line each time it changes. With a +# channel set, the watch LISTENs on its one connection and re-runs the +# query on every NOTIFY, and between those it sits idle far longer than +# the minute after which the database closes an idle session, so it only +# keeps watching because it lifts that limit for its own session. +# +# The database's name has a dot in it, which the pooler cannot read as +# written, so clients reach it through a door made from the name. + +db = PostgresDatabase { database: "watch.me", reachable: true } + +feed = Route { path: "__E2E_PATH__/count", method: "GET" } + +count = PostgresWatchQuery { + query: "SELECT count(*)::int AS n FROM notes" + channel: "notes_changed" + intervalSecs: 600 +} +count.account = db.access +count._should_flow = feed.path + +events = Stream { format: "ndjson" } +events.bus = count.row diff --git a/crates/weft-e2e/fixtures/postgres_watch/weft.toml b/crates/weft-e2e/fixtures/postgres_watch/weft.toml new file mode 100644 index 000000000..1434e896b --- /dev/null +++ b/crates/weft-e2e/fixtures/postgres_watch/weft.toml @@ -0,0 +1,4 @@ +[package] +name = "e2e_postgres_watch" +id = "00000000-0000-0000-0000-000000000058" +version = "0.1.0" diff --git a/crates/weft-e2e/tests/e2e/postgres.rs b/crates/weft-e2e/tests/e2e/postgres.rs index 21d26eb4a..1d7ac784e 100644 --- a/crates/weft-e2e/tests/e2e/postgres.rs +++ b/crates/weft-e2e/tests/e2e/postgres.rs @@ -13,6 +13,16 @@ //! connection was stored the first time: by then the database //! refuses to hand its password over again. The row count growing //! also proves the data is really in a database; +//! - one firing's statements share one session through the +//! database's pooler, and the program signs in as its own role, +//! never the superuser; +//! - a client past the connections the pooler holds waits its turn +//! and is served once one frees up, instead of being refused; +//! - a watching query reached through the pooler keeps its one +//! connection past the minute after which the database closes an +//! idle session, and a write still fires it; +//! - a database named with a dash keeps its name, and one the pooler +//! cannot read as written is reached through a door made from it; //! - terminating takes the connection away with the database. //! //! Needs an install, no external service and no credentials. @@ -39,6 +49,59 @@ async fn postgres_grants_of_this_project(project: &Project) -> Result): ...; of connections open`). +async fn pooler_query_connections(project: &Project) -> Result { + let card: serde_json::Value = serde_json::from_str(project.weft(&["infra", "show", "db", "--json"]).await?.trim())?; + let pooler = card["items"] + .as_array() + .and_then(|items| items.iter().find(|i| i["label"] == "Pooler")) + .and_then(|i| i["value"].as_str()) + .ok_or_else(|| anyhow::anyhow!("the card has no Pooler item: {card}"))?; + let queries = pooler.lines().find(|l| l.starts_with("queries")).ok_or_else(|| anyhow::anyhow!("no queries line: {pooler}"))?; + let size = queries + .rsplit_once(" connections open") + .and_then(|(before, _)| before.rsplit_once(" of ")) + .map(|(_, size)| size.trim().parse::()) + .ok_or_else(|| anyhow::anyhow!("no pool size in: {queries}"))??; + Ok(size) +} + +/// How a client outside weft signs in to the database `node` of +/// `project`, whose `reachable` input is on: its door from `weft infra +/// list-doors`, and the user, the database and a fresh password from its +/// card, written into the env file `file` by `weft infra env` (starting +/// the database took the password it had). Checks the user is the +/// program's role, never the superuser. +async fn outside_client(project: &Project, node: &str, file: &str) -> Result { + let listed: weft_core::infra::wire::DoorsResponse = serde_json::from_str(project.weft(&["infra", "list-doors", "--json"]).await?.trim())?; + let door = listed + .doors + .iter() + .find(|d| d.copy.node == node && d.endpoint == "sql") + .ok_or_else(|| anyhow::anyhow!("the database's sql endpoint has a door: {listed:?}"))?; + let (host, port) = door.address.rsplit_once(':').ok_or_else(|| anyhow::anyhow!("the door is host:port: {door:?}"))?; + let port: u16 = port.parse()?; + project.weft(&["infra", "press", node, "reset_password"]).await?; + let env = project.dir().join(file); + let into = env.to_string_lossy().to_string(); + project + .weft(&["infra", "env", node, "--into", &into, "--set", "PG_PASSWORD=Password", "--set", "PG_USER=User", "--set", "PG_DB=Database"]) + .await?; + let text = std::fs::read_to_string(&env)?; + let value = |name: &str| -> Result { + text.lines() + .find_map(|l| l.strip_prefix(&format!("{name}="))) + .map(|v| v.trim().to_string()) + .filter(|v| !v.is_empty()) + .ok_or_else(|| anyhow::anyhow!("the env file does not set {name}")) + }; + anyhow::ensure!(value("PG_USER")? == "app", "an outside client signs in as the program's role, never the superuser"); + let mut config = db::Config::new(); + config.host(host).port(port).user(value("PG_USER")?).password(value("PG_PASSWORD")?).dbname(value("PG_DB")?); + Ok(config) +} + /// Run the graph once and hand back the settled run to assert on. async fn round_trip(project: &mut Project) -> Result { let settled = run::run_and_settle(project).await?; @@ -66,6 +129,7 @@ async fn a_project_runs_its_own_postgres_and_talks_to_it() -> Result<()> { // sent nothing, and the row is written with an empty column: one // node, no branch, no split insert. first.assert_input("absent", "data", &json!([{ "body": "no picture", "tag": null }]))?; + first.assert_input("same", "data", &json!([{ "name": "one session", "signed_in": "app", "superuser": false }]))?; // The second run: the database no longer hands its password out, // so this only works through the stored connection. Two rows now, @@ -156,7 +220,8 @@ async fn the_password_goes_into_an_env_file_and_a_reset_gets_a_new_one() -> Resu } let mode = std::fs::metadata(&env)?.permissions().mode() & 0o777; anyhow::ensure!(mode == 0o600, "a new env file should be the owner's only, got {mode:o}"); - // The program's own next run picks the new password up by itself. + // The press reached the program's connection too, so its next run signs + // in with the new password. round_trip(&mut project).await?; // Another reset replaces only its own line. @@ -178,7 +243,9 @@ async fn the_password_goes_into_an_env_file_and_a_reset_gets_a_new_one() -> Resu /// `reachable` input on, find the door with `weft infra list-doors`, write /// the user, the database and a fresh password into an env file with /// `weft infra env`, and open Postgres from outside weft with nothing -/// else. The program's own run keeps working beside it. +/// else. More clients than the pooler holds connections for all get in: +/// the one past them waits, the card says so, and it is served once +/// another lets go. The program's own run keeps working beside it. #[tokio::test] async fn a_client_outside_weft_opens_the_database_through_its_door() -> Result<()> { let disp = ensure::up().await?; @@ -187,39 +254,119 @@ async fn a_client_outside_weft_opens_the_database_through_its_door() -> Result<( project.set_node_config("db", "reachable", "true")?; infra::start_and_wait_running(&mut project, "db").await?; - let listed: weft_core::infra::wire::DoorsResponse = serde_json::from_str(project.weft(&["infra", "list-doors", "--json"]).await?.trim())?; - let door = listed - .doors - .iter() - .find(|d| d.copy.node == "db" && d.endpoint == "sql") - .ok_or_else(|| anyhow::anyhow!("the database's sql endpoint has a door: {listed:?}"))?; - let (host, port) = door.address.rsplit_once(':').ok_or_else(|| anyhow::anyhow!("the door is host:port: {door:?}"))?; - let (host, port): (String, u16) = (host.to_string(), port.parse()?); - - // Starting the database took its password, so a fresh one is minted - // for the outside client first. - project.weft(&["infra", "press", "db", "reset_password"]).await?; - let env = project.dir().join("front.env"); - let into = env.to_string_lossy().to_string(); - project - .weft(&["infra", "env", "db", "--into", &into, "--set", "PG_PASSWORD=Password", "--set", "PG_USER=User", "--set", "PG_DB=Database"]) - .await?; - let text = std::fs::read_to_string(&env)?; - let value = |name: &str| -> Result { - text.lines() - .find_map(|l| l.strip_prefix(&format!("{name}="))) - .map(|v| v.trim().to_string()) - .filter(|v| !v.is_empty()) - .ok_or_else(|| anyhow::anyhow!("the env file does not set {name}")) - }; - let mut config = db::Config::new(); - config.host(&host).port(port).user(value("PG_USER")?).password(value("PG_PASSWORD")?).dbname(value("PG_DB")?); + let config = outside_client(&project, "db", "front.env").await?; + anyhow::ensure!(config.get_dbname() == Some("notes-db"), "a name with a dash is the database's own: {:?}", config.get_dbname()); let session = db::Session::connect(&config).await?; let one: i32 = session.scalar("SELECT 1", &[]).await?; anyhow::ensure!(one == 1, "the database answered through its door"); drop(session); + // Every connection the pooler holds for queries, taken: in session + // pooling a client keeps its connection for as long as it is + // connected. + let pooled = pooler_query_connections(&project).await?; + let mut held = Vec::new(); + for _ in 0..pooled { + let session = db::Session::connect(&config).await?; + let _: i32 = tokio::time::timeout(std::time::Duration::from_secs(60), session.scalar("SELECT 1", &[])).await??; + held.push(session); + } + // One more gets in and waits for a connection rather than being + // refused, and the card shows it waiting. + let past = db::Session::connect(&config).await?; + let waiting = tokio::spawn(async move { past.scalar::("SELECT 2", &[]).await }); + // The card is read when asked, so it is asked until the pooler has + // queued the client, for as long as a client takes to sign in at most. + let until = tokio::time::Instant::now() + std::time::Duration::from_secs(30); + let card = loop { + let card = project.weft(&["infra", "show", "db", "--json"]).await?; + if card.contains("1 waiting") || tokio::time::Instant::now() >= until { + break card; + } + tokio::time::sleep(std::time::Duration::from_millis(500)).await; + }; + anyhow::ensure!(card.contains("1 waiting"), "the card should show the client waiting: {card}"); + anyhow::ensure!(!waiting.is_finished(), "a client past the pooler's {pooled} connections was served at once"); + drop(held.pop()); + let two = tokio::time::timeout(std::time::Duration::from_secs(30), waiting).await???; + anyhow::ensure!(two == 2, "the waiting client was served once a connection came free"); + drop(held); + round_trip(&mut project).await?; infra::terminate_and_wait_gone(&project, "db").await?; project.finish().await } + +/// The next line of the ndjson answer `answer` as JSON, reading what +/// arrives until one is whole (`pending` keeps what came after it), and +/// failing after `within` with what was read. +async fn next_line(answer: &mut reqwest::Response, pending: &mut String, within: std::time::Duration) -> Result { + let until = tokio::time::Instant::now() + within; + loop { + if let Some((line, rest)) = pending.split_once('\n') { + // A blank line is the stream's keepalive, sent while the + // program has nothing new to say. + if line.trim().is_empty() { + *pending = rest.to_string(); + continue; + } + let value = serde_json::from_str(line).map_err(|e| anyhow::anyhow!("a line of the feed is not JSON ({e}): {line}"))?; + *pending = rest.to_string(); + return Ok(value); + } + let chunk = tokio::time::timeout_at(until, answer.chunk()) + .await + .map_err(|_| anyhow::anyhow!("no whole line within {}s; read so far: {pending:?}", within.as_secs()))?? + .ok_or_else(|| anyhow::anyhow!("the feed ended; read so far: {pending:?}"))?; + pending.push_str(&String::from_utf8_lossy(&chunk)); + } +} + +/// A route's caller keeps a count live through a PostgresWatchQuery +/// reached through the pooler: the first line comes at once, a write +/// with a NOTIFY sends the next, and after the watch sat idle longer than +/// the minute the database gives an idle session, a write still reaches +/// it. The database is named with a dot, so every client, the watch +/// included, reaches it through the door the pooler made from the name. +#[tokio::test] +async fn a_watching_query_keeps_watching_past_the_idle_limit() -> Result<()> { + let disp = ensure::up().await?; + let mut project = Project::prepare("postgres_watch", disp.clone()).await?; + let base = project.unique_live_path()?; + infra::start_and_wait_running(&mut project, "db").await?; + + let config = outside_client(&project, "db", "watch.env").await?; + let door = config.get_dbname().unwrap_or_default().to_string(); + anyhow::ensure!(door.starts_with("db_watch_me_"), "a name the pooler cannot read is reached by a door made from it: {door}"); + let writer = db::Session::connect(&config).await?; + // The writer sits idle through the wait below too, and the database + // would close it; it lifts the limit for its own session, as any + // client that keeps a connection idle on purpose does. + writer.batch_execute("SET idle_session_timeout = 0; CREATE TABLE notes (id serial PRIMARY KEY)").await?; + let database: String = writer.scalar("SELECT current_database()", &[]).await?; + anyhow::ensure!(database == "watch.me", "the door leads to the database under its own name, got {database}"); + let write = || writer.batch_execute("INSERT INTO notes DEFAULT VALUES; SELECT pg_notify('notes_changed', '')"); + + project.activate().await?; + let url = format!("{}/connect/{}/count", disp.base(), base.trim_start_matches('/')); + let mut answer = weft_core::net::http_client().build()?.get(&url).send().await?; + anyhow::ensure!(answer.status().is_success(), "GET {url} answered {}", answer.status()); + let mut pending = String::new(); + let soon = std::time::Duration::from_secs(60); + anyhow::ensure!(next_line(&mut answer, &mut pending, soon).await? == json!({ "n": 0 })); + write().await?; + anyhow::ensure!(next_line(&mut answer, &mut pending, soon).await? == json!({ "n": 1 })); + + // Longer than the minute after which the database closes a session + // that sits idle, and well short of the watch's own cadence, so only + // the NOTIFY can wake it. + tokio::time::sleep(std::time::Duration::from_secs(75)).await; + write().await?; + let after = next_line(&mut answer, &mut pending, soon).await?; + anyhow::ensure!(after == json!({ "n": 2 }), "the watch should still be watching after sitting idle, got {after}"); + + drop(answer); + drop(writer); + infra::terminate_and_wait_gone(&project, "db").await?; + project.finish().await +} diff --git a/crates/weft-engine/src/context.rs b/crates/weft-engine/src/context.rs index 8fe2fc660..721df5072 100644 --- a/crates/weft-engine/src/context.rs +++ b/crates/weft-engine/src/context.rs @@ -4226,6 +4226,7 @@ mod replay_tests { let runner = crate::test_rig::LiveTestRunner::start( test_clients, test_catalog(), + Default::default(), "test-instance-1".into(), "tenant-1".into(), uuid::Uuid::from_u128(0x108), diff --git a/crates/weft-engine/src/test_rig.rs b/crates/weft-engine/src/test_rig.rs index 4f5b848fa..1103e6761 100644 --- a/crates/weft-engine/src/test_rig.rs +++ b/crates/weft-engine/src/test_rig.rs @@ -70,9 +70,12 @@ pub struct LiveTestRunner { /// the already-given answer, of an earlier run's first wait. call_numbering: Arc, /// The package's catalog, so a node under test that publishes a - /// connection resolves the service's recipe exactly as it does in - /// a real run. + /// connection is known by the service it publishes. catalog: &'static dyn weft_core::NodeCatalog, + /// The recipe of each service the package's nodes publish, as the + /// build resolved it, so a node under test publishes against the + /// recipe a real run's definition carries. + services: Arc, /// What each rig runs beside its node (see `rig`), ended at `settle`. rigs: Mutex>, } @@ -95,34 +98,27 @@ impl RigKeeping { } /// The recipe a node under test publishes against, the same one the -/// compiler would have stamped onto its definition in a real project. -/// Resolved from the package's own catalog, which a node-test binary -/// carries whole. +/// compiler would have stamped onto its definition in a real project: +/// the build resolved it against the whole catalog and baked it into +/// this binary (`services`). /// /// A node that declares nothing publishes nothing, and answers `None`. -/// A node that DECLARES a service the package cannot supply is a -/// packaging mistake, and says so here rather than letting the run -/// report the node as having declared nothing: the real compile -/// resolves against the whole catalog, so a test that quietly differed -/// would pass on something production refuses. +/// A node that declares a service the build did not resolve for it is +/// a binary built some other way, and says so here rather than letting +/// the run report the node as having declared nothing. fn published_spec( catalog: &dyn weft_core::NodeCatalog, + services: &weft_core::node_test::PublishedServices, node_type: &str, ) -> Result, String> { let Some(service) = catalog.lookup(node_type).and_then(|n| n.manifest().publishes.clone()) else { return Ok(None); }; - weft_core::access::spec::spec_for_service( - catalog.all().into_iter().filter_map(|t| catalog.lookup(t)).map(|n| n.manifest()), - &service, - ) - .cloned() - .map(Some) - .ok_or_else(|| { + services.get(&service).cloned().map(Some).ok_or_else(|| { format!( - "this node publishes a '{service}' connection, but no node in its package \ - declares that service; a node test carries only its own package" + "this node publishes a '{service}' connection, but this test binary was built \ + without that service's recipe" ) }) } @@ -140,6 +136,7 @@ impl LiveTestRunner { pub async fn start( clients: EngineClients, catalog: &'static dyn weft_core::NodeCatalog, + services: Arc, replica: String, tenant_id: String, project_id: uuid::Uuid, @@ -223,6 +220,7 @@ impl LiveTestRunner { handles: Arc::new(Mutex::new(Vec::new())), call_numbering: Default::default(), catalog, + services, rigs: Mutex::new(Vec::new()), }) } @@ -245,6 +243,7 @@ impl LiveTestRunner { let call_numbering = self.call_numbering.clone(); let run_id = self.run_id; let catalog = self.catalog; + let services = self.services.clone(); let settings = self.settings; let cancellation = self.registered.flag.clone(); // ONE coordinator per rig, shared by every handle it mints: @@ -334,7 +333,7 @@ impl LiveTestRunner { }; // Resolved before the handle is built: `node_type` is moved // into the constructor, and the recipe is looked up by it. - let published = published_spec(catalog, &node_type) + let published = published_spec(catalog, &services, &node_type) .map_err(weft_core::WeftError::Config)?; let handle = RunnerHandle::new( project_id, @@ -542,6 +541,7 @@ mod tests { let runner = LiveTestRunner::start( clients, catalog.as_ref(), + Arc::default(), "replica".into(), "tenant".into(), project_id, diff --git a/crates/weft-engine/src/test_runner.rs b/crates/weft-engine/src/test_runner.rs index 009049063..81d8b39f9 100644 --- a/crates/weft-engine/src/test_runner.rs +++ b/crates/weft-engine/src/test_runner.rs @@ -1,5 +1,5 @@ //! The node-test binary's whole brain. Codegen emits a tiny `main.rs` -//! per package (`weft_engine::test_runner::main(®istry::PROJECT_CATALOG)`) +//! per package (`weft_engine::test_runner::main(®istry::PROJECT_CATALOG, ..)`) //! so every line of runner logic lives HERE, typechecked, instead of //! in generated source. //! @@ -21,10 +21,11 @@ //! failed, 2 = the runner itself could not do what was asked. use std::process::ExitCode; +use std::sync::Arc; use clap::Parser; -use weft_core::node_test::{NodeTestsListing, RunAllReport, TestReport}; +use weft_core::node_test::{NodeTestsListing, PublishedServices, RunAllReport, TestReport}; use weft_core::{NodeCatalog, TestTier}; use crate::test_rig::LiveTestRunner; @@ -91,9 +92,20 @@ impl TierArg { } /// Entry point the generated per-package `main.rs` calls. -pub fn main(catalog: &'static dyn NodeCatalog) -> ExitCode { +/// `published_services` is the JSON the emitter baked +/// (`codegen::baked_published_services`): the recipe of every service a +/// node of the package publishes, resolved against the whole catalog as +/// a project build resolves it. +pub fn main(catalog: &'static dyn NodeCatalog, published_services: &str) -> ExitCode { tracing_subscriber_init(); let args = Args::parse(); + let services = match PublishedServices::from_json(published_services) { + Ok(services) => Arc::new(services), + Err(e) => { + eprintln!("{e}"); + return ExitCode::from(2); + } + }; let runtime = match tokio::runtime::Runtime::new() { Ok(rt) => rt, Err(e) => { @@ -101,7 +113,7 @@ pub fn main(catalog: &'static dyn NodeCatalog) -> ExitCode { return ExitCode::from(2); } }; - runtime.block_on(run(catalog, args)) + runtime.block_on(run(catalog, services, args)) } /// Stderr-only logs, so stdout stays the one clean JSON channel. This @@ -145,7 +157,7 @@ fn refuse_misdeclared_live(catalog: &'static dyn NodeCatalog) -> Result<(), Stri Ok(()) } -async fn run(catalog: &'static dyn NodeCatalog, args: Args) -> ExitCode { +async fn run(catalog: &'static dyn NodeCatalog, services: Arc, args: Args) -> ExitCode { if let Err(e) = refuse_misdeclared_live(catalog) { eprintln!("{e}"); return ExitCode::from(2); @@ -166,7 +178,7 @@ async fn run(catalog: &'static dyn NodeCatalog, args: Args) -> ExitCode { Args::Run { node, test } => { let request = TestRequest { node, test, live: None }; - match run_one(catalog, request, None).await { + match run_one(catalog, &services, request, None).await { Ok(report) => { let passed = report.passed; println!( @@ -199,7 +211,7 @@ async fn run(catalog: &'static dyn NodeCatalog, args: Args) -> ExitCode { } }; let live = LiveEnv { broker_url, tenant_id, project_id, identity, run_cap: run_cap_secs.map(std::time::Duration::from_secs) }; - match serve(catalog, live, door, port).await { + match serve(catalog, services, live, door, port).await { Ok(()) => ExitCode::SUCCESS, Err(e) => { eprintln!("the test server stopped: {e:#}"); @@ -236,22 +248,25 @@ async fn run(catalog: &'static dyn NodeCatalog, args: Args) -> ExitCode { use futures::StreamExt as _; let mut reports: Vec<(usize, TestReport)> = futures::stream::iter(selected.into_iter().enumerate()) - .map(|(i, (node_type, declared))| async move { - let (name, tier) = (declared.name, declared.tier); - let result = match tier { - TestTier::Basic => { - tokio::task::spawn_blocking(move || declared.run_basic()) - .await - .unwrap_or_else(|e| { - Err(weft_core::error::WeftError::NodeFailed(format!( - "basic test task failed: {e}" - ))) - }) - } - TestTier::Fake => declared.run_fake().await, - TestTier::Live => unreachable!("run-all never selects live"), - }; - (i, report_of(node_type, name, tier, result, Vec::new())) + .map(|(i, (node_type, declared))| { + let services = services.clone(); + async move { + let (name, tier) = (declared.name, declared.tier); + let result = match tier { + TestTier::Basic => { + tokio::task::spawn_blocking(move || declared.run_basic()) + .await + .unwrap_or_else(|e| { + Err(weft_core::error::WeftError::NodeFailed(format!( + "basic test task failed: {e}" + ))) + }) + } + TestTier::Fake => declared.run_fake(services).await, + TestTier::Live => unreachable!("run-all never selects live"), + }; + (i, report_of(node_type, name, tier, result, Vec::new())) + } }) .buffer_unordered(limit) .collect() @@ -343,7 +358,12 @@ struct LiveEnv { /// Run one test. `Err` is a request that could not run at all (no such /// node or test, a tier mismatch); a test that ran and failed is a /// report with `passed: false`. -async fn run_one(catalog: &'static dyn NodeCatalog, request: TestRequest, live_env: Option<&LiveEnv>) -> Result { +async fn run_one( + catalog: &'static dyn NodeCatalog, + services: &Arc, + request: TestRequest, + live_env: Option<&LiveEnv>, +) -> Result { let TestRequest { node, test, live } = request; let node_impl = catalog.lookup(&node).ok_or_else(|| format!("no node type '{node}' in this package's registry"))?; let tests = node_impl.tests(); @@ -355,7 +375,9 @@ async fn run_one(catalog: &'static dyn NodeCatalog, request: TestRequest, live_e })?; match (declared.tier, live) { (TestTier::Basic, None) => Ok(report_of(&node, declared.name, declared.tier, declared.run_basic(), Vec::new())), - (TestTier::Fake, None) => Ok(report_of(&node, declared.name, declared.tier, declared.run_fake().await, Vec::new())), + (TestTier::Fake, None) => { + Ok(report_of(&node, declared.name, declared.tier, declared.run_fake(services.clone()).await, Vec::new())) + } (TestTier::Basic | TestTier::Fake, Some(_)) => { Err(format!("test '{test}' on node '{node}' is a free-tier test and takes no connection")) } @@ -377,6 +399,7 @@ async fn run_one(catalog: &'static dyn NodeCatalog, request: TestRequest, live_e let runner = LiveTestRunner::start( clients, catalog, + services.clone(), live.replica, env.tenant_id.clone(), env.project_id, @@ -412,13 +435,20 @@ async fn run_one(catalog: &'static dyn NodeCatalog, request: TestRequest, live_e /// `GET /_weft/tests`, until the platform stops the process. A test's /// answer is the report as JSON (200), or the reason the request could /// not run (422). -async fn serve(catalog: &'static dyn NodeCatalog, live: LiveEnv, door: crate::worker::WeftCredential, port: u16) -> anyhow::Result<()> { +async fn serve( + catalog: &'static dyn NodeCatalog, + services: Arc, + live: LiveEnv, + door: crate::worker::WeftCredential, + port: u16, +) -> anyhow::Result<()> { use axum::extract::State; use axum::http::{HeaderMap, StatusCode}; use axum::response::IntoResponse; #[derive(Clone)] struct Served { catalog: &'static dyn NodeCatalog, + services: Arc, live: std::sync::Arc, door: crate::worker::WeftCredential, } @@ -426,7 +456,7 @@ async fn serve(catalog: &'static dyn NodeCatalog, live: LiveEnv, door: crate::wo if !s.door.admits_headers(&headers) { return (StatusCode::UNAUTHORIZED, "this test server answers the install only").into_response(); } - match run_one(s.catalog, req, Some(&s.live)).await { + match run_one(s.catalog, &s.services, req, Some(&s.live)).await { Ok(report) => axum::Json(report).into_response(), Err(why) => (StatusCode::UNPROCESSABLE_ENTITY, why).into_response(), } @@ -442,7 +472,7 @@ async fn serve(catalog: &'static dyn NodeCatalog, live: LiveEnv, door: crate::wo .route("/_weft/test", axum::routing::post(test)) .route("/_weft/tests", axum::routing::get(tests)) .route("/_weft/healthz", axum::routing::get(|| async { StatusCode::OK })) - .with_state(Served { catalog, live: std::sync::Arc::new(live), door }) + .with_state(Served { catalog, services, live: std::sync::Arc::new(live), door }) .layer(axum::middleware::map_response(crate::worker::mark_worker_answer)); let listener = crate::listen::bind(port).await?; axum::serve(listener, app).await?; diff --git a/crates/weft-engine/src/worker.rs b/crates/weft-engine/src/worker.rs index a35329d5c..29420c374 100644 --- a/crates/weft-engine/src/worker.rs +++ b/crates/weft-engine/src/worker.rs @@ -1234,10 +1234,15 @@ async fn shutdown_signal() { /// Once the platform told the worker to stop ([`Worker::stopping`], which /// hands every recorded run back): wait for the runs it still drives to -/// end and every record queued to be written. No deadline: what is waited -/// on is the work already under way, and the platform's kill is the bound. +/// end, let go of what they shared, and wait for every record queued to +/// be written. No deadline: what is waited on is the work already under +/// way, and the platform's kill is the bound. async fn wind_down(worker: &Worker) { worker.background.wait_zero().await; + // No run is left to use them: a pool's connections are given back to + // the database now, while the records go out, rather than held until + // the process is gone. + worker.clients.shared.let_go_all(); // Every run's rows still queued go out before the process does: a // fast run that ended leaves its record trailing behind it. worker.clients.writer.written().await; @@ -2354,6 +2359,25 @@ mod tests { drop((waiting, quiet, journal)); } + /// A stopping worker lets go of what its runs shared once the last one + /// ended, and not before: a pool's connections are given back to the + /// database then, rather than when the process goes. + #[tokio::test] + async fn winding_down_lets_go_of_what_the_runs_shared_once_they_ended() { + let (worker, _broker) = worker_firing(|_, _| {}); + drop(worker.clients.shared.named::("pool", || async { Ok(1) }).await.unwrap()); + let run = worker.background.token(); + let winding = tokio::spawn({ + let worker = worker.clone(); + async move { wind_down(&worker).await } + }); + tokio::task::yield_now().await; + assert_eq!(worker.clients.shared.len(), 1, "a run still going keeps it"); + drop(run); + tokio::time::timeout(std::time::Duration::from_secs(5), winding).await.expect("wound down").unwrap(); + assert!(worker.clients.shared.is_empty(), "let go once the runs ended"); + } + /// A line the broker keeps for good never stops the worker. #[tokio::test(start_paused = true)] async fn a_line_kept_for_good_never_stops_the_worker() { diff --git a/crates/weft-infra-supervisor/src/lifecycle.rs b/crates/weft-infra-supervisor/src/lifecycle.rs index 8b04c8562..732092a31 100644 --- a/crates/weft-infra-supervisor/src/lifecycle.rs +++ b/crates/weft-infra-supervisor/src/lifecycle.rs @@ -946,9 +946,10 @@ async fn execute_apply( let resolved = infra::resolve(&spec, ©, &image_tags).map_err(|e| anyhow!("{e}"))?; state.host.check(&resolved).map_err(|why| anyhow!("{why}"))?; // What the host runs differently from what was asked (a GPU kind it - // cannot choose): stamped on the row with the apply, where every - // status read shows it to the person who started the node. - let notes = state.host.notes(&resolved); + // cannot choose), and what the node itself says of the copy: stamped + // on the row with the apply, where every status read shows it to the + // person who started the node. + let notes: Vec = state.host.notes(&resolved).into_iter().chain(spec.notes.iter().cloned()).collect(); let applied_spec_hash = resolved.hash(); // Per-unit apply. Reconcile only the units that are DOWN (or new); diff --git a/crates/weft-platform-gcp/src/runner.rs b/crates/weft-platform-gcp/src/runner.rs index 680e33dc7..337647309 100644 --- a/crates/weft-platform-gcp/src/runner.rs +++ b/crates/weft-platform-gcp/src/runner.rs @@ -5,7 +5,11 @@ //! project's workers can prove which project they are and nothing else. //! Each program and worker settings the project runs is a revision of that //! service, under a tag of its own (`names::worker_tag`) whose address -//! weft's own calls use; the project's callers reach the service's own +//! weft's own calls use, until the dispatcher settles the project's +//! workers on other settings for the same program and it is deleted +//! (`Runner::settle`: which settings are current is the dispatcher's to +//! say, from what the project has stored, never this runner's to guess +//! from the revisions); the project's callers reach the service's own //! address, whose traffic goes to the project's front //! (`Runner::front`). Instances kept warm (`min_instances`) are the //! front's alone: the service holds them (Cloud Run splits a service's @@ -317,6 +321,80 @@ fn takes_every_call(found: &Value, revision: &str) -> bool { all == 100 } +/// The revisions of the service `found` that the tag `tag` supersedes, +/// once Cloud Run serves `tag`: every one of `revisions` (the service's, +/// as Cloud Run v2 lists them) running the same image (`image_hash`) +/// under another tag, that is with other settings. Each keeps instances +/// of its own, under its own `maxInstanceCount`, for as long as it is +/// there. Nothing goes before `tag` serves: the calls would have nowhere +/// to go. Whether `tag`'s settings are the current ones is its caller's to +/// know ([`Runner::settle`]): nothing on the service says so, since a +/// dispatcher copy holding settings from before a change can deploy a +/// revision with them after the current one. A revision of another image +/// is left to `Runner::forget_image`. +fn superseded(found: &Value, revisions: &[Value], tag: &str, image_hash: &str) -> Vec { + if tagged_revision(found, tag).is_none() || tag_address(found, tag).is_none() { + return Vec::new(); + } + revisions + .iter() + .filter(|r| r.pointer("/labels/weft-image").and_then(Value::as_str) == Some(image_hash)) + .filter(|r| revision_tag(r).is_some_and(|other| other != tag)) + .filter_map(revision_name) + .map(str::to_string) + .collect() +} + +/// The ones of `going` that can be deleted: all but the revision the +/// service `found`'s template names, which Cloud Run refuses to delete +/// (it is the service's latest). Taken out of the traffic like the rest, +/// it takes no calls, scales to zero, and goes with the next settling once +/// a newer write named another revision in the template. +fn deletable(found: &Value, going: Vec) -> Vec { + let template = found.pointer("/template/revision").and_then(Value::as_str); + going.into_iter().filter(|g| Some(g.as_str()) != template).collect() +} + +/// The tags (their `weft-tag` labels) of the ones of `revisions` named +/// in `going`: the tags whose remembered addresses retiring them ends. +fn tags_of(revisions: &[Value], going: &[String]) -> Vec { + revisions + .iter() + .filter(|r| revision_name(r).is_some_and(|name| going.iter().any(|g| g == name))) + .filter_map(revision_tag) + .map(str::to_string) + .collect() +} + +/// A Cloud Run v2 revision's short name (the last part of its `name`). +fn revision_name(revision: &Value) -> Option<&str> { + revision.get("name").and_then(Value::as_str).and_then(|n| n.rsplit('/').next()) +} + +/// The tag weft labeled a Cloud Run v2 revision with. +fn revision_tag(revision: &Value) -> Option<&str> { + revision.pointer("/labels/weft-tag").and_then(Value::as_str) +} + +/// The service `found`'s traffic with every entry naming one of `going` +/// taken out and the calls they took handed to the tag `to`, or `None` +/// when no entry names one. `going` runs the same program as `to` +/// ([`superseded`]), so its callers are answered the same. +fn traffic_handed_over(found: &Value, going: &[String], to: &str) -> Option { + let percent = |t: &Value| t.get("percent").and_then(Value::as_u64).unwrap_or(0); + let traffic: Vec = found.get("traffic").and_then(Value::as_array).cloned().unwrap_or_default(); + let (leaving, mut kept): (Vec, Vec) = + traffic.into_iter().partition(|t| t.get("revision").and_then(Value::as_str).is_some_and(|r| going.iter().any(|g| g == r))); + if leaving.is_empty() { + return None; + } + let handed: u64 = leaving.iter().map(percent).sum(); + if let Some(t) = kept.iter_mut().find(|t| t.get("tag").and_then(Value::as_str) == Some(to)) { + t["percent"] = json!(percent(t) + handed); + } + Some(Value::Array(kept.into_iter().filter(written_back).collect())) +} + /// Whether Cloud Run takes a traffic entry back as written: one that /// names its revision, or the "latest" one. An entry whose revision was /// deleted by hand names none. @@ -652,6 +730,86 @@ impl CloudRunRunner { sent } + /// Retire every revision `target`'s revision supersedes on its + /// project's service ([`superseded`]), once `target`'s tag serves: out + /// of its traffic first (Cloud Run refuses to delete a revision it + /// routes to), its calls handed to `target`'s, then deleted, which ends + /// its instances and whatever they hold (database connections, + /// memory); a worker told to stop hands its recorded runs back. The + /// template's revision is taken out of the traffic and kept + /// ([`deletable`]). Decided from the service as Cloud Run holds it, so + /// whatever one attempt leaves the next retires. + async fn retire_superseded(&self, target: &WorkerTarget) -> anyhow::Result<()> { + let service = names::worker_service(target.project); + let tag = Self::tag_of(target); + let image_hash = names::image_hash(&target.image); + let url = self.service_url(target.project); + let revisions = self.revisions(&url).await?; + let gate = self.gate(&service); + let untagged = async { + let _one = gate.lock().await; + loop { + let Some(found) = self.google.get_opt(&url).await? else { return Ok((Vec::new(), Vec::new())) }; + let going = superseded(&found, &revisions, &tag, &image_hash); + let Some(traffic) = traffic_handed_over(&found, &going, &tag) else { + return Ok((going.clone(), deletable(&found, going))); + }; + let template = found.get("template").cloned().ok_or_else(|| anyhow::anyhow!("{service} has no template"))?; + let body = self.service_body(target.project, template, traffic, warm_of(&found), found.get("etag").and_then(Value::as_str)); + match self.write_service(&url, &body).await { + // Another copy is changing the service: read it again + // in a moment. + Err(e) if is_status(&e, 409) || is_status(&e, 412) => tokio::time::sleep(SETTLE_POLL).await, + written => return written.map(|()| (going.clone(), deletable(&found, going))), + } + } + } + .await; + self.ungate(&service, &gate); + let (untagged, deleting) = untagged?; + // Out of the traffic, their tags' addresses no longer answer. + let retired = tags_of(&revisions, &untagged); + self.known.lock().retain(|(s, t), _| *s != service || !retired.contains(t)); + // Every one is tried, so the error names all that are left. + let mut left = Vec::new(); + for revision in deleting { + let deleted = match self.google.delete(&format!("{url}/revisions/{revision}")).await { + Ok(Some(op)) => self.google.wait("https://run.googleapis.com/v2", op).await.map(|_| ()), + Ok(None) => Ok(()), + Err(e) => Err(e), + }; + if let Err(e) = deleted { + left.push(format!("{revision}: {e:#}")); + } + } + anyhow::ensure!( + left.is_empty(), + "{} (to end them now, delete each with `gcloud run revisions delete --project {} --region {}`)", + left.join("; "), + self.gcp.project, + self.gcp.region, + ); + Ok(()) + } + + /// Every revision of the service at `url`, as Cloud Run v2 lists them. + async fn revisions(&self, url: &str) -> anyhow::Result> { + let mut out = Vec::new(); + let mut page: Option = None; + loop { + let mut query = vec![("pageSize", "100".to_string())]; + if let Some(token) = page.take() { + query.push(("pageToken", token)); + } + let listed = self.google.get_query(&format!("{url}/revisions"), &query).await?; + out.extend(listed.get("revisions").and_then(Value::as_array).into_iter().flatten().cloned()); + page = listed.get("nextPageToken").and_then(Value::as_str).map(str::to_string); + if page.is_none() { + return Ok(out); + } + } + } + /// The names of the services labeled `label=value`. async fn labeled(&self, label: &str, value: &str) -> anyhow::Result> { let mut out = Vec::new(); @@ -693,12 +851,10 @@ impl CloudRunRunner { /// never runs an image nothing references: it is left alone. async fn forget_revisions(&self, name: &str, image_hash: &str) -> anyhow::Result<()> { let url = format!("https://run.googleapis.com/v2/{name}"); - let listed = self.google.get(&format!("{url}/revisions")).await?; - let going: Vec = listed - .get("revisions") - .and_then(Value::as_array) - .into_iter() - .flatten() + let going: Vec = self + .revisions(&url) + .await? + .iter() .filter(|r| r.pointer("/labels/weft-image").and_then(Value::as_str) == Some(image_hash)) .filter_map(|r| r.get("name").and_then(Value::as_str).and_then(|n| n.rsplit('/').next()).map(str::to_string)) .collect(); @@ -781,6 +937,10 @@ impl Runner for CloudRunRunner { Ok(weft_core::projects::ProjectAddress::Serving { url }) } + async fn settle(&self, target: &WorkerTarget) -> anyhow::Result<()> { + self.retire_superseded(target).await + } + // The service keeps no warm instance any more and scales to zero on // its own; its traffic stays where it was, and its routes answer that // they take no calls. @@ -996,6 +1156,91 @@ mod tests { assert_eq!(retried.as_array().unwrap().iter().filter(|t| t["tag"] == "v2").count(), 1); } + /// A Cloud Run v2 revision named `name`, labeled as weft labels the + /// revision running the image of digest `image` under `tag`. + fn listed(name: &str, image: &str, tag: &str) -> Value { + json!({ + "name": format!("projects/acme/locations/us-central1/services/wk-x/revisions/{name}"), + "labels": { "weft-image": image, "weft-tag": tag }, + }) + } + + /// Once the settled settings serve, every revision of the same program + /// under other settings goes, whatever traffic it held and however new + /// it is; another program's is left to `forget_image`, and the settled + /// tag's own (an earlier attempt at it included) stays. + #[test] + fn settling_supersedes_every_revision_of_the_same_program_under_other_settings() { + let found = json!({ + "template": { "revision": "r-new" }, + "traffic": [ + { "revision": "r-old", "tag": "vaaaa0001", "percent": 100 }, + { "revision": "r-older", "tag": "vaaaa0002" }, + { "revision": "r-other", "tag": "vbbbb0001" }, + { "revision": "r-new", "tag": "vaaaa0003" }, + ], + "trafficStatuses": [{ "revision": "r-new", "tag": "vaaaa0003", "uri": "https://vaaaa0003---wk-x-1.a.run.app" }], + }); + let revisions = [ + listed("r-old", "aaaa", "vaaaa0001"), + listed("r-older", "aaaa", "vaaaa0002"), + // Left untagged by an attempt that did not retire it. + listed("r-untagged", "aaaa", "vaaaa0004"), + listed("r-other", "bbbb", "vbbbb0001"), + listed("r-new", "aaaa", "vaaaa0003"), + // An earlier attempt at the settled tag itself. + listed("r-new-try", "aaaa", "vaaaa0003"), + ]; + let going = superseded(&found, &revisions, "vaaaa0003", "aaaa"); + assert_eq!(going, ["r-old", "r-older", "r-untagged"]); + assert_eq!(tags_of(&revisions, &going), ["vaaaa0001", "vaaaa0002", "vaaaa0004"], "only the retired tags' addresses are forgotten"); + let traffic = traffic_handed_over(&found, &going, "vaaaa0003").unwrap(); + assert_eq!( + traffic, + json!([{ "revision": "r-other", "tag": "vbbbb0001" }, { "revision": "r-new", "tag": "vaaaa0003", "percent": 100 }]), + "the calls the other settings took go to the settled ones" + ); + assert!(takes_every_call(&json!({ "traffic": traffic }), "r-new")); + assert_eq!(deletable(&found, going), ["r-old", "r-older", "r-untagged"]); + assert_eq!(traffic_handed_over(&found, &["r-untagged".to_string()], "vaaaa0003"), None, "nothing to take out of the traffic"); + } + + /// Nothing goes before the settled settings serve: the calls would + /// have nowhere to go. + #[test] + fn nothing_is_superseded_before_the_settled_settings_serve() { + let revisions = [listed("r-old", "aaaa", "vaaaa0001"), listed("r-new", "aaaa", "vaaaa0003")]; + let tagged = json!({ + "traffic": [{ "revision": "r-old", "tag": "vaaaa0001", "percent": 100 }, { "revision": "r-new", "tag": "vaaaa0003" }], + }); + assert!(superseded(&tagged, &revisions, "vaaaa0003", "aaaa").is_empty(), "no address yet"); + let untagged = json!({ + "traffic": [{ "revision": "r-old", "tag": "vaaaa0001", "percent": 100 }], + "trafficStatuses": [{ "revision": "r-old", "tag": "vaaaa0001", "uri": "https://vaaaa0001---wk-x-1.a.run.app" }], + }); + assert!(superseded(&untagged, &revisions, "vaaaa0003", "aaaa").is_empty(), "not in the traffic yet"); + } + + /// A revision deployed after the settled one with other settings (a + /// dispatcher copy that still held the settings from before a change) + /// is the service's template, which Cloud Run refuses to delete: it is + /// taken out of the traffic, its calls handed to the settled settings, + /// and kept for a later settling. + #[test] + fn the_templates_revision_leaves_the_traffic_and_is_kept() { + let found = json!({ + "template": { "revision": "r-stale" }, + "traffic": [{ "revision": "r-stale", "tag": "vaaaa0001", "percent": 100 }, { "revision": "r-new", "tag": "vaaaa0003" }], + "trafficStatuses": [{ "revision": "r-new", "tag": "vaaaa0003", "uri": "https://vaaaa0003---wk-x-1.a.run.app" }], + }); + let revisions = [listed("r-new", "aaaa", "vaaaa0003"), listed("r-stale", "aaaa", "vaaaa0001")]; + let going = superseded(&found, &revisions, "vaaaa0003", "aaaa"); + assert_eq!(going, ["r-stale"]); + let traffic = traffic_handed_over(&found, &going, "vaaaa0003").unwrap(); + assert!(takes_every_call(&json!({ "traffic": traffic }), "r-new")); + assert!(deletable(&found, going).is_empty()); + } + #[test] fn a_failed_revision_redeploys_only_while_the_account_spreads() { let revision = |state: &str, reason: &str, message: &str| { diff --git a/crates/weft-platform-local/src/infra_host.rs b/crates/weft-platform-local/src/infra_host.rs index 9ad5ff700..30c5184a3 100644 --- a/crates/weft-platform-local/src/infra_host.rs +++ b/crates/weft-platform-local/src/infra_host.rs @@ -1222,6 +1222,7 @@ mod tests { }], keep_on_terminate: vec!["data".into()], health_rules: None, + notes: Vec::new(), } } diff --git a/crates/weft-platform-local/src/runner.rs b/crates/weft-platform-local/src/runner.rs index c070eb817..9283749df 100644 --- a/crates/weft-platform-local/src/runner.rs +++ b/crates/weft-platform-local/src/runner.rs @@ -739,6 +739,16 @@ impl Runner for LocalRunner { Ok(weft_core::projects::ProjectAddress::Serving { url: port_url(self.cfg.project_ip, port) }) } + // A worker with other settings already finishes what it drives and + // stops once idle, since calls go to one started with the current + // ones (`prepare`, `front`). Settling only says again which one is + // kept warm: a `prepare` from a caller holding settings from before a + // change may have made the stale one the warm one. + async fn settle(&self, target: &WorkerTarget) -> anyhow::Result<()> { + self.keep_only_current_warm(target); + Ok(()) + } + async fn let_front_go(&self, project: uuid::Uuid) -> anyhow::Result<()> { // No longer a front, it stops once idle like any worker: a project // that takes no calls keeps no instance warm. diff --git a/crates/weft-platform-traits/src/runner.rs b/crates/weft-platform-traits/src/runner.rs index a0060c21d..ec4c9d86e 100644 --- a/crates/weft-platform-traits/src/runner.rs +++ b/crates/weft-platform-traits/src/runner.rs @@ -324,6 +324,20 @@ pub trait Runner: Send + Sync { /// [`PortTaken`], and nothing is started. Idempotent. async fn front(&self, target: &WorkerTarget, port: Option) -> anyhow::Result; + /// `target` is what the project's workers run now: its image with the + /// settings the project has stored, read from them by the caller, not + /// from a copy that may predate a change. Let go of whatever the + /// platform keeps running of that image under other settings (on a + /// cloud, the revisions it deployed for them, their calls handed to + /// `target`'s). Only the caller can say which settings are current: + /// what the platform holds cannot, since a caller holding settings + /// from before a change may bring them up again after it. Does + /// nothing until `target` serves. An error names what is left; by + /// then `target` already serves, so the caller logs it rather than + /// failing the change that led here, and what is left takes no calls + /// and goes at the next settling. Idempotent. + async fn settle(&self, target: &WorkerTarget) -> anyhow::Result<()>; + /// Stop serving the project's own address: nothing of it takes calls /// any more (every trigger it has was wiped). Its workers go once idle. async fn let_front_go(&self, project: uuid::Uuid) -> anyhow::Result<()>; @@ -355,6 +369,7 @@ pub mod fake { Prepare(WorkerTarget), Endpoint { project: uuid::Uuid }, Front { target: WorkerTarget, port: Option }, + Settle(WorkerTarget), LetFrontGo { project: uuid::Uuid }, KillWorkers { project: uuid::Uuid }, Retire { project: uuid::Uuid }, @@ -415,6 +430,10 @@ pub mod fake { self.calls.lock().push(RunnerCall::Front { target: target.clone(), port }); Ok(weft_core::projects::ProjectAddress::Serving { url: self.base_url.clone() }) } + async fn settle(&self, target: &WorkerTarget) -> anyhow::Result<()> { + self.calls.lock().push(RunnerCall::Settle(target.clone())); + Ok(()) + } async fn let_front_go(&self, project: uuid::Uuid) -> anyhow::Result<()> { self.calls.lock().push(RunnerCall::LetFrontGo { project }); Ok(()) diff --git a/crates/weft-runtime/src/unit_agent.rs b/crates/weft-runtime/src/unit_agent.rs index 33ba315dc..a4fd39450 100644 --- a/crates/weft-runtime/src/unit_agent.rs +++ b/crates/weft-runtime/src/unit_agent.rs @@ -71,7 +71,16 @@ pub async fn run_probe(probe: &ProbeKind, timeout: Duration) -> ProbeAnswer { .expect("reqwest client"); match client.get(format!("http://127.0.0.1:{port}{path}")).send().await { Ok(r) if r.status().is_success() || r.status().is_redirection() => ProbeAnswer { ok: true, why: None }, - Ok(r) => failed(format!("GET {path} on {port} answered {}", r.status().as_u16())), + Ok(r) => { + let status = r.status().as_u16(); + // The body rides the same client timeout; one that does + // not arrive is a check that said nothing. + let body = body_start(r).await; + match what_it_said(&body) { + Some(said) => failed(format!("GET {path} on {port} answered {status}: {said}")), + None => failed(format!("GET {path} on {port} answered {status}")), + } + } Err(e) => failed(format!("GET {path} on {port}: {e}")), } } @@ -79,6 +88,42 @@ pub async fn run_probe(probe: &ProbeKind, timeout: Duration) -> ProbeAnswer { } } +/// The longest stretch of a failing check's body that its reason carries. +const SAID_MAX_CHARS: usize = 300; + +/// How much of a failing check's body is read: enough for any JSON +/// `error` a container means to tell, never a whole page a misbehaving one +/// streams back. +const BODY_MAX_BYTES: usize = 4096; + +/// The first [`BODY_MAX_BYTES`] of a response's body, read lossily as +/// text; what arrived before an error when the body breaks off. +async fn body_start(mut response: reqwest::Response) -> String { + let mut body = Vec::new(); + while body.len() < BODY_MAX_BYTES { + match response.chunk().await { + Ok(Some(chunk)) => body.extend_from_slice(&chunk[..chunk.len().min(BODY_MAX_BYTES - body.len())]), + Ok(None) | Err(_) => break, + } + } + String::from_utf8_lossy(&body).into_owned() +} + +/// What a failing HTTP check's body says, so a container that knows why +/// it is not ready can tell it: the string `error` field of a JSON body, +/// else the body's text, trimmed and cut to [`SAID_MAX_CHARS`]. `None` +/// when it says nothing. +fn what_it_said(body: &str) -> Option { + let error = serde_json::from_str::(body) + .ok() + .and_then(|value| value.get("error").and_then(serde_json::Value::as_str).map(str::to_owned)); + let said = error.as_deref().unwrap_or(body).trim(); + if said.is_empty() { + return None; + } + Some(said.chars().take(SAID_MAX_CHARS).collect()) +} + /// Give every file under `paths` to group `gid`, readable and writable by /// it (directories searchable), the way a unit's `fsGroup` asks. pub fn own(gid: u32, paths: &[PathBuf]) -> anyhow::Result<()> { @@ -424,13 +469,42 @@ mod tests { async fn an_http_check_reads_the_status() { let app = Router::new() .route("/ok", axum::routing::get(|| async { "fine" })) - .route("/down", axum::routing::get(|| async { axum::http::StatusCode::SERVICE_UNAVAILABLE })); + .route("/down", axum::routing::get(|| async { axum::http::StatusCode::SERVICE_UNAVAILABLE })) + .route( + "/broken", + axum::routing::get(|| async { (axum::http::StatusCode::SERVICE_UNAVAILABLE, r#"{"error": "no extension"}"#) }), + ) + .route("/long", axum::routing::get(|| async { (axum::http::StatusCode::SERVICE_UNAVAILABLE, "x".repeat(1 << 20)) })); let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); let port = listener.local_addr().unwrap().port(); tokio::spawn(async move { axum::serve(listener, app).await.unwrap() }); assert!(run_probe(&ProbeKind::Http { path: "/ok".into(), port }, Duration::from_secs(2)).await.ok); let down = run_probe(&ProbeKind::Http { path: "/down".into(), port }, Duration::from_secs(2)).await; assert_eq!(down.why.as_deref(), Some(format!("GET /down on {port} answered 503").as_str())); + let broken = run_probe(&ProbeKind::Http { path: "/broken".into(), port }, Duration::from_secs(2)).await; + assert_eq!(broken.why.as_deref(), Some(format!("GET /broken on {port} answered 503: no extension").as_str())); + let long = run_probe(&ProbeKind::Http { path: "/long".into(), port }, Duration::from_secs(2)).await; + assert_eq!(long.why, Some(format!("GET /long on {port} answered 503: {}", "x".repeat(SAID_MAX_CHARS)))); + } + + /// A failing check's body is told by its JSON `error`, else by its + /// trimmed text, cut on a character boundary, and not at all when + /// it is empty. + #[test] + fn a_failing_check_tells_what_its_body_said() { + assert_eq!( + what_it_said(r#"{"error": "the database could not create the extension 'foo'", "ready": false}"#).as_deref(), + Some("the database could not create the extension 'foo'") + ); + assert_eq!(what_it_said(" still warming up\n").as_deref(), Some("still warming up")); + assert_eq!(what_it_said(r#"{"ready": false}"#).as_deref(), Some(r#"{"ready": false}"#)); + assert_eq!(what_it_said(""), None); + assert_eq!(what_it_said(" \n "), None); + assert_eq!(what_it_said(r#"{"error": " "}"#), None); + let long = "é".repeat(400); + let cut = what_it_said(&long).unwrap(); + assert_eq!(cut.chars().count(), SAID_MAX_CHARS); + assert!(long.starts_with(&cut)); } /// A restart changes what a look compares once, when the container diff --git a/docs/src/nodes/ctx.md b/docs/src/nodes/ctx.md index 637963ade..99c7eeb0a 100644 --- a/docs/src/nodes/ctx.md +++ b/docs/src/nodes/ctx.md @@ -193,7 +193,36 @@ connection changes), so pass the same limit everywhere you call it. The limit co past it between them, and so can the old thing and the new one for a moment after a connection's values change, while runs still hold handles to the old one. The Postgres nodes that query through the -shared pool ask for 100, Postgres's own default connection limit. +shared pool ask for 20. + +`with_limit` only caps what one worker takes, so that its runs never leave +its sibling workers without a connection. It cannot keep a database +under its connection limit, because it cannot see how many workers there are. If your node talks to something that only +serves so much (a database with a fixed number of connections, an API with a +rate limit), assume any number of workers run your node at once: Cloud Run +starts copies up to the project's `--max-instances` (set with `weft workers +set`), and a change of program +or worker settings runs the old copies beside the new ones for a while. Two +things follow: + +- **Hold nothing you are not using.** Close an idle connection within + seconds, because one a copy keeps between requests is one no other copy can + have. On Cloud Run a copy has no CPU between requests (unless the + project turned on `weft workers set --cpu-always-allocated true`), so a timer of yours that closes it may not + fire for minutes: check how long a connection sat idle when a run takes it, + too. The Postgres nodes close a pooled connection after 5 idle seconds and + never hand out one idle for longer. +- **Let the other end do the counting.** It is the one place that sees every + copy, so that is where a client past the limit should wait its turn, or be + refused with a clear answer. Your node then waits, or fails with a message + that says why. A PostgresDatabase does this with a pooler in front of + Postgres, where a client that finds every connection busy waits for one, + and Postgres closes a session that sat idle for a minute. If your node + keeps a connection idle on purpose (one that `LISTEN`s), lift that limit + for its own session when it connects, the way PostgresWatchQuery sends + `SELECT set_config('idle_session_timeout', '0', false) FROM pg_settings + WHERE name = 'idle_session_timeout'` (the `FROM pg_settings` part makes it + do nothing on a Postgres that has no such setting, instead of failing). If a task you spawn needs the thing after your run returns (handing a connection back to its pool later), move the handle into the task: it keeps @@ -207,7 +236,9 @@ By default, weft lets go of a shared thing about five minutes after the last run asked for it or dropped its handle, once no handle is left. Your thing's `Drop` runs once weft has let go of it and the last handle and `Arc` are gone, which is when a pool closes its connections. When the worker -stops, everything goes with the process. +stops, it lets go of every shared thing as soon as its last run has ended, +used recently or not, so a pool's connections go back to the database while +the worker is still writing its last records. If you want a quiet stretch not to cost a rebuild, raise the five minutes with `weft workers set --shared-idle-seconds ` (or lower it to let go diff --git a/docs/src/nodes/infrastructure.md b/docs/src/nodes/infrastructure.md index 97469282a..a088cadd6 100644 --- a/docs/src/nodes/infrastructure.md +++ b/docs/src/nodes/infrastructure.md @@ -41,6 +41,7 @@ naming your node when the infra is provisioned, rather than a silent nothing. | `endpoints` | Named ports your node and the project's other nodes reach it on, and whether anything outside the project may | | `keep_on_terminate` | Disks to keep even through a terminate, for the next `weft infra start` to pick up | | `health_rules` | What weft does to the project's triggers when a unit of this node stops being ready, and when it undoes it. For the defaults and how to write your own, go and read [when a unit stops being ready](#when-a-unit-stops-being-ready) | +| `notes` | If the copy you declare runs short of something it normally has (an older version kept for its data, say), one plain sentence each saying what and how to fix it. `weft infra status` prints them as warnings for as long as the copy runs. Usually empty | A unit is a group of containers that run side by side, with optional init containers that run one after another before them. It also says what machine @@ -157,14 +158,25 @@ address while the tunnel is open can reach. If an endpoint hands out a credential, keep it to the project. The Postgres -node shows the split: the port Postgres answers on is `SameNetwork`, because -reaching it still takes a password, but the small server that hands out that +node shows the split: the port clients sign in on can be `SameNetwork` (its +`reachable` input), because reaching it still takes a password, but the small server that hands out that password stays reachable only by the project. It works during provisioning after the apply, and in every later phase once the infrastructure is running. If the endpoint is not declared, or the infrastructure is down, the error says which and points at `weft infra status`. +If your unit serves something with a fixed limit (a database that takes only +so many connections), the waiting belongs in your unit, because it is the one +place that sees every client, and the number of clients keeps changing +(workers scale, and a frontend connects too). A PostgresDatabase runs a pooler +in front of Postgres for this, in the same unit: the pooler never opens more +connections than Postgres takes, a client that finds them all busy waits for +one, and Postgres closes a session that sat idle for a minute, so a connection +somebody holds without using it comes back. For what a node that talks to +such a unit should do on its own side, go and read +[sharing something between runs](ctx.md#sharing-something-between-runs). + When a program marks your node `@per_instance`, each instance of the program gets its own container, and `ctx.endpoint` answers with the copy of the instance the run is for; your node's code does not change. Each copy has its diff --git a/docs/src/nodes/testing.md b/docs/src/nodes/testing.md index 1a42f63e0..074b9ea6c 100644 --- a/docs/src/nodes/testing.md +++ b/docs/src/nodes/testing.md @@ -168,7 +168,8 @@ Looking at what happened: | `stops(&self) -> Vec<(String, StopSelf)>` | Every `ctx.stop_tagged` asked for | | `program_calls(&self) -> Vec<(ProgramCall, StopSelf)>` | Every program call made | | `minted_tokens(&self) -> Vec` | Every instance token minted: `instance`, `expires_in_secs`, `id` | -| `published_values(&self, service: &str) -> Option>` | What the node published as that service's connection | +| `published_values(&self, service: &str) -> Option>` | What the node published as that service's connection. A publish is held to the same rule as on an install: a value the service does not declare, a required one missing, or one the service's identity names left blank fails the run. The service's recipe is found in the whole catalog, as a project build finds it, so a node can publish a service another package declares; if no package declares it, the test build stops with an error naming it | +| `fresh(&self) -> FakeRig` | A second, empty rig that knows the same services, for a test that needs two | ### The live rig diff --git a/docs/src/running/architecture.md b/docs/src/running/architecture.md index 8c88a0d5e..9b6a91001 100644 --- a/docs/src/running/architecture.md +++ b/docs/src/running/architecture.md @@ -304,6 +304,25 @@ copies Cloud Run may start (`--max-instances`), or a copy's CPU and memory (`--cpu`, `--memory`), `weft workers set` changes them for one project. On your machine none of these four does anything: there is no Cloud Run sending calls or starting copies, and weft puts no CPU or memory limit on a worker. +If a node of yours talks to something every copy draws on (a +database's connections), plan for as many copies as `--max-instances` allows, plus the +old ones during a change: for what that asks of the node, go and read +[sharing something between runs](../nodes/ctx.md#sharing-something-between-runs). + +New worker settings start a new Cloud Run revision. Once it serves, weft hands +it the calls the old settings took and deletes the revisions of the +same program under other settings, so their copies stop and give back what +they held, and each run still on them is handed back the way it is on any stop +([what happens when something dies](#what-happens-when-something-dies)). Which +settings are the new ones is read from what the project has stored, never +from which revision is newest. A revision of another program stays until weft +forgets that program's image. If deleting one fails, it takes no calls and its +copies scale to zero once nothing calls them; the next activation or settings +change deletes it, and weft logs an error naming the `gcloud run revisions +delete` command that deletes it now. The one revision Cloud Run never lets +anyone delete is the service's latest: when a dispatcher copy that still held +the old settings brought them back up after the change, that revision takes no +calls and goes once a later settings change or build replaces it. On a cloud install, if a run keeps working after its caller has gone (a Route with `outlivesCaller` that replies and carries on), the nodes it was running diff --git a/tangle/claude-code/.claude/agents/node-smith.md b/tangle/claude-code/.claude/agents/node-smith.md index a409ed2f7..946a9c762 100644 --- a/tangle/claude-code/.claude/agents/node-smith.md +++ b/tangle/claude-code/.claude/agents/node-smith.md @@ -49,7 +49,7 @@ After each write under `nodes/`, the edit hook prints what `weft validate` finds 4. Write the files: - `metadata.json`: [the contract] verbatim, plus presentation. When the node is a trigger, this includes `firesWith`: EVERY field the wake payload can carry, `?` on the ones that only sometimes arrive. The engine checks a real firing (and a hand-typed `weft run --fire`) before your `run` starts, and the check is exact, so a firing carrying a field you never named is refused just like one missing a field you required. Name only what you fan onto ports and the trigger dies the first time the provider sends anything else; when the payload comes from a connection's events, copy the names from `events..fields` in the service's recipe. Skip it only when the node cannot name a payload shape at all (it reads its own connection instead of the wake payload, or its fields are per-instance config typed in by the author), and say why in the report. - Spell every name in `metadata.json` the way Spelling in the `weft-node-authoring` manual says, and clear every catalog warning `weft describe-nodes --list` prints for your node before you report. If the ports a program adds to your node are named after data from outside weft (a query's parameters, a template's ids), set `features.addedPortsAreDataKeys`, and they keep that data's spelling. - - `mod.rs`: the body, one job, no orchestration, no plumbing, no fallbacks; every failure is a loud `node_bail!` error. Every value you emit on a port weighs at most 100 KB as JSON, checked on your emission, so a node whose output could be bigger bounds it itself, decided now: it pages (an offset in, `nextOffset` and `hasMore` out), caps (a size setting with a default and a `truncated` output; `weft::storage::cap_text` does the cut), or puts the bytes in storage and emits the file value; the manual's "The wire limit" has the rule. A client, a pool or a process slow to build (a database connection pool, a set of interpreter processes) is built with `ctx.shared`, which keeps it on the worker for its runs to share, rather than opened on every run. + - `mod.rs`: the body, one job, no orchestration, no plumbing, no fallbacks; every failure is a loud `node_bail!` error. Every value you emit on a port weighs at most 100 KB as JSON, checked on your emission, so a node whose output could be bigger bounds it itself, decided now: it pages (an offset in, `nextOffset` and `hasMore` out), caps (a size setting with a default and a `truncated` output; `weft::storage::cap_text` does the cut), or puts the bytes in storage and emits the file value; the manual's "The wire limit" has the rule. A client, a pool or a process slow to build (a database connection pool, a set of interpreter processes) is built with `ctx.shared`, which keeps it on the worker for its runs to share, rather than opened on every run. Any number of workers can run your node at once (they scale up to the project's `--max-instances`, and a change of program or worker settings runs the old copies beside the new for a while), so a node that talks to something with a fixed capacity (a database's connections, an API's rate limit) holds none of it while idle: close an idle connection within seconds, and check how long one sat idle before handing it to a run, because a Cloud Run copy has no CPU between requests to run your timer. `.with_limit(n)` on `ctx.shared` only caps how many of one worker's runs hold it at once; what protects the database (or the API) is that end itself making the extra clients wait or refusing them cleanly (a PostgresDatabase has a pooler for that), and your node waiting, or failing loudly, on that answer. If your node holds a connection idle on purpose (one that `LISTEN`s), the database may close it after a while (a PostgresDatabase does after a minute), so lift that limit for its own session when it connects, the way PostgresWatchQuery sends `SELECT set_config('idle_session_timeout', '0', false) FROM pg_settings WHERE name = 'idle_session_timeout'` (a no-op on a Postgres without the setting). - `deps.toml`: only if you need crates or OS packages beyond the always-available ones. - Every node has an `_error` output that weft adds and fills with the node's failure when the program wires it, so you write no error handling for it and never declare `_error` yourself (a plain `error` output of your own is fine). A node that can fail on a call outside covers both paths in its `fake` tests, calling `rig.wire_output("_error")` for the caught one and reading `outcome.output("_error")?`. A node that streams opens its bus on its first piece (`ctx.pending_bus`, then `bus.open()`), never before the call that can fail, so that failure still reaches `_error`. A node whose body does nothing outside its run beyond answering its caller and the run's own files (no network, no connection, no `ctx.run`, no wait, no tag, no storage beyond this run) sets `"pure": true` in its `features`: a durable run starts it without waiting for the database, and runs it again if its worker died before it sent anything on (when the node reads no stream). A node that reaches outside never sets it: weft fails the first outside call made through the ctx of one that does, naming the flag, and cannot see a socket or file the body opens on its own. A value the node keeps that grows with use goes in a file it edits in place (`ctx.storage(scope).edit`), never on a port. A node that answers a live caller declares `answersCaller` in its `features`. A `validate` rule picks other nodes by a feature they declare (`with: {feature: value}`), never by a type name. The manual has all four. - An infra node gets no `live` test for now: its container is proven inside a real program. Make a scratch project in `.scratch/node-smith/` (inside the project, so writing there never asks; add `.scratch/` to the project's `.gitignore` if it is not listed) with the node under its `nodes/`, then `weft infra start`, `weft infra status`, `weft infra logs`, a run through it (carved with `weft run --from` / `--emit` / `--target`), and, if the node has a baked output, `weft bake` then `weft run --fire` through a trigger that reads it, which proves the saved value, and `weft infra terminate --yes` at the end. Your report says what you ran. Size every unit's `machine` (CPU, memory, GPU) as the manual's "The machine" section says: the user pays for the machine it picks. Mark `"baked": true` on every output that holds as long as the infra does (an address, a connection, a handle), so a run that reads only those uses the saved values instead of running the node. When the container changes such a value itself (a reset button), it `POST`s the new value to `$WEFT_VALUES_URL` and answers with what that said. For the full rule, go and read Infra node in the manual. diff --git a/tangle/claude-code/.claude/skills/weft-node-authoring/SKILL.md b/tangle/claude-code/.claude/skills/weft-node-authoring/SKILL.md index 60452db6a..888efc472 100644 --- a/tangle/claude-code/.claude/skills/weft-node-authoring/SKILL.md +++ b/tangle/claude-code/.claude/skills/weft-node-authoring/SKILL.md @@ -392,7 +392,7 @@ You emit only through `ctx.pulse_downstream(NodeOutput::new().set(port, value))` ports you did not emit are closed, which is the skip signal downstream. For user-added output ports use `ctx.fan_declared(...)`. -**Share what is slow to build**: a client pool, a process. `ctx.shared(&access, |opened| async move { .. })` builds it the first time a run asks and hands every later run on that worker the same one; it is built again when the connection's values change, or after nobody used it for the project's idle window (300 seconds by default, `weft workers set --shared-idle-seconds`). The closure returns a `WeftResult` of the thing and you get a handle you use as the thing. `ctx.shared("name", |()| async { .. })` is the same when nothing is built from a connection, and `.with_limit(n)` before the `.await` lets at most `n` runs of a worker hold it at once (a pool kept under a database's connection limit). A database client opened on every run pays the connection each time; for a worked pool, go and read `nodes/base_catalog/postgres/postgres.rs`. +**Share what is slow to build**: a client pool, a process. `ctx.shared(&access, |opened| async move { .. })` builds it the first time a run asks and hands every later run on that worker the same one; it is built again when the connection's values change, or after nobody used it for the project's idle window (300 seconds by default, `weft workers set --shared-idle-seconds`). The closure returns a `WeftResult` of the thing and you get a handle you use as the thing. `ctx.shared("name", |()| async { .. })` is the same when nothing is built from a connection, and `.with_limit(n)` before the `.await` lets at most `n` runs of a worker hold it at once. That never keeps a database under its limit, because any number of workers run a node at once. So have your pool close a connection after a few idle seconds (the Postgres nodes use 5) and never hand a run one idle for longer, because a Cloud Run copy has no CPU between requests to run your close timer, and leave the waiting to whatever sits in front of the database, such as a PostgresDatabase's pooler. If your node keeps a connection idle on purpose (one that `LISTEN`s), the database may close it (a PostgresDatabase does after a minute), so lift `idle_session_timeout` for that session when it connects, the way PostgresWatchQuery does. A database client opened on every run pays the connection each time; for a worked pool, go and read `nodes/base_catalog/postgres/postgres.rs`. If the worker dies while your body runs, a fast run (the default) ends cancelled, and a durable run fails the node: its start was on record before the body began, so the next worker knows it was running and fails it rather than start it again. With `_error` wired, that failure goes to `_error` like any other. A node marked `pure` is the exception: a firing of it that had sent nothing on and read no stream runs again, since it did nothing anyone could see. The one other body weft starts again on purpose is one parked on `ctx.await_signal`, which replays from the top when its answer comes, so the work before the wait goes through `ctx.run(...)`, which gives back the recorded result. An emit on a port already sent before the wait does nothing on the replay, so a node can send `stats` at once, wait, then send `image`; what it cannot do is wait after yielding on or closing a stream output, which is refused. In a run that cannot pause (a route's caller on the line, `recorded: false`, a bus open), `ctx.await_signal` holds in your call instead and returns the answer without a replay; if the run stays quiet for its trigger's `holdSecs`, the call fails with `WeftError::WaitGaveUp`, which your node may match and handle (test it with `rig.signal_given_up()`). diff --git a/tangle/cline/.cline/skills/node-smith/SKILL.md b/tangle/cline/.cline/skills/node-smith/SKILL.md index cbab68cde..4b38ad8aa 100644 --- a/tangle/cline/.cline/skills/node-smith/SKILL.md +++ b/tangle/cline/.cline/skills/node-smith/SKILL.md @@ -63,7 +63,7 @@ After each write under `nodes/`, the edit hook prints what `weft validate` finds 4. Write the files: - `metadata.json`: [the contract] verbatim, plus presentation. Building a trigger? Add `firesWith` there too: EVERY field the wake payload can carry, with `?` on the ones that only sometimes arrive, so the engine can check a real firing (and a hand-typed `weft run --fire`) against that shape before `run` starts. The check is exact: a firing carrying a field you never named is refused just like one missing a field you required. Name only what you fan onto ports and the trigger dies the first time the provider sends anything else; when the payload comes from a connection's events, copy the names from `events..fields` in the service's recipe. Only skip it when the trigger truly has nothing to name (it reads its own connection, or its fields are the author's own per-instance config), and say why in the report. - Spell every name in `metadata.json` the way Spelling in the `weft-node-authoring` manual says, and clear every catalog warning `weft describe-nodes --list` prints for your node before you report. If the ports a program adds to your node are named after data from outside weft (a query's parameters, a template's ids), set `features.addedPortsAreDataKeys`, and they keep that data's spelling. - - `mod.rs`: the body, one job, no orchestration, no plumbing, no fallbacks; every failure is a loud `node_bail!` error. Every value you emit on a port is at most 100 KB, checked on your emission, so a node whose output could be bigger bounds it itself, decided now (pages it with an offset in and `hasMore` out, caps it with a size setting and a `truncated` output, or puts the bytes in storage and emits the file value); the manual's "The wire limit" has the rule. A client, a pool or a process slow to build (a database connection pool, a set of interpreter processes) is built with `ctx.shared`, which keeps it on the worker for its runs to share, rather than opened on every run. + - `mod.rs`: the body, one job, no orchestration, no plumbing, no fallbacks; every failure is a loud `node_bail!` error. Every value you emit on a port is at most 100 KB, checked on your emission, so a node whose output could be bigger bounds it itself, decided now (pages it with an offset in and `hasMore` out, caps it with a size setting and a `truncated` output, or puts the bytes in storage and emits the file value); the manual's "The wire limit" has the rule. A client, a pool or a process slow to build (a database connection pool, a set of interpreter processes) is built with `ctx.shared`, which keeps it on the worker for its runs to share, rather than opened on every run. Any number of workers can run your node at once (they scale up to the project's `--max-instances`, and a change of program or worker settings runs the old copies beside the new for a while), so a node that talks to something with a fixed capacity (a database's connections, an API's rate limit) holds none of it while idle: close an idle connection within seconds, and check how long one sat idle before handing it to a run, because a Cloud Run copy has no CPU between requests to run your timer. `.with_limit(n)` on `ctx.shared` only caps how many of one worker's runs hold it at once; what protects the database (or the API) is that end itself making the extra clients wait or refusing them cleanly (a PostgresDatabase has a pooler for that), and your node waiting, or failing loudly, on that answer. If your node holds a connection idle on purpose (one that `LISTEN`s), the database may close it after a while (a PostgresDatabase does after a minute), so lift that limit for its own session when it connects, the way PostgresWatchQuery sends `SELECT set_config('idle_session_timeout', '0', false) FROM pg_settings WHERE name = 'idle_session_timeout'` (a no-op on a Postgres without the setting). - `deps.toml`: only if you need crates or OS packages beyond the always-available ones. - Every node gets a `_error` output from weft, which catches the node's failures there when the program wires it, so you write no error handling and never declare `_error` yourself. Its `fake` tests cover both paths, calling `rig.wire_output("_error")` for the caught one. A node with a `Bus` output opens it on its first piece (`ctx.pending_bus`, then `open()`), so a failure before that still lands on `_error`. A node whose body does nothing outside its run beyond answering its caller and the run's own files (no network, no connection, no `ctx.run`, no wait, no tag, no storage beyond this run) sets `"pure": true` in its `features`: a durable run starts it without waiting for the database, and runs it again if its worker died before it sent anything on (when the node reads no stream). A node that reaches outside never sets it: weft fails the first outside call made through the ctx of one that does, naming the flag, and cannot see a socket or file the body opens on its own. A value the node keeps that grows with use goes in a file it edits in place (`ctx.storage(scope).edit`), never on a port. A node that answers a live caller declares `answersCaller` in its `features`. A `validate` rule picks other nodes by a feature they declare (`with: {feature: value}`), never by a type name. The manual has all four. - An infra node gets no `live` test for now: its container is proven inside a real program. Make a scratch project in `.scratch/node-smith/` (inside the project, so the "Edit project files" auto-approve setting covers it; add `.scratch/` to the project's `.gitignore` if it is not listed) with the node under `nodes/`, then `weft infra start`, `weft infra status`, `weft infra logs`, a run through it (carved with `weft run --from` / `--emit` / `--target`), and, if the node has a baked output, `weft bake` then `weft run --fire` through a trigger that reads it, which proves the saved value, and `weft infra terminate --yes` at the end. Your report says what you ran. Size every unit's `machine` (CPU, memory, GPU) as the manual's "The machine" section says: the user pays for the machine it picks. Mark `"baked": true` on every output that holds as long as the infra does (an address, a connection, a handle), so a run that reads only those uses the saved values instead of running the node. When the container changes such a value itself (a reset button), it `POST`s the new value to `$WEFT_VALUES_URL` and answers with what that said. For the full rule, go and read Infra node in the manual. diff --git a/tangle/cline/.cline/skills/weft-node-authoring/SKILL.md b/tangle/cline/.cline/skills/weft-node-authoring/SKILL.md index 1621cfeda..ba85a8532 100644 --- a/tangle/cline/.cline/skills/weft-node-authoring/SKILL.md +++ b/tangle/cline/.cline/skills/weft-node-authoring/SKILL.md @@ -387,7 +387,7 @@ You emit only through `ctx.pulse_downstream(NodeOutput::new().set(port, value))` ports you did not emit are closed, which is the skip signal downstream. For user-added output ports use `ctx.fan_declared(...)`. -**Share what is slow to build**: a client pool, a process. `ctx.shared(&access, |opened| async move { .. })` builds it the first time a run asks and hands every later run on that worker the same one; it is built again when the connection's values change, or after nobody used it for the project's idle window (300 seconds by default, `weft workers set --shared-idle-seconds`). The closure returns a `WeftResult` of the thing and you get a handle you use as the thing. `ctx.shared("name", |()| async { .. })` is the same when nothing is built from a connection, and `.with_limit(n)` before the `.await` lets at most `n` runs of a worker hold it at once (a pool kept under a database's connection limit). A database client opened on every run pays the connection each time; for a worked pool, go and read `nodes/base_catalog/postgres/postgres.rs`. +**Share what is slow to build**: a client pool, a process. `ctx.shared(&access, |opened| async move { .. })` builds it the first time a run asks and hands every later run on that worker the same one; it is built again when the connection's values change, or after nobody used it for the project's idle window (300 seconds by default, `weft workers set --shared-idle-seconds`). The closure returns a `WeftResult` of the thing and you get a handle you use as the thing. `ctx.shared("name", |()| async { .. })` is the same when nothing is built from a connection, and `.with_limit(n)` before the `.await` lets at most `n` runs of a worker hold it at once. That never keeps a database under its limit, because any number of workers run a node at once. So have your pool close a connection after a few idle seconds (the Postgres nodes use 5) and never hand a run one idle for longer, because a Cloud Run copy has no CPU between requests to run your close timer, and leave the waiting to whatever sits in front of the database, such as a PostgresDatabase's pooler. If your node keeps a connection idle on purpose (one that `LISTEN`s), the database may close it (a PostgresDatabase does after a minute), so lift `idle_session_timeout` for that session when it connects, the way PostgresWatchQuery does. A database client opened on every run pays the connection each time; for a worked pool, go and read `nodes/base_catalog/postgres/postgres.rs`. If the worker dies while your body runs, a fast run (the default) ends cancelled, and a durable run fails the node: its start was on record before the body began, so the next worker knows it was running and fails it rather than start it again. With `_error` wired, that failure goes to `_error` like any other. A node marked `pure` is the exception: a firing of it that had sent nothing on and read no stream runs again, since it did nothing anyone could see. The one other body weft starts again on purpose is one parked on `ctx.await_signal`, which replays from the top when its answer comes, so the work before the wait goes through `ctx.run(...)`, which gives back the recorded result. An emit on a port already sent before the wait does nothing on the replay, so a node can send `stats` at once, wait, then send `image`; what it cannot do is wait after yielding on or closing a stream output, which is refused. In a run that cannot pause (a route's caller on the line, `recorded: false`, a bus open), `ctx.await_signal` holds in your call instead and returns the answer without a replay; if the run stays quiet for its trigger's `holdSecs`, the call fails with `WeftError::WaitGaveUp`, which your node may match and handle (test it with `rig.signal_given_up()`). diff --git a/tangle/codex/.agents/skills/weft-node-authoring/SKILL.md b/tangle/codex/.agents/skills/weft-node-authoring/SKILL.md index 5424a7c94..cef125393 100644 --- a/tangle/codex/.agents/skills/weft-node-authoring/SKILL.md +++ b/tangle/codex/.agents/skills/weft-node-authoring/SKILL.md @@ -389,7 +389,7 @@ You emit only through `ctx.pulse_downstream(NodeOutput::new().set(port, value))` ports you did not emit are closed, which is the skip signal downstream. For user-added output ports use `ctx.fan_declared(...)`. -**Share what is slow to build**: a client pool, a process. `ctx.shared(&access, |opened| async move { .. })` builds it the first time a run asks and hands every later run on that worker the same one; it is built again when the connection's values change, or after nobody used it for the project's idle window (300 seconds by default, `weft workers set --shared-idle-seconds`). The closure returns a `WeftResult` of the thing and you get a handle you use as the thing. `ctx.shared("name", |()| async { .. })` is the same when nothing is built from a connection, and `.with_limit(n)` before the `.await` lets at most `n` runs of a worker hold it at once (a pool kept under a database's connection limit). A database client opened on every run pays the connection each time; for a worked pool, go and read `nodes/base_catalog/postgres/postgres.rs`. +**Share what is slow to build**: a client pool, a process. `ctx.shared(&access, |opened| async move { .. })` builds it the first time a run asks and hands every later run on that worker the same one; it is built again when the connection's values change, or after nobody used it for the project's idle window (300 seconds by default, `weft workers set --shared-idle-seconds`). The closure returns a `WeftResult` of the thing and you get a handle you use as the thing. `ctx.shared("name", |()| async { .. })` is the same when nothing is built from a connection, and `.with_limit(n)` before the `.await` lets at most `n` runs of a worker hold it at once. That never keeps a database under its limit, because any number of workers run a node at once. So have your pool close a connection after a few idle seconds (the Postgres nodes use 5) and never hand a run one idle for longer, because a Cloud Run copy has no CPU between requests to run your close timer, and leave the waiting to whatever sits in front of the database, such as a PostgresDatabase's pooler. If your node keeps a connection idle on purpose (one that `LISTEN`s), the database may close it (a PostgresDatabase does after a minute), so lift `idle_session_timeout` for that session when it connects, the way PostgresWatchQuery does. A database client opened on every run pays the connection each time; for a worked pool, go and read `nodes/base_catalog/postgres/postgres.rs`. If the worker dies while your body runs, a fast run (the default) ends cancelled, and a durable run fails the node: its start was on record before the body began, so the next worker knows it was running and fails it rather than start it again. With `_error` wired, that failure goes to `_error` like any other. A node marked `pure` is the exception: a firing of it that had sent nothing on and read no stream runs again, since it did nothing anyone could see. The one other body weft starts again on purpose is one parked on `ctx.await_signal`, which replays from the top when its answer comes, so the work before the wait goes through `ctx.run(...)`, which gives back the recorded result. An emit on a port already sent before the wait does nothing on the replay, so a node can send `stats` at once, wait, then send `image`; what it cannot do is wait after yielding on or closing a stream output, which is refused. In a run that cannot pause (a route's caller on the line, `recorded: false`, a bus open), `ctx.await_signal` holds in your call instead and returns the answer without a replay; if the run stays quiet for its trigger's `holdSecs`, the call fails with `WeftError::WaitGaveUp`, which your node may match and handle (test it with `rig.signal_given_up()`). diff --git a/tangle/codex/.codex/agents/node-smith.toml b/tangle/codex/.codex/agents/node-smith.toml index 7b00d219e..9f606c9ab 100644 --- a/tangle/codex/.codex/agents/node-smith.toml +++ b/tangle/codex/.codex/agents/node-smith.toml @@ -46,7 +46,7 @@ After each write under `nodes/`, the edit hook prints what `weft validate` finds 4. Write the files: - `metadata.json`: [the contract] verbatim, plus presentation. When the node is a trigger, this includes `firesWith`: EVERY field the wake payload can carry, `?` on the ones that only sometimes arrive. The engine checks a real firing (and a hand-typed `weft run --fire`) before your `run` starts, and the check is exact, so a firing carrying a field you never named is refused just like one missing a field you required. Name only what you fan onto ports and the trigger dies the first time the provider sends anything else; when the payload comes from a connection's events, copy the names from `events..fields` in the service's recipe. Skip it only when the node cannot name a payload shape at all (it reads its own connection instead of the wake payload, or its fields are per-instance config typed in by the author), and say why in the report. - Spell every name in `metadata.json` the way Spelling in the `weft-node-authoring` manual says, and clear every catalog warning `weft describe-nodes --list` prints for your node before you report. If the ports a program adds to your node are named after data from outside weft (a query's parameters, a template's ids), set `features.addedPortsAreDataKeys`, and they keep that data's spelling. - - `mod.rs`: the body, one job, no orchestration, no plumbing, no fallbacks; every failure is a loud `node_bail!` error. Every value you emit on a port is at most 100 KB, checked on your emission, so a node whose output could be bigger bounds it itself (pages it, or caps it with a size setting and an output saying it cut, `weft::storage::cap_text` does the cut) or puts the bytes in storage and emits the file value; the manual's "The wire limit" has the rule. A client, a pool or a process slow to build (a database connection pool, a set of interpreter processes) is built with `ctx.shared`, which keeps it on the worker for its runs to share, rather than opened on every run. + - `mod.rs`: the body, one job, no orchestration, no plumbing, no fallbacks; every failure is a loud `node_bail!` error. Every value you emit on a port is at most 100 KB, checked on your emission, so a node whose output could be bigger bounds it itself (pages it, or caps it with a size setting and an output saying it cut, `weft::storage::cap_text` does the cut) or puts the bytes in storage and emits the file value; the manual's "The wire limit" has the rule. A client, a pool or a process slow to build (a database connection pool, a set of interpreter processes) is built with `ctx.shared`, which keeps it on the worker for its runs to share, rather than opened on every run. Any number of workers can run your node at once (they scale up to the project's `--max-instances`, and a change of program or worker settings runs the old copies beside the new for a while), so a node that talks to something with a fixed capacity (a database's connections, an API's rate limit) holds none of it while idle: close an idle connection within seconds, and check how long one sat idle before handing it to a run, because a Cloud Run copy has no CPU between requests to run your timer. `.with_limit(n)` on `ctx.shared` only caps how many of one worker's runs hold it at once; what protects the database (or the API) is that end itself making the extra clients wait or refusing them cleanly (a PostgresDatabase has a pooler for that), and your node waiting, or failing loudly, on that answer. If your node holds a connection idle on purpose (one that `LISTEN`s), the database may close it after a while (a PostgresDatabase does after a minute), so lift that limit for its own session when it connects, the way PostgresWatchQuery sends `SELECT set_config('idle_session_timeout', '0', false) FROM pg_settings WHERE name = 'idle_session_timeout'` (a no-op on a Postgres without the setting). - `deps.toml`: only if you need crates or OS packages beyond the always-available ones. - Every node has an `_error` output the language gives it, and weft catches a failure there when the program wires it, so the node writes no error handling and never declares `_error` itself, and sets no feature for it. Its `fake` tests cover both paths, calling `rig.wire_output("_error")` for the caught one. A node that sends on a bus opens it on its first piece (`ctx.pending_bus`, then `bus.open().await?`), never before the call that can fail, so that failure still reaches `_error`; the bus ends with the body, failed if the body failed. A node whose body does nothing outside its run beyond answering its caller and the run's own files (no network, no connection, no `ctx.run`, no wait, no tag, no storage beyond this run) sets `"pure": true` in its `features`: a durable run starts it without waiting for the database, and runs it again if its worker died before it sent anything on (when the node reads no stream). A node that reaches outside never sets it: weft fails the first outside call made through the ctx of one that does, naming the flag, and cannot see a socket or file the body opens on its own. A value the node keeps that grows with use goes in a file it edits in place (`ctx.storage(scope).edit`), never on a port. A node that answers a live caller declares `answersCaller` in its `features`. A `validate` rule picks other nodes by a feature they declare (`with: {feature: value}`), never by a type name. The manual has all four. - An infra node gets no `live` test for now: its container is proven inside a real program. Make a scratch project in `.scratch/node-smith/` (inside the project, so writing there never asks; add `.scratch/` to the project's `.gitignore` if it is not listed) with the node under its `nodes/` (every command there runs with your tool's working folder, `exec_command`'s `workdir`, set to that project, never after a `cd`, which is on no list), then `weft infra start`, `weft infra status`, `weft infra logs`, a run through it (carved with `weft run --from` / `--emit` / `--target`), and, if the node has a baked output, `weft bake` then `weft run --fire` through a trigger that reads it, which proves the saved value, and `weft infra terminate --yes` at the end. Your report says what you ran. Size every unit's `machine` (CPU, memory, GPU) as the manual's "The machine" section says: the user pays for the machine it picks. Mark `"baked": true` on every output that holds as long as the infra does (an address, a connection, a handle), so a run that reads only those uses the saved values instead of running the node. When the container changes such a value itself (a reset button), it `POST`s the new value to `$WEFT_VALUES_URL` and answers with what that said. For the full rule, go and read Infra node in the manual. diff --git a/tangle/cursor/.cursor/agents/node-smith.md b/tangle/cursor/.cursor/agents/node-smith.md index ea91d63ef..bc7bd3eca 100644 --- a/tangle/cursor/.cursor/agents/node-smith.md +++ b/tangle/cursor/.cursor/agents/node-smith.md @@ -49,7 +49,7 @@ After each write under `nodes/`, the edit hook prints what `weft validate` finds 4. Write the files: - `metadata.json`: [the contract] verbatim, plus presentation. If the node is a trigger, this includes `firesWith`: EVERY field the wake payload can carry, `?` on the ones that only sometimes arrive. The engine holds a real firing, and a hand-typed `weft run --fire`, to that shape before `run` starts, and it holds it exactly: a firing carrying a field you never named is refused just like one missing a field you required. Name only what you fan onto ports and the trigger dies the first time the provider sends anything else; when the payload comes from a connection's events, copy the names from `events..fields` in the service's recipe. Skip it only when there is no fixed shape to name (the node reads its own connection, or its fields are the author's own per-instance config), and say so in the report. - Spell every name in `metadata.json` the way Spelling in the `weft-node-authoring` manual says, and clear every catalog warning `weft describe-nodes --list` prints for your node before you report. If the ports a program adds to your node are named after data from outside weft (a query's parameters, a template's ids), set `features.addedPortsAreDataKeys`, and they keep that data's spelling. - - `mod.rs`: the body, one job, no orchestration, no plumbing, no fallbacks; every failure is a loud `node_bail!` error. Every value you emit on a port weighs at most 100 KB as JSON, checked on your emission, so a node whose output could be bigger bounds it itself (a page at a time with `hasMore`, a cap setting with a `truncated` output, a `LIMIT`) or puts the bytes in storage and emits the file value; the manual's "The wire limit" has the rule. A client, a pool or a process slow to build (a database connection pool, a set of interpreter processes) is built with `ctx.shared`, which keeps it on the worker for its runs to share, rather than opened on every run. + - `mod.rs`: the body, one job, no orchestration, no plumbing, no fallbacks; every failure is a loud `node_bail!` error. Every value you emit on a port weighs at most 100 KB as JSON, checked on your emission, so a node whose output could be bigger bounds it itself (a page at a time with `hasMore`, a cap setting with a `truncated` output, a `LIMIT`) or puts the bytes in storage and emits the file value; the manual's "The wire limit" has the rule. A client, a pool or a process slow to build (a database connection pool, a set of interpreter processes) is built with `ctx.shared`, which keeps it on the worker for its runs to share, rather than opened on every run. Any number of workers can run your node at once (they scale up to the project's `--max-instances`, and a change of program or worker settings runs the old copies beside the new for a while), so a node that talks to something with a fixed capacity (a database's connections, an API's rate limit) holds none of it while idle: close an idle connection within seconds, and check how long one sat idle before handing it to a run, because a Cloud Run copy has no CPU between requests to run your timer. `.with_limit(n)` on `ctx.shared` only caps how many of one worker's runs hold it at once; what protects the database (or the API) is that end itself making the extra clients wait or refusing them cleanly (a PostgresDatabase has a pooler for that), and your node waiting, or failing loudly, on that answer. If your node holds a connection idle on purpose (one that `LISTEN`s), the database may close it after a while (a PostgresDatabase does after a minute), so lift that limit for its own session when it connects, the way PostgresWatchQuery sends `SELECT set_config('idle_session_timeout', '0', false) FROM pg_settings WHERE name = 'idle_session_timeout'` (a no-op on a Postgres without the setting). - `deps.toml`: only if you need crates or OS packages beyond the always-available ones. - You write no error handling: every node has a `_error` output weft adds, and weft catches the body's failures there when the program wires it. Never declare `_error`; metadata or a body that does is refused. The `fake` tests cover both paths, calling `rig.wire_output("_error")` for the caught one. A node with a bus output opens it on its first piece (`ctx.pending_bus`, then `bus.open()`), never before the call that can fail. A node whose body does nothing outside its run beyond answering its caller and the run's own files (no network, no connection, no `ctx.run`, no wait, no tag, no storage beyond this run) sets `"pure": true` in its `features`: a durable run starts it without waiting for the database, and runs it again if its worker died before it sent anything on (when the node reads no stream). A node that reaches outside never sets it: weft fails the first outside call made through the ctx of one that does, naming the flag, and cannot see a socket or file the body opens on its own. A value the node keeps that grows with use goes in a file it edits in place (`ctx.storage(scope).edit`), never on a port. A node that answers a live caller declares `answersCaller` in its `features`. A `validate` rule picks other nodes by a feature they declare (`with: {feature: value}`), never by a type name. The manual has all four. - An infra node gets no `live` test for now: its container is proven inside a real program. Make a scratch project in `.scratch/node-smith/` (inside the project, where Cursor lets a write through without approval; add `.scratch/` to the project's `.gitignore` if it is not listed) with the node under its `nodes/`, then `weft infra start`, `weft infra status`, `weft infra logs`, a run through it (carved with `weft run --from` / `--emit` / `--target`), and, if the node has a baked output, `weft bake` then `weft run --fire` through a trigger that reads it, which proves the saved value, and `weft infra terminate --yes` at the end. Your report says what you ran. Size every unit's `machine` (CPU, memory, GPU) as the manual's "The machine" section says: the user pays for the machine it picks. Mark `"baked": true` on every output that holds as long as the infra does (an address, a connection, a handle), so a run that reads only those uses the saved values instead of running the node. When the container changes such a value itself (a reset button), it `POST`s the new value to `$WEFT_VALUES_URL` and answers with what that said. For the full rule, go and read Infra node in the manual. diff --git a/tangle/cursor/.cursor/skills/weft-node-authoring/SKILL.md b/tangle/cursor/.cursor/skills/weft-node-authoring/SKILL.md index 0264d5b82..1e54ecb7a 100644 --- a/tangle/cursor/.cursor/skills/weft-node-authoring/SKILL.md +++ b/tangle/cursor/.cursor/skills/weft-node-authoring/SKILL.md @@ -389,7 +389,7 @@ You emit only through `ctx.pulse_downstream(NodeOutput::new().set(port, value))` ports you did not emit are closed, which is the skip signal downstream. For user-added output ports use `ctx.fan_declared(...)`. -**Share what is slow to build**: a client pool, a process. `ctx.shared(&access, |opened| async move { .. })` builds it the first time a run asks and hands every later run on that worker the same one; it is built again when the connection's values change, or after nobody used it for the project's idle window (300 seconds by default, `weft workers set --shared-idle-seconds`). The closure returns a `WeftResult` of the thing and you get a handle you use as the thing. `ctx.shared("name", |()| async { .. })` is the same when nothing is built from a connection, and `.with_limit(n)` before the `.await` lets at most `n` runs of a worker hold it at once (a pool kept under a database's connection limit). A database client opened on every run pays the connection each time; for a worked pool, go and read `nodes/base_catalog/postgres/postgres.rs`. +**Share what is slow to build**: a client pool, a process. `ctx.shared(&access, |opened| async move { .. })` builds it the first time a run asks and hands every later run on that worker the same one; it is built again when the connection's values change, or after nobody used it for the project's idle window (300 seconds by default, `weft workers set --shared-idle-seconds`). The closure returns a `WeftResult` of the thing and you get a handle you use as the thing. `ctx.shared("name", |()| async { .. })` is the same when nothing is built from a connection, and `.with_limit(n)` before the `.await` lets at most `n` runs of a worker hold it at once. That never keeps a database under its limit, because any number of workers run a node at once. So have your pool close a connection after a few idle seconds (the Postgres nodes use 5) and never hand a run one idle for longer, because a Cloud Run copy has no CPU between requests to run your close timer, and leave the waiting to whatever sits in front of the database, such as a PostgresDatabase's pooler. If your node keeps a connection idle on purpose (one that `LISTEN`s), the database may close it (a PostgresDatabase does after a minute), so lift `idle_session_timeout` for that session when it connects, the way PostgresWatchQuery does. A database client opened on every run pays the connection each time; for a worked pool, go and read `nodes/base_catalog/postgres/postgres.rs`. If the worker dies while your body runs, a fast run (the default) ends cancelled, and a durable run fails the node: its start was on record before the body began, so the next worker knows it was running and fails it rather than start it again. With `_error` wired, that failure goes to `_error` like any other. A node marked `pure` is the exception: a firing of it that had sent nothing on and read no stream runs again, since it did nothing anyone could see. The one other body weft starts again on purpose is one parked on `ctx.await_signal`, which replays from the top when its answer comes, so the work before the wait goes through `ctx.run(...)`, which gives back the recorded result. An emit on a port already sent before the wait does nothing on the replay, so a node can send `stats` at once, wait, then send `image`; what it cannot do is wait after yielding on or closing a stream output, which is refused. In a run that cannot pause (a route's caller on the line, `recorded: false`, a bus open), `ctx.await_signal` holds in your call instead and returns the answer without a replay; if the run stays quiet for its trigger's `holdSecs`, the call fails with `WeftError::WaitGaveUp`, which your node may match and handle (test it with `rig.signal_given_up()`). diff --git a/tangle/devin-desktop/.devin/agents/node-smith.md b/tangle/devin-desktop/.devin/agents/node-smith.md index ca20b0e80..8b7ed0e10 100644 --- a/tangle/devin-desktop/.devin/agents/node-smith.md +++ b/tangle/devin-desktop/.devin/agents/node-smith.md @@ -46,7 +46,7 @@ After each write under `nodes/`, the edit hook prints what `weft validate` finds 4. Write the files: - `metadata.json`: [the contract] verbatim, plus presentation. A trigger's metadata also gets `firesWith`: EVERY field the wake payload can carry, with `?` on the ones that only sometimes arrive, so a real firing (and a hand-typed `weft run --fire`) is checked against that shape before `run` starts. The check is exact: a firing carrying a field you never named is refused just like one missing a field you required. Name only what you fan onto ports and the trigger dies the first time the provider sends anything else; when the payload comes from a connection's events, copy the names from `events..fields` in the service's recipe. Leave it off only when the node genuinely has nothing to name (it reads its own connection instead of the payload, or its fields are per-instance config the author typed in), and note why in the report. - Spell every name in `metadata.json` the way Spelling in the `weft-node-authoring` manual says, and clear every catalog warning `weft describe-nodes --list` prints for your node before you report. If the ports a program adds to your node are named after data from outside weft (a query's parameters, a template's ids), set `features.addedPortsAreDataKeys`, and they keep that data's spelling. - - `mod.rs`: the body, one job, no orchestration, no plumbing, no fallbacks; every failure is a loud `node_bail!` error. Every value you emit on a port is at most 100 KB, checked on your emission, so a node whose output could be bigger bounds it itself, decided now: it pages it (an offset in, `nextOffset` and `hasMore` out), caps it (a size setting with a default and a `truncated` output, `weft::storage::cap_text` does the cut), or puts the bytes in storage and emits the file value; the manual's "The wire limit" has the rule. A client, a pool or a process slow to build (a database connection pool, a set of interpreter processes) is built with `ctx.shared`, which keeps it on the worker for its runs to share, rather than opened on every run. + - `mod.rs`: the body, one job, no orchestration, no plumbing, no fallbacks; every failure is a loud `node_bail!` error. Every value you emit on a port is at most 100 KB, checked on your emission, so a node whose output could be bigger bounds it itself, decided now: it pages it (an offset in, `nextOffset` and `hasMore` out), caps it (a size setting with a default and a `truncated` output, `weft::storage::cap_text` does the cut), or puts the bytes in storage and emits the file value; the manual's "The wire limit" has the rule. A client, a pool or a process slow to build (a database connection pool, a set of interpreter processes) is built with `ctx.shared`, which keeps it on the worker for its runs to share, rather than opened on every run. Any number of workers can run your node at once (they scale up to the project's `--max-instances`, and a change of program or worker settings runs the old copies beside the new for a while), so a node that talks to something with a fixed capacity (a database's connections, an API's rate limit) holds none of it while idle: close an idle connection within seconds, and check how long one sat idle before handing it to a run, because a Cloud Run copy has no CPU between requests to run your timer. `.with_limit(n)` on `ctx.shared` only caps how many of one worker's runs hold it at once; what protects the database (or the API) is that end itself making the extra clients wait or refusing them cleanly (a PostgresDatabase has a pooler for that), and your node waiting, or failing loudly, on that answer. If your node holds a connection idle on purpose (one that `LISTEN`s), the database may close it after a while (a PostgresDatabase does after a minute), so lift that limit for its own session when it connects, the way PostgresWatchQuery sends `SELECT set_config('idle_session_timeout', '0', false) FROM pg_settings WHERE name = 'idle_session_timeout'` (a no-op on a Postgres without the setting). - `deps.toml`: only if you need crates or OS packages beyond the always-available ones. - Every node has the language's `_error` output, so you write no error handling for it and never declare `_error`: weft puts the failure there when the program wires it. A node that can fail at the outside world has `fake` tests for both paths, calling `rig.wire_output("_error")` for the caught one. A node whose body does nothing outside its run beyond answering its caller and the run's own files (no network, no connection, no `ctx.run`, no wait, no tag, no storage beyond this run) sets `"pure": true` in its `features`: a durable run starts it without waiting for the database, and runs it again if its worker died before it sent anything on (when the node reads no stream). A node that reaches outside never sets it: weft fails the first outside call made through the ctx of one that does, naming the flag, and cannot see a socket or file the body opens on its own. A value the node keeps that grows with use goes in a file it edits in place (`ctx.storage(scope).edit`), never on a port. A node that answers a live caller declares `answersCaller` in its `features`. A `validate` rule picks other nodes by a feature they declare (`with: {feature: value}`), never by a type name. The manual has all four. - An infra node gets no `live` test for now: its container is proven inside a real program. Make a scratch project in `.scratch/node-smith/` (inside the project, so the allow list lets you write there without asking; add `.scratch/` to the project's `.gitignore` if it is not listed) with the node under its `nodes/`, then `weft infra start`, `weft infra status`, `weft infra logs`, a run through it (carved with `weft run --from` / `--emit` / `--target`), and, if the node has a baked output, `weft bake` then `weft run --fire` through a trigger that reads it, which proves the saved value, and `weft infra terminate --yes` at the end. Your report says what you ran. Size every unit's `machine` (CPU, memory, GPU) as the manual's "The machine" section says: the user pays for the machine it picks. Mark `"baked": true` on every output that holds as long as the infra does (an address, a connection, a handle), so a run that reads only those uses the saved values instead of running the node. When the container changes such a value itself (a reset button), it `POST`s the new value to `$WEFT_VALUES_URL` and answers with what that said. For the full rule, go and read Infra node in the manual. diff --git a/tangle/devin-desktop/.windsurf/skills/weft-node-authoring/SKILL.md b/tangle/devin-desktop/.windsurf/skills/weft-node-authoring/SKILL.md index 3441b6da2..6ca3228d6 100644 --- a/tangle/devin-desktop/.windsurf/skills/weft-node-authoring/SKILL.md +++ b/tangle/devin-desktop/.windsurf/skills/weft-node-authoring/SKILL.md @@ -393,7 +393,7 @@ You emit only through `ctx.pulse_downstream(NodeOutput::new().set(port, value))` ports you did not emit are closed, which is the skip signal downstream. For user-added output ports use `ctx.fan_declared(...)`. -**Share what is slow to build**: a client pool, a process. `ctx.shared(&access, |opened| async move { .. })` builds it the first time a run asks and hands every later run on that worker the same one; it is built again when the connection's values change, or after nobody used it for the project's idle window (300 seconds by default, `weft workers set --shared-idle-seconds`). The closure returns a `WeftResult` of the thing and you get a handle you use as the thing. `ctx.shared("name", |()| async { .. })` is the same when nothing is built from a connection, and `.with_limit(n)` before the `.await` lets at most `n` runs of a worker hold it at once (a pool kept under a database's connection limit). A database client opened on every run pays the connection each time; for a worked pool, go and read `nodes/base_catalog/postgres/postgres.rs`. +**Share what is slow to build**: a client pool, a process. `ctx.shared(&access, |opened| async move { .. })` builds it the first time a run asks and hands every later run on that worker the same one; it is built again when the connection's values change, or after nobody used it for the project's idle window (300 seconds by default, `weft workers set --shared-idle-seconds`). The closure returns a `WeftResult` of the thing and you get a handle you use as the thing. `ctx.shared("name", |()| async { .. })` is the same when nothing is built from a connection, and `.with_limit(n)` before the `.await` lets at most `n` runs of a worker hold it at once. That never keeps a database under its limit, because any number of workers run a node at once. So have your pool close a connection after a few idle seconds (the Postgres nodes use 5) and never hand a run one idle for longer, because a Cloud Run copy has no CPU between requests to run your close timer, and leave the waiting to whatever sits in front of the database, such as a PostgresDatabase's pooler. If your node keeps a connection idle on purpose (one that `LISTEN`s), the database may close it (a PostgresDatabase does after a minute), so lift `idle_session_timeout` for that session when it connects, the way PostgresWatchQuery does. A database client opened on every run pays the connection each time; for a worked pool, go and read `nodes/base_catalog/postgres/postgres.rs`. If the worker dies while your body runs, a fast run (the default) ends cancelled, and a durable run fails the node: its start was on record before the body began, so the next worker knows it was running and fails it rather than start it again. With `_error` wired, that failure goes to `_error` like any other. A node marked `pure` is the exception: a firing of it that had sent nothing on and read no stream runs again, since it did nothing anyone could see. The one other body weft starts again on purpose is one parked on `ctx.await_signal`, which replays from the top when its answer comes, so the work before the wait goes through `ctx.run(...)`, which gives back the recorded result. An emit on a port already sent before the wait does nothing on the replay, so a node can send `stats` at once, wait, then send `image`; what it cannot do is wait after yielding on or closing a stream output, which is refused. In a run that cannot pause (a route's caller on the line, `recorded: false`, a bus open), `ctx.await_signal` holds in your call instead and returns the answer without a replay; if the run stays quiet for its trigger's `holdSecs`, the call fails with `WeftError::WaitGaveUp`, which your node may match and handle (test it with `rig.signal_given_up()`). diff --git a/tangle/fallback/.agents/skills/node-smith/SKILL.md b/tangle/fallback/.agents/skills/node-smith/SKILL.md index ee3aed2fc..25ec11d67 100644 --- a/tangle/fallback/.agents/skills/node-smith/SKILL.md +++ b/tangle/fallback/.agents/skills/node-smith/SKILL.md @@ -46,7 +46,7 @@ After each write under `nodes/`, the edit hook prints what `weft validate` finds 4. Write the files: - `metadata.json`: [the contract] verbatim, plus presentation. For a trigger, add `firesWith`: EVERY field the wake payload can carry, with `?` on the ones that only sometimes arrive, so the engine can hold a real firing, and a hand-typed `weft run --fire`, to that shape before `run` runs. It holds it exactly: a firing carrying a field you never named is refused just like one missing a field you required. Name only what you fan onto ports and the trigger dies the first time the provider sends anything else; when the payload comes from a connection's events, copy the names from `events..fields` in the service's recipe. Only skip it when the node has no fixed payload to name (it reads its own connection, or its fields are the author's own per-instance config), and say why in the report. - Spell every name in `metadata.json` the way Spelling in the `weft-node-authoring` manual says, and clear every catalog warning `weft describe-nodes --list` prints for your node before you report. If the ports a program adds to your node are named after data from outside weft (a query's parameters, a template's ids), set `features.addedPortsAreDataKeys`, and they keep that data's spelling. - - `mod.rs`: the body, one job, no orchestration, no plumbing, no fallbacks; every failure is a loud `node_bail!` error. Every value you emit on a port weighs at most 100 KB as JSON, checked on your emission, so a node whose output could be bigger bounds it itself: it pages (an offset in, where the next page starts and whether there is one out), caps (a size setting with a default and an output saying it was cut, `weft::storage::cap_text` does the cutting), or puts the bytes in storage and emits the file value; the manual's "The wire limit" has the rule. A client, a pool or a process slow to build (a database connection pool, a set of interpreter processes) is built with `ctx.shared`, which keeps it on the worker for its runs to share, rather than opened on every run. + - `mod.rs`: the body, one job, no orchestration, no plumbing, no fallbacks; every failure is a loud `node_bail!` error. Every value you emit on a port weighs at most 100 KB as JSON, checked on your emission, so a node whose output could be bigger bounds it itself: it pages (an offset in, where the next page starts and whether there is one out), caps (a size setting with a default and an output saying it was cut, `weft::storage::cap_text` does the cutting), or puts the bytes in storage and emits the file value; the manual's "The wire limit" has the rule. A client, a pool or a process slow to build (a database connection pool, a set of interpreter processes) is built with `ctx.shared`, which keeps it on the worker for its runs to share, rather than opened on every run. Any number of workers can run your node at once (they scale up to the project's `--max-instances`, and a change of program or worker settings runs the old copies beside the new for a while), so a node that talks to something with a fixed capacity (a database's connections, an API's rate limit) holds none of it while idle: close an idle connection within seconds, and check how long one sat idle before handing it to a run, because a Cloud Run copy has no CPU between requests to run your timer. `.with_limit(n)` on `ctx.shared` only caps how many of one worker's runs hold it at once; what protects the database (or the API) is that end itself making the extra clients wait or refusing them cleanly (a PostgresDatabase has a pooler for that), and your node waiting, or failing loudly, on that answer. If your node holds a connection idle on purpose (one that `LISTEN`s), the database may close it after a while (a PostgresDatabase does after a minute), so lift that limit for its own session when it connects, the way PostgresWatchQuery sends `SELECT set_config('idle_session_timeout', '0', false) FROM pg_settings WHERE name = 'idle_session_timeout'` (a no-op on a Postgres without the setting). - `deps.toml`: only if you need crates or OS packages beyond the always-available ones. - The node writes no error handling and never declares `_error`: weft gives every node that output and puts the body's failure on it when the program wires it. A node that can fail at a call has `fake` tests for both paths, calling `rig.wire_output("_error")` before the run for the caught one. A node whose body does nothing outside its run beyond answering its caller and the run's own files (no network, no connection, no `ctx.run`, no wait, no tag, no storage beyond this run) sets `"pure": true` in its `features`: a durable run starts it without waiting for the database, and runs it again if its worker died before it sent anything on (when the node reads no stream). A node that reaches outside never sets it: weft fails the first outside call made through the ctx of one that does, naming the flag, and cannot see a socket or file the body opens on its own. A value the node keeps that grows with use goes in a file it edits in place (`ctx.storage(scope).edit`), never on a port. A node that answers a live caller declares `answersCaller` in its `features`. A `validate` rule picks other nodes by a feature they declare (`with: {feature: value}`), never by a type name. The manual has all four. - An infra node gets no `live` test for now: its container is proven inside a real program. Make a scratch project in `.scratch/node-smith/` (inside the project, so it needs no access beyond the project's own; add `.scratch/` to the project's `.gitignore` if it is not listed) with the node under its `nodes/`, then `weft infra start`, `weft infra status`, `weft infra logs`, a run through it (carved with `weft run --from` / `--emit` / `--target`), and, if the node has a baked output, `weft bake` then `weft run --fire` through a trigger that reads it, which proves the saved value, and `weft infra terminate --yes` at the end. Your report says what you ran. Size every unit's `machine` (CPU, memory, GPU) as the manual's "The machine" section says: the user pays for the machine it picks. Mark `"baked": true` on every output that holds as long as the infra does (an address, a connection, a handle), so a run that reads only those uses the saved values instead of running the node. When the container changes such a value itself (a reset button), it `POST`s the new value to `$WEFT_VALUES_URL` and answers with what that said. For the full rule, go and read Infra node in the manual. diff --git a/tangle/fallback/.agents/skills/weft-node-authoring/SKILL.md b/tangle/fallback/.agents/skills/weft-node-authoring/SKILL.md index 046d443e2..45db3ba8b 100644 --- a/tangle/fallback/.agents/skills/weft-node-authoring/SKILL.md +++ b/tangle/fallback/.agents/skills/weft-node-authoring/SKILL.md @@ -390,7 +390,7 @@ You emit only through `ctx.pulse_downstream(NodeOutput::new().set(port, value))` ports you did not emit are closed, which is the skip signal downstream. For user-added output ports use `ctx.fan_declared(...)`. -**Share what is slow to build**: a client pool, a process. `ctx.shared(&access, |opened| async move { .. })` builds it the first time a run asks and hands every later run on that worker the same one; it is built again when the connection's values change, or after nobody used it for the project's idle window (300 seconds by default, `weft workers set --shared-idle-seconds`). The closure returns a `WeftResult` of the thing and you get a handle you use as the thing. `ctx.shared("name", |()| async { .. })` is the same when nothing is built from a connection, and `.with_limit(n)` before the `.await` lets at most `n` runs of a worker hold it at once (a pool kept under a database's connection limit). A database client opened on every run pays the connection each time; for a worked pool, go and read `nodes/base_catalog/postgres/postgres.rs`. +**Share what is slow to build**: a client pool, a process. `ctx.shared(&access, |opened| async move { .. })` builds it the first time a run asks and hands every later run on that worker the same one; it is built again when the connection's values change, or after nobody used it for the project's idle window (300 seconds by default, `weft workers set --shared-idle-seconds`). The closure returns a `WeftResult` of the thing and you get a handle you use as the thing. `ctx.shared("name", |()| async { .. })` is the same when nothing is built from a connection, and `.with_limit(n)` before the `.await` lets at most `n` runs of a worker hold it at once. That never keeps a database under its limit, because any number of workers run a node at once. So have your pool close a connection after a few idle seconds (the Postgres nodes use 5) and never hand a run one idle for longer, because a Cloud Run copy has no CPU between requests to run your close timer, and leave the waiting to whatever sits in front of the database, such as a PostgresDatabase's pooler. If your node keeps a connection idle on purpose (one that `LISTEN`s), the database may close it (a PostgresDatabase does after a minute), so lift `idle_session_timeout` for that session when it connects, the way PostgresWatchQuery does. A database client opened on every run pays the connection each time; for a worked pool, go and read `nodes/base_catalog/postgres/postgres.rs`. If the worker dies while your body runs, a fast run (the default) ends cancelled, and a durable run fails the node: its start was on record before the body began, so the next worker knows it was running and fails it rather than start it again. With `_error` wired, that failure goes to `_error` like any other. A node marked `pure` is the exception: a firing of it that had sent nothing on and read no stream runs again, since it did nothing anyone could see. The one other body weft starts again on purpose is one parked on `ctx.await_signal`, which replays from the top when its answer comes, so the work before the wait goes through `ctx.run(...)`, which gives back the recorded result. An emit on a port already sent before the wait does nothing on the replay, so a node can send `stats` at once, wait, then send `image`; what it cannot do is wait after yielding on or closing a stream output, which is refused. In a run that cannot pause (a route's caller on the line, `recorded: false`, a bus open), `ctx.await_signal` holds in your call instead and returns the answer without a replay; if the run stays quiet for its trigger's `holdSecs`, the call fails with `WeftError::WaitGaveUp`, which your node may match and handle (test it with `rig.signal_given_up()`). diff --git a/tangle/gemini-cli/.gemini/agents/node-smith.md b/tangle/gemini-cli/.gemini/agents/node-smith.md index b6d7ecc29..a33d4add3 100644 --- a/tangle/gemini-cli/.gemini/agents/node-smith.md +++ b/tangle/gemini-cli/.gemini/agents/node-smith.md @@ -50,7 +50,7 @@ After each write under `nodes/`, the edit hook prints what `weft validate` finds 4. Write the files: - `metadata.json`: [the contract] verbatim, plus presentation. When the node is a trigger, declare `firesWith` too, naming EVERY field the wake payload can carry, with `?` on the ones that only sometimes arrive: the engine checks a real firing, and a hand-typed `weft run --fire`, against that shape before `run` starts, and the check is exact, so a firing carrying a field you never named is refused just like one missing a field you required. Name only what you fan onto ports and the trigger dies the first time the provider sends anything else; when the payload comes from a connection's events, copy the names from `events..fields` in the service's recipe. Skip it only when the node truly has nothing to name (it reads its own connection instead of the payload, or its fields come from the author's own per-instance config), and say so in the report. - Spell every name in `metadata.json` the way Spelling in the `weft-node-authoring` manual says, and clear every catalog warning `weft describe-nodes --list` prints for your node before you report. If the ports a program adds to your node are named after data from outside weft (a query's parameters, a template's ids), set `features.addedPortsAreDataKeys`, and they keep that data's spelling. - - `mod.rs`: the body, one job, no orchestration, no plumbing, no fallbacks; every failure is a loud `node_bail!` error. Every value you emit on a port is at most 100 KB, checked on your emission, so a node whose output could be bigger bounds it itself, decided as you write it: it pages (an offset in, `nextOffset` and `hasMore` out), caps (a size setting with a default, plus an output like `truncated` saying it cut; `weft::storage::cap_text` does the cutting), or puts the bytes in storage and emits the file value; the manual's "The wire limit" has the rule. A client, a pool or a process slow to build (a database connection pool, a set of interpreter processes) is built with `ctx.shared`, which keeps it on the worker for its runs to share, rather than opened on every run. + - `mod.rs`: the body, one job, no orchestration, no plumbing, no fallbacks; every failure is a loud `node_bail!` error. Every value you emit on a port is at most 100 KB, checked on your emission, so a node whose output could be bigger bounds it itself, decided as you write it: it pages (an offset in, `nextOffset` and `hasMore` out), caps (a size setting with a default, plus an output like `truncated` saying it cut; `weft::storage::cap_text` does the cutting), or puts the bytes in storage and emits the file value; the manual's "The wire limit" has the rule. A client, a pool or a process slow to build (a database connection pool, a set of interpreter processes) is built with `ctx.shared`, which keeps it on the worker for its runs to share, rather than opened on every run. Any number of workers can run your node at once (they scale up to the project's `--max-instances`, and a change of program or worker settings runs the old copies beside the new for a while), so a node that talks to something with a fixed capacity (a database's connections, an API's rate limit) holds none of it while idle: close an idle connection within seconds, and check how long one sat idle before handing it to a run, because a Cloud Run copy has no CPU between requests to run your timer. `.with_limit(n)` on `ctx.shared` only caps how many of one worker's runs hold it at once; what protects the database (or the API) is that end itself making the extra clients wait or refusing them cleanly (a PostgresDatabase has a pooler for that), and your node waiting, or failing loudly, on that answer. If your node holds a connection idle on purpose (one that `LISTEN`s), the database may close it after a while (a PostgresDatabase does after a minute), so lift that limit for its own session when it connects, the way PostgresWatchQuery sends `SELECT set_config('idle_session_timeout', '0', false) FROM pg_settings WHERE name = 'idle_session_timeout'` (a no-op on a Postgres without the setting). - `deps.toml`: only if you need crates or OS packages beyond the always-available ones. - You write no error handling: weft gives every node an `_error` output and catches its failures there when the program wires it. Never declare `_error` yourself, and set no feature flag for it: there is none, and the loader refuses the old one. Your `fake` tests cover both paths, calling `rig.wire_output("_error")` for the caught one. A node with a `Bus` output opens it on its first piece (`ctx.pending_bus(..)`, then `bus.open().await?` once the first piece is in hand), never before the call that can fail, so that failure is still caught; the manual's bus section has the shape. A node whose body does nothing outside its run beyond answering its caller and the run's own files (no network, no connection, no `ctx.run`, no wait, no tag, no storage beyond this run) sets `"pure": true` in its `features`: a durable run starts it without waiting for the database, and runs it again if its worker died before it sent anything on (when the node reads no stream). A node that reaches outside never sets it: weft fails the first outside call made through the ctx of one that does, naming the flag, and cannot see a socket or file the body opens on its own. A value the node keeps that grows with use goes in a file it edits in place (`ctx.storage(scope).edit`), never on a port. A node that answers a live caller declares `answersCaller` in its `features`. A `validate` rule picks other nodes by a feature they declare (`with: {feature: value}`), never by a type name. The manual has all four. - An infra node gets no `live` test for now: its container is proven inside a real program. Make a scratch project in `.scratch/node-smith/` (inside the project, where Gemini lets you write; add `.scratch/` to the project's `.gitignore` if it is not listed) with the node under its `nodes/`, then `weft infra start`, `weft infra status`, `weft infra logs`, a run through it (carved with `weft run --from` / `--emit` / `--target`), and, if the node has a baked output, `weft bake` then `weft run --fire` through a trigger that reads it, which proves the saved value, and `weft infra terminate --yes` at the end. Your report says what you ran. Size every unit's `machine` (CPU, memory, GPU) as the manual's "The machine" section says: the user pays for the machine it picks. Mark `"baked": true` on every output that holds as long as the infra does (an address, a connection, a handle), so a run that reads only those uses the saved values instead of running the node. When the container changes such a value itself (a reset button), it `POST`s the new value to `$WEFT_VALUES_URL` and answers with what that said. For the full rule, go and read Infra node in the manual. diff --git a/tangle/gemini-cli/.gemini/skills/weft-node-authoring/SKILL.md b/tangle/gemini-cli/.gemini/skills/weft-node-authoring/SKILL.md index 8d2da5c2a..60270ff46 100644 --- a/tangle/gemini-cli/.gemini/skills/weft-node-authoring/SKILL.md +++ b/tangle/gemini-cli/.gemini/skills/weft-node-authoring/SKILL.md @@ -392,7 +392,7 @@ You emit only through `ctx.pulse_downstream(NodeOutput::new().set(port, value))` ports you did not emit are closed, which is the skip signal downstream. For user-added output ports use `ctx.fan_declared(...)`. -**Share what is slow to build**: a client pool, a process. `ctx.shared(&access, |opened| async move { .. })` builds it the first time a run asks and hands every later run on that worker the same one; it is built again when the connection's values change, or after nobody used it for the project's idle window (300 seconds by default, `weft workers set --shared-idle-seconds`). The closure returns a `WeftResult` of the thing and you get a handle you use as the thing. `ctx.shared("name", |()| async { .. })` is the same when nothing is built from a connection, and `.with_limit(n)` before the `.await` lets at most `n` runs of a worker hold it at once (a pool kept under a database's connection limit). A database client opened on every run pays the connection each time; for a worked pool, go and read `nodes/base_catalog/postgres/postgres.rs`. +**Share what is slow to build**: a client pool, a process. `ctx.shared(&access, |opened| async move { .. })` builds it the first time a run asks and hands every later run on that worker the same one; it is built again when the connection's values change, or after nobody used it for the project's idle window (300 seconds by default, `weft workers set --shared-idle-seconds`). The closure returns a `WeftResult` of the thing and you get a handle you use as the thing. `ctx.shared("name", |()| async { .. })` is the same when nothing is built from a connection, and `.with_limit(n)` before the `.await` lets at most `n` runs of a worker hold it at once. That never keeps a database under its limit, because any number of workers run a node at once. So have your pool close a connection after a few idle seconds (the Postgres nodes use 5) and never hand a run one idle for longer, because a Cloud Run copy has no CPU between requests to run your close timer, and leave the waiting to whatever sits in front of the database, such as a PostgresDatabase's pooler. If your node keeps a connection idle on purpose (one that `LISTEN`s), the database may close it (a PostgresDatabase does after a minute), so lift `idle_session_timeout` for that session when it connects, the way PostgresWatchQuery does. A database client opened on every run pays the connection each time; for a worked pool, go and read `nodes/base_catalog/postgres/postgres.rs`. If the worker dies while your body runs, a fast run (the default) ends cancelled, and a durable run fails the node: its start was on record before the body began, so the next worker knows it was running and fails it rather than start it again. With `_error` wired, that failure goes to `_error` like any other. A node marked `pure` is the exception: a firing of it that had sent nothing on and read no stream runs again, since it did nothing anyone could see. The one other body weft starts again on purpose is one parked on `ctx.await_signal`, which replays from the top when its answer comes, so the work before the wait goes through `ctx.run(...)`, which gives back the recorded result. An emit on a port already sent before the wait does nothing on the replay, so a node can send `stats` at once, wait, then send `image`; what it cannot do is wait after yielding on or closing a stream output, which is refused. In a run that cannot pause (a route's caller on the line, `recorded: false`, a bus open), `ctx.await_signal` holds in your call instead and returns the answer without a replay; if the run stays quiet for its trigger's `holdSecs`, the call fails with `WeftError::WaitGaveUp`, which your node may match and handle (test it with `rig.signal_given_up()`). diff --git a/tangle/github-copilot/.github/agents/node-smith.agent.md b/tangle/github-copilot/.github/agents/node-smith.agent.md index eeda37e79..16df38bbd 100644 --- a/tangle/github-copilot/.github/agents/node-smith.agent.md +++ b/tangle/github-copilot/.github/agents/node-smith.agent.md @@ -47,7 +47,7 @@ After each write under `nodes/`, the edit hook prints what `weft validate` finds 4. Write the files: - `metadata.json`: [the contract] verbatim, plus presentation. When the node is a trigger, this includes `firesWith`: EVERY field the wake payload can carry, `?` on the ones that only sometimes arrive. The engine checks a real firing (and a hand-typed `weft run --fire`) before your `run` starts, and the check is exact, so a firing carrying a field you never named is refused just like one missing a field you required. Name only what you fan onto ports and the trigger dies the first time the provider sends anything else; when the payload comes from a connection's events, copy the names from `events..fields` in the service's recipe. Skip it only when the node cannot name a payload shape at all (it reads its own connection instead of the wake payload, or its fields are per-instance config typed in by the author), and say why in the report. - Spell every name in `metadata.json` the way Spelling in the `weft-node-authoring` manual says, and clear every catalog warning `weft describe-nodes --list` prints for your node before you report. If the ports a program adds to your node are named after data from outside weft (a query's parameters, a template's ids), set `features.addedPortsAreDataKeys`, and they keep that data's spelling. - - `mod.rs`: the body, one job, no orchestration, no plumbing, no fallbacks; every failure is a loud `node_bail!` error. Every value you emit on a port is at most 100 KB, checked on your emission, so a node whose output could be bigger bounds it itself, decided as you write it: it pages (an offset in, `nextOffset` and `hasMore` out), caps (a size setting with a default and a `truncated` output, `weft::storage::cap_text` does the cut) or puts the bytes in storage and emits the file value; the manual's "The wire limit" has the rule. A client, a pool or a process slow to build (a database connection pool, a set of interpreter processes) is built with `ctx.shared`, which keeps it on the worker for its runs to share, rather than opened on every run. + - `mod.rs`: the body, one job, no orchestration, no plumbing, no fallbacks; every failure is a loud `node_bail!` error. Every value you emit on a port is at most 100 KB, checked on your emission, so a node whose output could be bigger bounds it itself, decided as you write it: it pages (an offset in, `nextOffset` and `hasMore` out), caps (a size setting with a default and a `truncated` output, `weft::storage::cap_text` does the cut) or puts the bytes in storage and emits the file value; the manual's "The wire limit" has the rule. A client, a pool or a process slow to build (a database connection pool, a set of interpreter processes) is built with `ctx.shared`, which keeps it on the worker for its runs to share, rather than opened on every run. Any number of workers can run your node at once (they scale up to the project's `--max-instances`, and a change of program or worker settings runs the old copies beside the new for a while), so a node that talks to something with a fixed capacity (a database's connections, an API's rate limit) holds none of it while idle: close an idle connection within seconds, and check how long one sat idle before handing it to a run, because a Cloud Run copy has no CPU between requests to run your timer. `.with_limit(n)` on `ctx.shared` only caps how many of one worker's runs hold it at once; what protects the database (or the API) is that end itself making the extra clients wait or refusing them cleanly (a PostgresDatabase has a pooler for that), and your node waiting, or failing loudly, on that answer. If your node holds a connection idle on purpose (one that `LISTEN`s), the database may close it after a while (a PostgresDatabase does after a minute), so lift that limit for its own session when it connects, the way PostgresWatchQuery sends `SELECT set_config('idle_session_timeout', '0', false) FROM pg_settings WHERE name = 'idle_session_timeout'` (a no-op on a Postgres without the setting). - `deps.toml`: only if you need crates or OS packages beyond the always-available ones. - You write no error handling: every node has an `_error` output that weft adds, and catches the body's failures there when the program wires it. Never declare `_error` yourself. Its `fake` tests cover both paths, calling `rig.wire_output("_error")` for the caught one. A node whose body does nothing outside its run beyond answering its caller and the run's own files (no network, no connection, no `ctx.run`, no wait, no tag, no storage beyond this run) sets `"pure": true` in its `features`: a durable run starts it without waiting for the database, and runs it again if its worker died before it sent anything on (when the node reads no stream). A node that reaches outside never sets it: weft fails the first outside call made through the ctx of one that does, naming the flag, and cannot see a socket or file the body opens on its own. A value the node keeps that grows with use goes in a file it edits in place (`ctx.storage(scope).edit`), never on a port. A node that answers a live caller declares `answersCaller` in its `features`. A `validate` rule picks other nodes by a feature they declare (`with: {feature: value}`), never by a type name. The manual has all four. - An infra node gets no `live` test for now: its container is proven inside a real program. Make a scratch project in `.scratch/node-smith/` (inside the project; add `.scratch/` to the project's `.gitignore` if it is not listed) with the node under its `nodes/`, then `weft infra start`, `weft infra status`, `weft infra logs`, a run through it (carved with `weft run --from` / `--emit` / `--target`), and, if the node has a baked output, `weft bake` then `weft run --fire` through a trigger that reads it, which proves the saved value, and `weft infra terminate --yes` at the end. Your report says what you ran. Size every unit's `machine` (CPU, memory, GPU) as the manual's "The machine" section says: the user pays for the machine it picks. Mark `"baked": true` on every output that holds as long as the infra does (an address, a connection, a handle), so a run that reads only those uses the saved values instead of running the node. When the container changes such a value itself (a reset button), it `POST`s the new value to `$WEFT_VALUES_URL` and answers with what that said. For the full rule, go and read Infra node in the manual. diff --git a/tangle/github-copilot/.github/skills/weft-node-authoring/SKILL.md b/tangle/github-copilot/.github/skills/weft-node-authoring/SKILL.md index e99e531c8..8bc8a8683 100644 --- a/tangle/github-copilot/.github/skills/weft-node-authoring/SKILL.md +++ b/tangle/github-copilot/.github/skills/weft-node-authoring/SKILL.md @@ -392,7 +392,7 @@ You emit only through `ctx.pulse_downstream(NodeOutput::new().set(port, value))` ports you did not emit are closed, which is the skip signal downstream. For user-added output ports use `ctx.fan_declared(...)`. -**Share what is slow to build**: a client pool, a process. `ctx.shared(&access, |opened| async move { .. })` builds it the first time a run asks and hands every later run on that worker the same one; it is built again when the connection's values change, or after nobody used it for the project's idle window (300 seconds by default, `weft workers set --shared-idle-seconds`). The closure returns a `WeftResult` of the thing and you get a handle you use as the thing. `ctx.shared("name", |()| async { .. })` is the same when nothing is built from a connection, and `.with_limit(n)` before the `.await` lets at most `n` runs of a worker hold it at once (a pool kept under a database's connection limit). A database client opened on every run pays the connection each time; for a worked pool, go and read `nodes/base_catalog/postgres/postgres.rs`. +**Share what is slow to build**: a client pool, a process. `ctx.shared(&access, |opened| async move { .. })` builds it the first time a run asks and hands every later run on that worker the same one; it is built again when the connection's values change, or after nobody used it for the project's idle window (300 seconds by default, `weft workers set --shared-idle-seconds`). The closure returns a `WeftResult` of the thing and you get a handle you use as the thing. `ctx.shared("name", |()| async { .. })` is the same when nothing is built from a connection, and `.with_limit(n)` before the `.await` lets at most `n` runs of a worker hold it at once. That never keeps a database under its limit, because any number of workers run a node at once. So have your pool close a connection after a few idle seconds (the Postgres nodes use 5) and never hand a run one idle for longer, because a Cloud Run copy has no CPU between requests to run your close timer, and leave the waiting to whatever sits in front of the database, such as a PostgresDatabase's pooler. If your node keeps a connection idle on purpose (one that `LISTEN`s), the database may close it (a PostgresDatabase does after a minute), so lift `idle_session_timeout` for that session when it connects, the way PostgresWatchQuery does. A database client opened on every run pays the connection each time; for a worked pool, go and read `nodes/base_catalog/postgres/postgres.rs`. If the worker dies while your body runs, a fast run (the default) ends cancelled, and a durable run fails the node: its start was on record before the body began, so the next worker knows it was running and fails it rather than start it again. With `_error` wired, that failure goes to `_error` like any other. A node marked `pure` is the exception: a firing of it that had sent nothing on and read no stream runs again, since it did nothing anyone could see. The one other body weft starts again on purpose is one parked on `ctx.await_signal`, which replays from the top when its answer comes, so the work before the wait goes through `ctx.run(...)`, which gives back the recorded result. An emit on a port already sent before the wait does nothing on the replay, so a node can send `stats` at once, wait, then send `image`; what it cannot do is wait after yielding on or closing a stream output, which is refused. In a run that cannot pause (a route's caller on the line, `recorded: false`, a bus open), `ctx.await_signal` holds in your call instead and returns the answer without a replay; if the run stays quiet for its trigger's `holdSecs`, the call fails with `WeftError::WaitGaveUp`, which your node may match and handle (test it with `rig.signal_given_up()`). diff --git a/tangle/junie/.junie/agents/node-smith.md b/tangle/junie/.junie/agents/node-smith.md index a9a7c12ae..ecff21f37 100644 --- a/tangle/junie/.junie/agents/node-smith.md +++ b/tangle/junie/.junie/agents/node-smith.md @@ -49,7 +49,7 @@ After each write under `nodes/`, the edit hook prints what `weft validate` finds 4. Write the files: - `metadata.json`: [the contract] verbatim, plus presentation. A trigger's contract includes `firesWith`: name EVERY field the wake payload can carry, with `?` on the ones that only sometimes arrive, so the engine can check a real firing, and a hand-typed `weft run --fire`, against that shape before `run` starts. The check is exact: a firing carrying a field you never named is refused just like one missing a field you required. Name only what you fan onto ports and the trigger dies the first time the provider sends anything else; when the payload comes from a connection's events, copy the names from `events..fields` in the service's recipe. Skip it only when there is genuinely nothing to name (the node reads its own connection, or its fields are the author's own per-instance config), and record why in the report. - Spell every name in `metadata.json` the way Spelling in the `weft-node-authoring` manual says, and clear every catalog warning `weft describe-nodes --list` prints for your node before you report. If the ports a program adds to your node are named after data from outside weft (a query's parameters, a template's ids), set `features.addedPortsAreDataKeys`, and they keep that data's spelling. - - `mod.rs`: the body, one job, no orchestration, no plumbing, no fallbacks; every failure is a loud `node_bail!` error. Every value you emit on a port is at most 100 KB, checked on your emission, so a node whose output could be bigger bounds it itself, decided now (a page at a time with where the next starts and `hasMore`, or a cap setting with a default plus a `truncated`-style output) or puts the bytes in storage and emits the file value; the manual's "The wire limit" has the rule. A client, a pool or a process slow to build (a database connection pool, a set of interpreter processes) is built with `ctx.shared`, which keeps it on the worker for its runs to share, rather than opened on every run. + - `mod.rs`: the body, one job, no orchestration, no plumbing, no fallbacks; every failure is a loud `node_bail!` error. Every value you emit on a port is at most 100 KB, checked on your emission, so a node whose output could be bigger bounds it itself, decided now (a page at a time with where the next starts and `hasMore`, or a cap setting with a default plus a `truncated`-style output) or puts the bytes in storage and emits the file value; the manual's "The wire limit" has the rule. A client, a pool or a process slow to build (a database connection pool, a set of interpreter processes) is built with `ctx.shared`, which keeps it on the worker for its runs to share, rather than opened on every run. Any number of workers can run your node at once (they scale up to the project's `--max-instances`, and a change of program or worker settings runs the old copies beside the new for a while), so a node that talks to something with a fixed capacity (a database's connections, an API's rate limit) holds none of it while idle: close an idle connection within seconds, and check how long one sat idle before handing it to a run, because a Cloud Run copy has no CPU between requests to run your timer. `.with_limit(n)` on `ctx.shared` only caps how many of one worker's runs hold it at once; what protects the database (or the API) is that end itself making the extra clients wait or refusing them cleanly (a PostgresDatabase has a pooler for that), and your node waiting, or failing loudly, on that answer. If your node holds a connection idle on purpose (one that `LISTEN`s), the database may close it after a while (a PostgresDatabase does after a minute), so lift that limit for its own session when it connects, the way PostgresWatchQuery sends `SELECT set_config('idle_session_timeout', '0', false) FROM pg_settings WHERE name = 'idle_session_timeout'` (a no-op on a Postgres without the setting). - `deps.toml`: only if you need crates or OS packages beyond the always-available ones. - The node writes no error handling: weft gives every node an `_error` output and catches its failures there when the program wires it. It never declares `_error` itself and sets no flag for it. A node that can fail in the outside world (a service, a database, a file, a model) has `fake` tests for both paths, calling `rig.wire_output("_error")` for the caught one and reading `outcome.output("_error")?`. A node whose body does nothing outside its run beyond answering its caller and the run's own files (no network, no connection, no `ctx.run`, no wait, no tag, no storage beyond this run) sets `"pure": true` in its `features`: a durable run starts it without waiting for the database, and runs it again if its worker died before it sent anything on (when the node reads no stream). A node that reaches outside never sets it: weft fails the first outside call made through the ctx of one that does, naming the flag, and cannot see a socket or file the body opens on its own. A value the node keeps that grows with use goes in a file it edits in place (`ctx.storage(scope).edit`), never on a port. A node that answers a live caller declares `answersCaller` in its `features`. A `validate` rule picks other nodes by a feature they declare (`with: {feature: value}`), never by a type name. The manual has all four. - An infra node gets no `live` test for now: its container is proven inside a real program. Make a scratch project in `.scratch/node-smith/` (inside the project, since Junie asks before any edit outside it; add `.scratch/` to the project's `.gitignore` if it is not listed) with the node under its `nodes/`, then `weft infra start`, `weft infra status`, `weft infra logs`, a run through it (carved with `weft run --from` / `--emit` / `--target`), and, if the node has a baked output, `weft bake` then `weft run --fire` through a trigger that reads it, which proves the saved value, and `weft infra terminate --yes` at the end. Your report says what you ran. Size every unit's `machine` (CPU, memory, GPU) as the manual's "The machine" section says: the user pays for the machine it picks. Mark `"baked": true` on every output that holds as long as the infra does (an address, a connection, a handle), so a run that reads only those uses the saved values instead of running the node. When the container changes such a value itself (a reset button), it `POST`s the new value to `$WEFT_VALUES_URL` and answers with what that said. For the full rule, go and read Infra node in the manual. diff --git a/tangle/junie/.junie/skills/weft-node-authoring/SKILL.md b/tangle/junie/.junie/skills/weft-node-authoring/SKILL.md index bed4e6e39..18ca63d64 100644 --- a/tangle/junie/.junie/skills/weft-node-authoring/SKILL.md +++ b/tangle/junie/.junie/skills/weft-node-authoring/SKILL.md @@ -391,7 +391,7 @@ You emit only through `ctx.pulse_downstream(NodeOutput::new().set(port, value))` ports you did not emit are closed, which is the skip signal downstream. For user-added output ports use `ctx.fan_declared(...)`. -**Share what is slow to build**: a client pool, a process. `ctx.shared(&access, |opened| async move { .. })` builds it the first time a run asks and hands every later run on that worker the same one; it is built again when the connection's values change, or after nobody used it for the project's idle window (300 seconds by default, `weft workers set --shared-idle-seconds`). The closure returns a `WeftResult` of the thing and you get a handle you use as the thing. `ctx.shared("name", |()| async { .. })` is the same when nothing is built from a connection, and `.with_limit(n)` before the `.await` lets at most `n` runs of a worker hold it at once (a pool kept under a database's connection limit). A database client opened on every run pays the connection each time; for a worked pool, go and read `nodes/base_catalog/postgres/postgres.rs`. +**Share what is slow to build**: a client pool, a process. `ctx.shared(&access, |opened| async move { .. })` builds it the first time a run asks and hands every later run on that worker the same one; it is built again when the connection's values change, or after nobody used it for the project's idle window (300 seconds by default, `weft workers set --shared-idle-seconds`). The closure returns a `WeftResult` of the thing and you get a handle you use as the thing. `ctx.shared("name", |()| async { .. })` is the same when nothing is built from a connection, and `.with_limit(n)` before the `.await` lets at most `n` runs of a worker hold it at once. That never keeps a database under its limit, because any number of workers run a node at once. So have your pool close a connection after a few idle seconds (the Postgres nodes use 5) and never hand a run one idle for longer, because a Cloud Run copy has no CPU between requests to run your close timer, and leave the waiting to whatever sits in front of the database, such as a PostgresDatabase's pooler. If your node keeps a connection idle on purpose (one that `LISTEN`s), the database may close it (a PostgresDatabase does after a minute), so lift `idle_session_timeout` for that session when it connects, the way PostgresWatchQuery does. A database client opened on every run pays the connection each time; for a worked pool, go and read `nodes/base_catalog/postgres/postgres.rs`. If the worker dies while your body runs, a fast run (the default) ends cancelled, and a durable run fails the node: its start was on record before the body began, so the next worker knows it was running and fails it rather than start it again. With `_error` wired, that failure goes to `_error` like any other. A node marked `pure` is the exception: a firing of it that had sent nothing on and read no stream runs again, since it did nothing anyone could see. The one other body weft starts again on purpose is one parked on `ctx.await_signal`, which replays from the top when its answer comes, so the work before the wait goes through `ctx.run(...)`, which gives back the recorded result. An emit on a port already sent before the wait does nothing on the replay, so a node can send `stats` at once, wait, then send `image`; what it cannot do is wait after yielding on or closing a stream output, which is refused. In a run that cannot pause (a route's caller on the line, `recorded: false`, a bus open), `ctx.await_signal` holds in your call instead and returns the answer without a replay; if the run stays quiet for its trigger's `holdSecs`, the call fails with `WeftError::WaitGaveUp`, which your node may match and handle (test it with `rig.signal_given_up()`). diff --git a/tangle/kilo-code/.kilo/agent/node-smith.md b/tangle/kilo-code/.kilo/agent/node-smith.md index 7592160f5..fb1ac00a4 100644 --- a/tangle/kilo-code/.kilo/agent/node-smith.md +++ b/tangle/kilo-code/.kilo/agent/node-smith.md @@ -49,7 +49,7 @@ After each write under `nodes/`, the edit hook prints what `weft validate` finds 4. Write the files: - `metadata.json`: [the contract] verbatim, plus presentation. If you are building a trigger, also declare `firesWith`: EVERY field the wake payload can carry, with `?` on the ones that only sometimes arrive, so the engine can check a real firing, and a hand-typed `weft run --fire`, against that shape before `run` executes. The check is exact: a firing carrying a field you never named is refused just like one missing a field you required. Name only what you fan onto ports and the trigger dies the first time the provider sends anything else; when the payload comes from a connection's events, copy the names from `events..fields` in the service's recipe. Skip it only when the node has no fixed shape to declare (it reads its own connection instead of the payload, or its fields are the author's own per-instance config), and say why in the report. - Spell every name in `metadata.json` the way Spelling in the `weft-node-authoring` manual says, and clear every catalog warning `weft describe-nodes --list` prints for your node before you report. If the ports a program adds to your node are named after data from outside weft (a query's parameters, a template's ids), set `features.addedPortsAreDataKeys`, and they keep that data's spelling. - - `mod.rs`: the body, one job, no orchestration, no plumbing, no fallbacks; every failure is a loud `node_bail!` error. Every value you emit on a port weighs at most 100 KB as JSON, checked on your emission, so an output that can grow is bounded when you write the node: paged (an offset in, `nextOffset` and `hasMore` out), capped (a size setting with a default plus an output saying it was cut, `weft::storage::cap_text` does the cut), or the bytes put in storage and the file value emitted; the manual's "The wire limit" has the rule. A client, a pool or a process slow to build (a database connection pool, a set of interpreter processes) is built with `ctx.shared`, which keeps it on the worker for its runs to share, rather than opened on every run. + - `mod.rs`: the body, one job, no orchestration, no plumbing, no fallbacks; every failure is a loud `node_bail!` error. Every value you emit on a port weighs at most 100 KB as JSON, checked on your emission, so an output that can grow is bounded when you write the node: paged (an offset in, `nextOffset` and `hasMore` out), capped (a size setting with a default plus an output saying it was cut, `weft::storage::cap_text` does the cut), or the bytes put in storage and the file value emitted; the manual's "The wire limit" has the rule. A client, a pool or a process slow to build (a database connection pool, a set of interpreter processes) is built with `ctx.shared`, which keeps it on the worker for its runs to share, rather than opened on every run. Any number of workers can run your node at once (they scale up to the project's `--max-instances`, and a change of program or worker settings runs the old copies beside the new for a while), so a node that talks to something with a fixed capacity (a database's connections, an API's rate limit) holds none of it while idle: close an idle connection within seconds, and check how long one sat idle before handing it to a run, because a Cloud Run copy has no CPU between requests to run your timer. `.with_limit(n)` on `ctx.shared` only caps how many of one worker's runs hold it at once; what protects the database (or the API) is that end itself making the extra clients wait or refusing them cleanly (a PostgresDatabase has a pooler for that), and your node waiting, or failing loudly, on that answer. If your node holds a connection idle on purpose (one that `LISTEN`s), the database may close it after a while (a PostgresDatabase does after a minute), so lift that limit for its own session when it connects, the way PostgresWatchQuery sends `SELECT set_config('idle_session_timeout', '0', false) FROM pg_settings WHERE name = 'idle_session_timeout'` (a no-op on a Postgres without the setting). - `deps.toml`: only if you need crates or OS packages beyond the always-available ones. - Every node has a `_error` output the language adds, which catches its failures when the program wires it, so you write no error handling and never declare `_error` yourself. A node that reaches outside (a service, a database, a file, a model) has `fake` tests for both paths, calling `rig.wire_output("_error")` for the caught one. A node whose body does nothing outside its run beyond answering its caller and the run's own files (no network, no connection, no `ctx.run`, no wait, no tag, no storage beyond this run) sets `"pure": true` in its `features`: a durable run starts it without waiting for the database, and runs it again if its worker died before it sent anything on (when the node reads no stream). A node that reaches outside never sets it: weft fails the first outside call made through the ctx of one that does, naming the flag, and cannot see a socket or file the body opens on its own. A value the node keeps that grows with use goes in a file it edits in place (`ctx.storage(scope).edit`), never on a port. A node that answers a live caller declares `answersCaller` in its `features`. A `validate` rule picks other nodes by a feature they declare (`with: {feature: value}`), never by a type name. The manual has all four. - An infra node gets no `live` test for now: its container is proven inside a real program. Make a scratch project in `.scratch/node-smith/` (inside the project, so writing there never asks; add `.scratch/` to the project's `.gitignore` if it is not listed) with the node under its `nodes/`, then `weft infra start`, `weft infra status`, `weft infra logs`, a run through it (carved with `weft run --from` / `--emit` / `--target`), and, if the node has a baked output, `weft bake` then `weft run --fire` through a trigger that reads it, which proves the saved value, and `weft infra terminate --yes` at the end. Your report says what you ran. Size every unit's `machine` (CPU, memory, GPU) as the manual's "The machine" section says: the user pays for the machine it picks. Mark `"baked": true` on every output that holds as long as the infra does (an address, a connection, a handle), so a run that reads only those uses the saved values instead of running the node. When the container changes such a value itself (a reset button), it `POST`s the new value to `$WEFT_VALUES_URL` and answers with what that said. For the full rule, go and read Infra node in the manual. diff --git a/tangle/kilo-code/.kilo/skills/weft-node-authoring/SKILL.md b/tangle/kilo-code/.kilo/skills/weft-node-authoring/SKILL.md index 234973447..e81b4dfa5 100644 --- a/tangle/kilo-code/.kilo/skills/weft-node-authoring/SKILL.md +++ b/tangle/kilo-code/.kilo/skills/weft-node-authoring/SKILL.md @@ -389,7 +389,7 @@ You emit only through `ctx.pulse_downstream(NodeOutput::new().set(port, value))` ports you did not emit are closed, which is the skip signal downstream. For user-added output ports use `ctx.fan_declared(...)`. -**Share what is slow to build**: a client pool, a process. `ctx.shared(&access, |opened| async move { .. })` builds it the first time a run asks and hands every later run on that worker the same one; it is built again when the connection's values change, or after nobody used it for the project's idle window (300 seconds by default, `weft workers set --shared-idle-seconds`). The closure returns a `WeftResult` of the thing and you get a handle you use as the thing. `ctx.shared("name", |()| async { .. })` is the same when nothing is built from a connection, and `.with_limit(n)` before the `.await` lets at most `n` runs of a worker hold it at once (a pool kept under a database's connection limit). A database client opened on every run pays the connection each time; for a worked pool, go and read `nodes/base_catalog/postgres/postgres.rs`. +**Share what is slow to build**: a client pool, a process. `ctx.shared(&access, |opened| async move { .. })` builds it the first time a run asks and hands every later run on that worker the same one; it is built again when the connection's values change, or after nobody used it for the project's idle window (300 seconds by default, `weft workers set --shared-idle-seconds`). The closure returns a `WeftResult` of the thing and you get a handle you use as the thing. `ctx.shared("name", |()| async { .. })` is the same when nothing is built from a connection, and `.with_limit(n)` before the `.await` lets at most `n` runs of a worker hold it at once. That never keeps a database under its limit, because any number of workers run a node at once. So have your pool close a connection after a few idle seconds (the Postgres nodes use 5) and never hand a run one idle for longer, because a Cloud Run copy has no CPU between requests to run your close timer, and leave the waiting to whatever sits in front of the database, such as a PostgresDatabase's pooler. If your node keeps a connection idle on purpose (one that `LISTEN`s), the database may close it (a PostgresDatabase does after a minute), so lift `idle_session_timeout` for that session when it connects, the way PostgresWatchQuery does. A database client opened on every run pays the connection each time; for a worked pool, go and read `nodes/base_catalog/postgres/postgres.rs`. If the worker dies while your body runs, a fast run (the default) ends cancelled, and a durable run fails the node: its start was on record before the body began, so the next worker knows it was running and fails it rather than start it again. With `_error` wired, that failure goes to `_error` like any other. A node marked `pure` is the exception: a firing of it that had sent nothing on and read no stream runs again, since it did nothing anyone could see. The one other body weft starts again on purpose is one parked on `ctx.await_signal`, which replays from the top when its answer comes, so the work before the wait goes through `ctx.run(...)`, which gives back the recorded result. An emit on a port already sent before the wait does nothing on the replay, so a node can send `stats` at once, wait, then send `image`; what it cannot do is wait after yielding on or closing a stream output, which is refused. In a run that cannot pause (a route's caller on the line, `recorded: false`, a bus open), `ctx.await_signal` holds in your call instead and returns the answer without a replay; if the run stays quiet for its trigger's `holdSecs`, the call fails with `WeftError::WaitGaveUp`, which your node may match and handle (test it with `rig.signal_given_up()`). diff --git a/tangle/opencode/.opencode/agents/node-smith.md b/tangle/opencode/.opencode/agents/node-smith.md index 244b66fb9..f64370108 100644 --- a/tangle/opencode/.opencode/agents/node-smith.md +++ b/tangle/opencode/.opencode/agents/node-smith.md @@ -55,7 +55,7 @@ After each write under `nodes/`, the edit hook prints what `weft validate` finds 4. Write the files: - `metadata.json`: [the contract] verbatim, plus presentation. For a trigger, that includes `firesWith`, naming EVERY field the wake payload can carry, with `?` on the ones that only sometimes arrive: the engine checks a real firing, and a hand-typed `weft run --fire`, against that shape before `run` starts, and the check is exact, so a firing carrying a field you never named is refused just like one missing a field you required. Name only what you fan onto ports and the trigger dies the first time the provider sends anything else; when the payload comes from a connection's events, copy the names from `events..fields` in the service's recipe. Only skip it when the node has nothing to name (it reads its own connection instead of the payload, or its fields are the author's own per-instance config), and note why in the report. - Spell every name in `metadata.json` the way Spelling in the `weft-node-authoring` manual says, and clear every catalog warning `weft describe-nodes --list` prints for your node before you report. If the ports a program adds to your node are named after data from outside weft (a query's parameters, a template's ids), set `features.addedPortsAreDataKeys`, and they keep that data's spelling. - - `mod.rs`: the body, one job, no orchestration, no plumbing, no fallbacks; every failure is a loud `node_bail!` error. Every value you emit on a port is at most 100 KB, checked on your emission, so a node whose output could be bigger bounds it itself (pages it with an offset in and `nextOffset`/`hasMore` out, caps it with a size setting and a `truncated`-style output, or a `LIMIT`) or puts the bytes in storage and emits the file value; the manual's "The wire limit" has the rule. A client, a pool or a process slow to build (a database connection pool, a set of interpreter processes) is built with `ctx.shared`, which keeps it on the worker for its runs to share, rather than opened on every run. + - `mod.rs`: the body, one job, no orchestration, no plumbing, no fallbacks; every failure is a loud `node_bail!` error. Every value you emit on a port is at most 100 KB, checked on your emission, so a node whose output could be bigger bounds it itself (pages it with an offset in and `nextOffset`/`hasMore` out, caps it with a size setting and a `truncated`-style output, or a `LIMIT`) or puts the bytes in storage and emits the file value; the manual's "The wire limit" has the rule. A client, a pool or a process slow to build (a database connection pool, a set of interpreter processes) is built with `ctx.shared`, which keeps it on the worker for its runs to share, rather than opened on every run. Any number of workers can run your node at once (they scale up to the project's `--max-instances`, and a change of program or worker settings runs the old copies beside the new for a while), so a node that talks to something with a fixed capacity (a database's connections, an API's rate limit) holds none of it while idle: close an idle connection within seconds, and check how long one sat idle before handing it to a run, because a Cloud Run copy has no CPU between requests to run your timer. `.with_limit(n)` on `ctx.shared` only caps how many of one worker's runs hold it at once; what protects the database (or the API) is that end itself making the extra clients wait or refusing them cleanly (a PostgresDatabase has a pooler for that), and your node waiting, or failing loudly, on that answer. If your node holds a connection idle on purpose (one that `LISTEN`s), the database may close it after a while (a PostgresDatabase does after a minute), so lift that limit for its own session when it connects, the way PostgresWatchQuery sends `SELECT set_config('idle_session_timeout', '0', false) FROM pg_settings WHERE name = 'idle_session_timeout'` (a no-op on a Postgres without the setting). - `deps.toml`: only if you need crates or OS packages beyond the always-available ones. - A node writes no error handling: every node has a `_error` output weft adds, and weft catches the body's failures there when the program wires it. It never declares `_error` itself. A node that can fail on something outside (a service, a database, a file, a model) has `fake` tests for both paths, calling `rig.wire_output("_error")` for the caught one. A node with a `Bus` output opens it on its first piece (`ctx.pending_bus`, then `open()`), never before the call that can fail, so that failure still reaches `_error`. A node whose body does nothing outside its run beyond answering its caller and the run's own files (no network, no connection, no `ctx.run`, no wait, no tag, no storage beyond this run) sets `"pure": true` in its `features`: a durable run starts it without waiting for the database, and runs it again if its worker died before it sent anything on (when the node reads no stream). A node that reaches outside never sets it: weft fails the first outside call made through the ctx of one that does, naming the flag, and cannot see a socket or file the body opens on its own. A value the node keeps that grows with use goes in a file it edits in place (`ctx.storage(scope).edit`), never on a port. A node that answers a live caller declares `answersCaller` in its `features`. A `validate` rule picks other nodes by a feature they declare (`with: {feature: value}`), never by a type name. The manual has all four. - An infra node gets no `live` test for now: its container is proven inside a real program. Make a scratch project in `.scratch/node-smith/` (inside the project, so writing there never asks; add `.scratch/` to the project's `.gitignore` if it is not listed) with the node under its `nodes/`, then `weft infra start`, `weft infra status`, `weft infra logs`, a run through it (carved with `weft run --from` / `--emit` / `--target`), and, if the node has a baked output, `weft bake` then `weft run --fire` through a trigger that reads it, which proves the saved value, and `weft infra terminate --yes` at the end. Your report says what you ran. Size every unit's `machine` (CPU, memory, GPU) as the manual's "The machine" section says: the user pays for the machine it picks. Mark `"baked": true` on every output that holds as long as the infra does (an address, a connection, a handle), so a run that reads only those uses the saved values instead of running the node. When the container changes such a value itself (a reset button), it `POST`s the new value to `$WEFT_VALUES_URL` and answers with what that said. For the full rule, go and read Infra node in the manual. diff --git a/tangle/opencode/.opencode/skills/weft-node-authoring/SKILL.md b/tangle/opencode/.opencode/skills/weft-node-authoring/SKILL.md index 3f7103159..c74fa0c68 100644 --- a/tangle/opencode/.opencode/skills/weft-node-authoring/SKILL.md +++ b/tangle/opencode/.opencode/skills/weft-node-authoring/SKILL.md @@ -394,7 +394,7 @@ You emit only through `ctx.pulse_downstream(NodeOutput::new().set(port, value))` ports you did not emit are closed, which is the skip signal downstream. For user-added output ports use `ctx.fan_declared(...)`. -**Share what is slow to build**: a client pool, a process. `ctx.shared(&access, |opened| async move { .. })` builds it the first time a run asks and hands every later run on that worker the same one; it is built again when the connection's values change, or after nobody used it for the project's idle window (300 seconds by default, `weft workers set --shared-idle-seconds`). The closure returns a `WeftResult` of the thing and you get a handle you use as the thing. `ctx.shared("name", |()| async { .. })` is the same when nothing is built from a connection, and `.with_limit(n)` before the `.await` lets at most `n` runs of a worker hold it at once (a pool kept under a database's connection limit). A database client opened on every run pays the connection each time; for a worked pool, go and read `nodes/base_catalog/postgres/postgres.rs`. +**Share what is slow to build**: a client pool, a process. `ctx.shared(&access, |opened| async move { .. })` builds it the first time a run asks and hands every later run on that worker the same one; it is built again when the connection's values change, or after nobody used it for the project's idle window (300 seconds by default, `weft workers set --shared-idle-seconds`). The closure returns a `WeftResult` of the thing and you get a handle you use as the thing. `ctx.shared("name", |()| async { .. })` is the same when nothing is built from a connection, and `.with_limit(n)` before the `.await` lets at most `n` runs of a worker hold it at once. That never keeps a database under its limit, because any number of workers run a node at once. So have your pool close a connection after a few idle seconds (the Postgres nodes use 5) and never hand a run one idle for longer, because a Cloud Run copy has no CPU between requests to run your close timer, and leave the waiting to whatever sits in front of the database, such as a PostgresDatabase's pooler. If your node keeps a connection idle on purpose (one that `LISTEN`s), the database may close it (a PostgresDatabase does after a minute), so lift `idle_session_timeout` for that session when it connects, the way PostgresWatchQuery does. A database client opened on every run pays the connection each time; for a worked pool, go and read `nodes/base_catalog/postgres/postgres.rs`. If the worker dies while your body runs, a fast run (the default) ends cancelled, and a durable run fails the node: its start was on record before the body began, so the next worker knows it was running and fails it rather than start it again. With `_error` wired, that failure goes to `_error` like any other. A node marked `pure` is the exception: a firing of it that had sent nothing on and read no stream runs again, since it did nothing anyone could see. The one other body weft starts again on purpose is one parked on `ctx.await_signal`, which replays from the top when its answer comes, so the work before the wait goes through `ctx.run(...)`, which gives back the recorded result. An emit on a port already sent before the wait does nothing on the replay, so a node can send `stats` at once, wait, then send `image`; what it cannot do is wait after yielding on or closing a stream output, which is refused. In a run that cannot pause (a route's caller on the line, `recorded: false`, a bus open), `ctx.await_signal` holds in your call instead and returns the answer without a replay; if the run stays quiet for its trigger's `holdSecs`, the call fails with `WeftError::WaitGaveUp`, which your node may match and handle (test it with `rig.signal_given_up()`).