[iskakaushik]

Builds on #222, now merged.

Problem

Shadow::materialize_conf rewrites the managed shadow's postgresql.conf on every start with shared_buffers = 256MB, max_wal_size = 512MB and min_wal_size = 128MB, and empties postgresql.auto.conf. Operators cannot tune the shadow, and ALTER SYSTEM on it is lost at the next restart.

In a production deployment the shadow held about 4.5 TB on a host with about 66 GB of RAM, with walshadow's out/ segments, the shadow's pg_wal and the shadow's data on one EBS volume:

• Each source WAL byte was written about three times: to out/, to the shadow's pg_wal, and to zero-fill each new segment first (WalInitWrite / WalInitSync waits).
• With max_wal_size = 512MB, restartpoints ran constantly (restartpoint starting: wal, each flushing about 200 MB).
• Once the instance ran out of EBS burst credits it was capped at 312.5 MB/s, and fetch_toast service time on the shadow rose from about 120 µs to 250-800 µs, with 256 MB of shared_buffers in front of a multi-TB TOAST relation.

Change

A [shadow] table in the ch-config TOML holds postgresql.conf settings for the managed shadow. walshadow writes them ahead of the settings it owns, each time it starts the shadow.

[shadow]
shared_buffers = "4GB"
max_wal_size = "16GB"
checkpoint_timeout = "15min"
effective_io_concurrency = 64

Validation fails config load instead of ignoring the entry:

• Settings walshadow derives itself are refused: hot_standby, wal_level, fsync, full_page_writes, autovacuum, listen_addresses, unix_socket_directories, port, restore_command, wal_retrieve_retry_interval, shared_preload_libraries, dynamic_library_path, the floor-derived max_connections / max_worker_processes / max_wal_senders / max_prepared_transactions / max_locks_per_transaction, anything starting with primary_, recovery_ or walshadow., plus config_file, data_directory, hba_file, ident_file and the include directives.
• Names must be lowercase setting names, so an entry can neither alias an owned setting through case folding nor break the conf grammar.
• shared_buffers, max_wal_size, min_wal_size and checkpoint_timeout need a unit, with PostgreSQL's case-sensitive spelling. PostgreSQL reads a bare number in the base unit, 8 kB pages for shared_buffers.
• Everything else passes through quoted. PostgreSQL refuses unknown names and bad values at startup, and the start error already carries the shadow's log tail.

Precedence follows the existing TOML rules (fragments in ch-config.d/ merge per key). There is no CLI flag or source-side row for these settings. They apply when walshadow starts or restarts the shadow; an adopted --keep-shadow-running shadow keeps its running values until it restarts. The settings live in their own module, src/catalog/shadow/settings.rs, and BridgeConf is untouched, so this rebases cleanly against the compact shadow stream work in #223.

Defaults

setting	before	now	why
wal_init_zero	PG default on	off	Zero-filling a new segment doubles the walreceiver's writes. ext4 and XFS stay crash-safe without it, because fdatasync persists the block allocation with the data. The cost: a full disk fails a WAL write rather than segment creation, and allocating as WAL arrives adds some write latency (see the unthrottled variant below). Set true where bandwidth is plentiful or space must be reserved; on ZFS/btrfs also set wal_recycle = false
wal_recycle	PG default on	on	Listed so copy-on-write filesystems have the knob next to wal_init_zero
max_wal_size	512MB	8GB	A restartpoint only recycles segments within max_wal_size of its redo point. At 512MB the walreceiver outruns that window: the main runs below log restartpoints with 54 removed, 0 recycled, so every segment is created and zero-filled. 8GB leaves restartpoints to checkpoint_timeout below about 14 MB/s of WAL. pg_wal can grow to about this size, more when source checkpoints are rare. 16GB would halve WAL-driven restartpoints again at twice the disk
min_wal_size	128MB	128MB	unchanged
shared_buffers	256MB	an eighth of the cgroup or host memory limit, 256MB to 8GB	In shadow value mode the shadow serves TOAST reads and replays TOAST writes. At 256MB, replay keeps evicting dirty buffers, so bridge workers write pages in the fetch path. Sized like the resident_payload_max default, which takes half of the same limit, with the same dedicated-host assumption. Hosts under 2 GB keep 256MB; a 66 GB host gets 8GB. A catalog-only shadow touches few buffers, unless the host has free huge pages, which PostgreSQL then reserves for all of them

Names, units and contexts were checked against REL_16_STABLE through REL_19_STABLE: shared_buffers (postmaster, 8 kB blocks), max_wal_size / min_wal_size (sighup, MB), checkpoint_timeout (sighup, s, 30s-1d), wal_init_zero / wal_recycle (superuser, reloadable), effective_io_concurrency (user). The memory and time unit tables are identical across these versions. Since walshadow rewrites the conf only at start, reloadable settings still take effect at the next shadow start.

Benchmark

These runs use bench/ceramic from #229 on a 64-core x86 host with 123 GB RAM and an EBS volume. The golden snapshot was prepared from main 97f9305: PG 18.6 source and shadow, [toast] mode = "shadow", a backlog of 4.25 GB / 5.64M records. The daemon was pinned to 8 CPUs. A is main 97f9305 and B is this branch. Each table is ab.sh with 4 interleaved rounds per side, shown as mean ± stdev. Other jobs shared the host, and the load per run is noted.

I added these measurements to a local copy of run.sh (they are not in #229):

• writes per source WAL byte, from /proc/<pid>/io write_bytes − cancelled_write_bytes for the daemon and the shadow's live processes over the drain window;
• device writes, from the run's cgroup io.stat from drain start through a sync after the drain;
• WAL-triggered restartpoints, counted in the shadow's log.

The throttled case runs everything (daemon, shadow, source, ClickHouse) in a cgroup with io.max rbps=wbps=312 MB/s on the EBS device. Source data and ClickHouse are on tmpfs, as in #229.

Unthrottled (load 2.5-4.9)

	main	this PR	ratio
rec/s	236,045 ± 2,488	316,940 ± 743	1.34
c_fetch	52.2 ± 1.0 µs	37.2 ± 0.5 µs	0.71
bridge fetch_toast	36.0 ± 0.8 µs	23.8 ± 0.5 µs	0.66
daemon cores	0.94	1.25	1.33
pump wait	0.70 s/s	0.73 s/s
shadow bytes written / source WAL byte	4.12	1.25	0.30
daemon bytes written / source WAL byte	1.00	0.99
device writes, daemon + shadow / source WAL byte	3.74	2.24	0.60
WAL-triggered restartpoints per run	6.5	0

Write-throttled, 312 MB/s read and write (load 5.5-10.4; main drops to 45% of its unthrottled rate, so it is disk-bound)

	main	this PR	ratio
rec/s, sustained	106,833 ± 779	182,080 ± 182	1.70
c_fetch	158 ± 1 µs	82.2 ± 0.5 µs	0.52
bridge fetch_toast	76.5 ± 5.8 µs	25.8 ± 1.5 µs	0.34
daemon cores	0.43	0.73
shadow bytes written / source WAL byte	4.27	1.26	0.29
device writes, daemon + shadow / source WAL byte	3.86	2.25	0.58
WAL-triggered restartpoints per run	13	0

Page cache dropped for the shadow's base/ after restore (dd iflag=nocache count=0 per file, so first reads go to EBS)

	main	this PR	ratio
unthrottled rec/s (load 2.6-6.5)	115,104 ± 326	117,687 ± 235	1.02
unthrottled c_fetch / fetch_toast	145 / 132 µs	141 / 128 µs	0.97
throttled rec/s (load 2.5-7.9)	99,461 ± 726	117,039 ± 123	1.18
throttled c_fetch / fetch_toast	172 / 146 µs	142 / 128 µs	0.83

When fetches are cold reads, both sides wait on EBS reads, so most of the cache gain above relies on the recent TOAST working set fitting in buffers. The gain that remains with a cold cache is the write savings, and it grows when the disk is capped.

Where the shadow's writes go (single unthrottled runs, bytes per source WAL byte)

process	main	this PR
walreceiver (WAL, plus zero-fill on main)	2.10	0.98
checkpointer (restartpoints)	0.28	0
startup (dirty evictions, extension)	1.24	0.26
8 bridge workers (dirty evictions in the fetch path)	0.41	0
bgwriter	0.05	0

Effect of each default (separate ab.sh sessions, 3-4 rounds each; the rows between main and the last run this PR's code with the listed values)

wal_init_zero	max_wal_size	shared_buffers	throttled rec/s	throttled c_fetch	unthrottled rec/s	unthrottled c_fetch	shadow B/B
on	512MB	256MB (main)	107,061 ± 2,165	158 µs	240,252 ± 7,213	51.5 µs	4.1-4.3
on	8GB	256MB	106,625 ± 172	159 µs	237,940 ± 20,502	58.3 µs	4.2-4.4
off	512MB	256MB	142,353 ± 533	112 µs	213,351 ± 15,552	66.3 µs	3.2-3.3
off	8GB	256MB	144,083 ± 432	112 µs	231,233 ± 8,264	60.8 µs	3.2-3.4
off	8GB	2GB			300,677 ± 7,772	41.3 µs	1.41
off	8GB	8GB (default here)	182,080 ± 182	82 µs	316,940 ± 743	37.2 µs	1.25

The unthrottled column mixes sessions with load between 2.5 and 11, so read it as direction only. Under the cap, wal_init_zero = off gives +33% on its own. With 256MB buffers it costs a few percent and about 9 µs of c_fetch on the uncapped disk, and the larger buffers remove that cost. max_wal_size alone barely moves this benchmark, because its 4.25 GB backlog is shorter than one 8GB restartpoint interval (about 4.3 GB of WAL). The PR runs show no restartpoints, and therefore also no recycling. Under sustained load the shadow still restartpoints about every 4.3 GB of WAL, against about 270 MB before, and the segments it recycles need neither zero-fill nor allocation.

Tests

• ShadowSettings unit tests: every size and duration setting parses exactly when its unit is one of PostgreSQL's units for that kind, matched case-sensitively, and fails for a bare number or a unit with no number. Names that could inject lines or alias an owned setting are refused.
• materialize_conf_writes_each_setting_once, over default and overridden settings × no bridge, a bridge and a bridge with a library dir × with and without a walsender: each name appears once, each [shadow] setting is written with its value, and [shadow] refuses every other name walshadow writes. It replaces the conninfo-absent example test.
• shadow_lifecycle::standby_mode_lifecycle starts a standby with [shadow] overrides (including a quoted value with ' and \) and reads every setting back with SHOW. It passes on PG 17 and 18.
• Config wiring: [shadow] through EmitterConfig into the owned shadow's conf, and rejections of an owned key and a bad unit.
• PATH=/usr/lib/postgresql/17/bin:$PATH WALSHADOW_REQUIRE_TOOLS=1 cargo nextest run --workspace --all-targets --locked: 1689 passed, 7 skipped. The WAL fixtures were regenerated with capture.sh first, as CI does.
• cargo fmt --check, cargo clippy --workspace --all-targets -- -D warnings.

🤖 Generated with Claude Code

[serprex]

we should just embrace bare numbers over incomplete list

[serprex]

this doesn't match postgres

[serprex]

created #278 as simpler alternative. tuning shared_buffers should be handled outside of walshadow