[venkat1701]

The background checkpoint used to wait for 256 MiB of WAL (or 256 MiB of pages written) since the last one. A small store never got there. In the scale-1 crash test the child wrote a 4.46 MB WAL and was killed before any checkpoint, so opening the crashed store replayed every commit from an empty catalog.

The threshold is now worked out from the size of the store by default, and it can be set from HQL.

The automatic rule

With nothing set, the threshold is a quarter of the live bytes in the data segments, held between 2 MiB and 256 MiB (CheckpointThreshold.automatic):

Live data	Threshold
up to 8 MiB	2 MiB
12 MiB	3 MiB
64 MiB	16 MiB
512 MiB	128 MiB
1 GiB and more	256 MiB

Why a quarter. Recovery replays every commit since the last checkpoint and then writes the pages those commits changed, so the log it has to redo should stay small next to the store. A checkpoint writes the pages changed since the previous one, which is at most the whole store. Checkpointing once the log reaches a quarter of the live data keeps replay to a quarter of the store's size, and keeps checkpoint page writes to at most about four bytes per byte of log even if every checkpoint rewrites the whole store. A store of 1 GiB or more keeps the old 256 MiB.

Why a 2 MiB floor. Without one, a nearly empty store would pay the fixed cost of a checkpoint (catalog image, reference journal, several fsyncs) every few hundred kilobytes of log. I also tried 4 MiB. The scale-1 crash child writes 4.46 MB of WAL in total, so with 4 MiB it passes the floor only in its last commit or two and is killed before that checkpoint is published: in 10 runs it replayed exactly as much as main. 2 MiB is low enough for that case.

The rule is evaluated on every maintenance pass, so the threshold grows with the store. The write buffer trigger (write_buffer_mb / 2) and the reference count trigger are unchanged.

Waking on commit

The maintenance thread used to look at the WAL only every maintenanceInterval (500 ms). Now a commit that passes the threshold wakes it. At the end of each maintenance pass, StorageEngine.armWake turns the threshold into two marks, one on wal.bytesAppended() and one on pages.bytesWritten(). TransactionManager runs a new afterCommit hook, and StorageEngine.wakeIfDue compares the two counters with the marks: two atomic reads and two volatile reads per commit. When either mark is passed it unparks the maintenance thread, which decides and runs the checkpoint as before. The commit never runs the checkpoint itself. The first commit past a mark sets a flag, and the next maintenance pass clears it, so commits that follow while that checkpoint is pending or running don't wake the thread again. This is its own commit.

CHECKPOINT AT

CHECKPOINT AT 64MB;
CHECKPOINT AT 1 gb;
CHECKPOINT AT 512KB;
CHECKPOINT AT AUTO;

Units are B, KB, MB, GB, TB (KiB, MiB, GiB, TiB also accepted), any case, space optional, powers of 1024 like checkpoint_wal_mb. A size with no unit, a fraction of a byte, zero or a negative size is a syntax error. The statement needs ADMIN, the same check as CHECKPOINT and COMPACT (Executor.administrative), and log_statement = ddl logs it.

The value is kept in the database, not in a file. It goes into a new engine slot, settings (slot 8, tree schema 15, key 1), through a system transaction on main, the same way Dictionary.intern writes symbols. So it is in the WAL, in the next checkpoint and in replay like any other commit. StorageEngine reads it on open before the maintenance thread starts, and checkpointAt updates the in-memory value right after the commit, so a change applies without a restart. CHECKPOINT AT AUTO deletes the key.

Order of precedence: a value stored with CHECKPOINT AT, then an explicit checkpoint_wal_mb (or EngineOptions.withCheckpointWalBytes), then the rule.

Configuration and API

• checkpoint_wal_mb defaults to auto. A number is still megabytes and is now range-checked like the other megabyte settings (1 to 16,777,216). Anything else fails at startup with a message naming the setting.
• EngineOptions.defaults() now uses AUTOMATIC_CHECKPOINTS (0). withCheckpointWalBytes(n) still fixes the threshold, and withAutomaticCheckpoints() goes back to the rule. Negative values are rejected.
• withCheckpointWalBytes(0) changes meaning. It used to checkpoint on every maintenance pass that saw any new WAL. It now means automatic.
• StorageEngine.checkpointThreshold() returns the bytes in use and the source (AUTOMATIC, DATABASE, CONFIGURATION). StorageEngine.checkpointAt(OptionalLong) stores or clears the value.
• STATS gains checkpoint after wal bytes and checkpoint threshold.
• GET /api/stats reports the threshold in use as checkpointWalBytes and adds checkpointSource. The dashboard's "Checkpoint after" row shows the size followed by "automatic", "set by CHECKPOINT AT" or "set by checkpoint_wal_mb".

Compatibility

Older builds cannot open a database that has run CHECKPOINT AT <size>. They fail with no slot registered with id 8, whether the value has been checkpointed or is still only in the WAL; I checked both against main's engine jar. Nothing has been released yet, so this was accepted. The HQL reference says so where the statement is described.

There is no storage format change and FORMAT stays at 8. The new slot has no root until a value is stored, so a database that never ran CHECKPOINT AT has nothing new on disk, and main opens it.

Measurements

A/B, main (0c847f7) against this branch, same probe on both sides. Byte counts only; the laptop was running other builds, so no timing claims.

Crash: the benchmark's crash child (Comparison crash --store hstore --scale 1 --durability async --history 64) killed with SIGKILL after 25,000 acknowledged nodes, then the crashed directory opened with the benchmark's options. "WAL replayed" is the LSN of the first checkpoint after open minus the checkpoint LSN found in the crashed catalog. 10 runs each:

	main	branch
WAL replayed, bytes	4,458,363 in all 10	2,277,380 in all 10
commits replayed	28 in all 10	13 in all 10

All 10 branch runs replayed less than main: 49% of the bytes and 13 of 28 commits. Before the wake-up, with the same floor, only 4 of 10 runs did, because a checkpoint waited for the next 500 ms pass and the kill often came first. With the wake-up the checkpoint starts at the commit that passes 2 MiB, so every run publishes the same checkpoint.

Ingest cost: HStoreStore.ingestNodes then ingestEdges at scale 1, batches of 1,000, async durability, then one checkpoint, as in the benchmark's ingest workloads. Bytes are walBytes + dataBytesWritten from EngineStats. Medians of 5 runs each (range in brackets):

	main	branch
total bytes written	51,069,075 (50,780,465 to 52,049,612)	61,975,108 (61,975,108 to 61,975,139)
page bytes written	12,251,578 (11,962,968 to 13,232,114)	23,157,309 (all 5 runs)
WAL bytes	38,817,497	38,817,799
checkpoints	4 (all 5 runs)	14 (14 to 15)

The extra checkpoints cost about 10.9 MB of page writes at scale 1, 21% more than main writes during ingest. That is the price of the small floor on a small store. The threshold grows with the store, so at 1 GiB of live data and above checkpoints come at the old 256 MiB. With the wake-up, checkpoints start at the same points in every run, which is why the branch's page bytes don't vary.

Tests

• CheckpointThresholdTest (engine):
  • the rule, from an empty store to 100 GiB;
  • precedence of a stored value over the configured one over the rule;
  • a store with default options checkpoints in the background once its WAL passes the floor, and replays fewer commits after halt();
  • with a one-hour maintenance interval, the commit that passes the floor still gets exactly one checkpoint within 10 seconds;
  • a stored value survives close and reopen, and AUTO clears it;
  • a stored value survives a crash at WAL_SYNC, CHECKPOINT_PAGES, CHECKPOINT_CATALOG and CHECKPOINT_JOURNAL;
  • a stored 1 MiB value triggers a background checkpoint at once on an engine configured with Long.MAX_VALUE.
• QueryLanguageTest: CHECKPOINT AT with 64MB, 64 mb, 1GB, 512KB, 1.5 GiB, 4096 b, AUTO, and ten malformed forms. The value survives closing and reopening the database and shows in STATS.
• SecurityTest: a WRITER gets the ADMIN error and the threshold stays automatic.
• ServerTest: checkpoint_wal_mb unset, AUTO, 64 and 512, and the bad values 0, -1, too large and sometimes.
• StudioTest: /api/stats shows automatic, then database and 64 MiB after CHECKPOINT AT 64MB.

Each was run once against the code with its part of the change reverted, and each failed:

Test	Reverted
background checkpoint	fixed 256 MiB threshold
wake test	afterCommit hook removed
persistence and no-restart tests	stored value not loaded or not applied
security test	CheckpointAt left out of administrative
server and Studio tests	main's Setting, ServerConfig and Studio

./mvnw -q install passes on every commit: 242 tests, 13 of them new.