[venkat1701]

Closes #62

CrashRecoveryTest stops the engine at a CrashPoint, but every byte written before that point stays on disk. That is a process crash. This branch adds tests for a power failure, where only what was forced is certain to survive, and fixes the five bugs they found.

How the tests see the I/O

The engine now opens every file it writes or syncs through FileHandle.channel, and forces pages.dir and pages.ref (memory-mapped) through FileHandle.force. Both go through FileHandle.channels, a package-private static that defaults to FileChannel::open. Nothing else changes in production: the catalog, the reference journal and the WAL and feed directory syncs used to call FileChannel.open themselves, and CatalogStore and ReferenceJournal now share FileHandle.syncDirectory instead of keeping their own copies. The seam has to sit in FileHandle because the mapped files are written through memory, so a wrapped channel alone can't see when they reach the disk.

PowerLoss (test sources, io.hstore.engine.page) replaces channels for one directory. Per file it records every write, truncation and force, every force of a mapping and every force of a directory. It can cut the power before any of these events (the call throws PowerLoss.Cut; later I/O is no longer recorded), and restore rewrites the directory as the disk could hold it:

• NOTHING: each file holds what its last force saw
• ALL: every unsynced write kept, as after a process crash
• HALF_LAST: the last unsynced write of each file cut in half (a WAL record cut short, half a page)
• FLIP_LAST: one byte flipped in the middle of the last unsynced write, so its checksum fails
• RANDOM: per file a prefix, a random subset or all of the unsynced writes, some applied only in part; mapped files mix 4 KiB blocks from the last force and from the moment of the crash; sometimes one unsynced byte flipped

A file created since its directory was last forced may vanish (always under NOTHING). A file written without going through FileHandle comes back empty, unless it was renamed from a forced .tmp file. PowerLossModelTest checks these rules on plain files.

The tests

PowerLossTest.powerCutsAtRandomEventsRecoverAPrefixOfTheCommits runs a seeded workload for SYNC/ASYNC x PAGE_IMAGES/PAGE_REFERENCES: small commits appending a node to one ordered edge, bulk loads of 300 or 5,000 members (spilled pages, pages written at commit), deletes of earlier bulk edges, aborted transactions with and without spilled pages, checkpoints and compactions. A dry run counts the I/O events, then it cuts the power at four random events, each with a random tear. After restoring it opens the database, checks it, commits once more and checkpoints, with a second power cut somewhere in that sequence (recovery included), restores and checks again. Each check requires:

• the surviving commits are a prefix of the attempted ones
• under SYNC every acknowledged commit survived; under ASYNC every commit covered by a completed checkpoint
• no aborted work appears, and each bulk edge exists with its full size exactly when the prefix says so
• the change feed holds one event per surviving commit, generations 1 to n (this check found bugs 4 and 5)
• the second recovery found the same prefix as the first
• TreeVerifier over every tree and verifyReferences, the same checks hstore check runs

One commit may have been in flight when the power went; it is allowed to be there or not.

Smaller tests pin the four cases from the issue: async commits after the last checkpoint are lost together; a Commit record cut in half or with a flipped byte loses only the unacknowledged commit; a PAGE_REFERENCES commit whose page was torn is discarded although its log records survived.

PowerLossTest took 14 to 28 s and PowerLossModelTest under 1 s in my last runs, on a laptop busy with other builds (load average 13 to 20). -Dhstore.powerloss.seeds=N runs more seeds. Locally I also ran seeds 100 to 111 and 200 to 211 (8 cuts each, the second set with longer workloads) before the feed check existed, and seeds 300 to 311 with it, all passing after the fixes.

Bugs found

1. FORMAT could come back empty (StorageEngine.verifyFormat). It was written with Files.newOutputStream and never forced. After a power loss shortly after creating a database, FORMAT could be empty while the WAL held acknowledged commits. The next open read no format, took it for format 1 and refused the database with uses storage format 1. Now it is written to FORMAT.tmp, forced, renamed and the directory forced. Test: aDatabaseCreatedJustBeforeAPowerLossStillOpens.

2. data/segments was never forced (SegmentFile.open, MappedLongs.open). A new segment file, or pages.dir and pages.ref themselves, could disappear after a power loss even though their contents were forced. With PAGE_REFERENCES, a commit acknowledged after pages.sync() and wal.sync() lost its pages, and recovery discarded it and everything after it. Now the directory is forced when one of these files is created: once per new segment (256 MiB by default) and twice per database. Test: pagesInASegmentCreatedSinceTheLastCheckpointSurvive (fails for PAGE_REFERENCES; PAGE_IMAGES rewrites the pages from the log and passed already).

3. pages.dir could keep next page number 0 (PageDirectory.open). A new directory's header is forced with only magic and format, and the next number is set to 1 after that, to be forced at the first checkpoint. A power cut in between left next number 0; allocate handed out 0 and every checkpoint after that failed with page number out of range: 0, so the database could no longer write pages. Number 0 is never used, so a stored 0 is now read as 1, which also repairs a directory already in that state. Test: aPageDirectoryWhoseFirstPageNumberWasNeverForcedStillHandsOutNumbers.

4. Zeros in the feed read as frames (ChangeFeed.restore, FrameReader). A feed frame is [length][crc32c of payload][generation][payload]; the CRC doesn't cover the header. Where an unsynced frame never reached the disk but a later one did, the zeros read as frames of length 0, CRC 0 (the CRC of an empty payload) and generation 0. restore set lastGeneration back to 0, and recovery appended the events of every replayed commit again, so the feed listed acknowledged commits twice. Test: aZeroedFeedFrameBeforeASurvivingOneDoesNotRepeatEvents.

5. A damaged feed generation hid real events (same code). A frame of generation 3 with one bad byte in its header came back as generation 5898243 and passed the CRC. Recovery skipped the Feed records of the replayed commits as already present, then truncateAfter(3) removed the bad frame, so the event of commit 3, which had been acknowledged, was gone from the feed. In the random suite the same thing dropped the events of two acknowledged commits. Test: aFeedFrameWhoseUnsyncedGenerationWasDamagedDoesNotHideCommittedEvents.

   For 4 and 5, FeedCodec already writes the generation first in the payload, which the CRC covers. A frame is now valid only if its payload is not empty and starts with the header's generation, and restore stops at a frame that isn't newer than the one before it. Covering the header with the CRC would be cleaner but changes the feed format, so I left that alone.

Each test failed before its fix and passes after it. None of the fixes changes a file format.

Not covered

• Directories created by Files.createDirectories are assumed durable, and a deleted file is assumed to stay deleted.
• The workload is single-threaded, so races between the committer and a checkpoint are not explored.
• Files of the database module (PayloadStore, SemanticPlane) don't go through FileHandle and are not modelled.

Docs

docs/transactions/wal-and-recovery.md describes the power loss tests and the new FORMAT write, docs/storage/pages.md the directory syncs and the next-number rule, and docs/storage/change-feed.md the stricter frame check.