Repository navigation
Add shared raster storage, RF tools, and envelope content hashes - #40
Merged
Merged
Conversation
Add rf-merger, which merges two published RF snapshots by attribution priority. The output partition follows both input indices, coarse suppliers are upscaled with the shared halo reader and paired scaler, and unchanged tiles are hard-linked. Checkpoints record per-category statistics for recovery and the final report. Move TilePool to raster_store so the RF builder and merger share it, and run the merger suite in CI.
Error records a stack trace where it is made and gains Error::Exception, raise, throwing_unwrap and asserting_unwrap. The code style now separates bugs (assertions), recoverable failures (Expected, including all of terrainlib) and unrecoverable failures in tools (exceptions). rf-merger throws for invalid input and production failures, asserts bug-only results, and keeps Expected in its reusable functions. Preflight rejects sRGB on scalar payloads and zoom gaps above 30 levels. A statistics guard keeps a failed merge recoverable, and cancellation saves active tiles before throwing Cancelled. The priority list is parsed with CPLJSON.
link_from only hard-links: it fails if the target node exists or the codecs differ, instead of overwriting or re-encoding. sf-merger cut now keeps the input's layout and codec instead of converting to .glb.
log and ProgressIndicator move to terrainlib/cli_support in namespace cli_support. The SIGINT/SIGTERM handling duplicated in the rf-builder and rf-merger mains becomes cli_support::stop_signal.
store::snapshot owns the .part lifecycle (create, publish, link filesystem check, checkpoint interval) and store::inputs the input record, fingerprints and cache validation. raster_store, rf-builder and rf-merger now use them instead of their own copies, and octree::storage::create/publish give octree stores the same lifecycle.
batch writes <output>.part and publishes it to a new <output>; it no longer resumes or overwrites an existing output. --cache reuses the nodes of an interrupted build whose inputs.tmp matches. The index is checkpointed every two minutes, and SIGINT/SIGTERM finish the active nodes and keep the .part.
merge and cut write <output>.part and publish it to a new <output>. merge no longer skips nodes the output already has; --cache reuses the nodes of an interrupted merge whose inputs.tmp matches, and --overwrite (which was never used) is gone. The merge checkpoints its index every two minutes and stops before the next node on SIGINT/SIGTERM, keeping the .part. NodeWriter no longer writes a leftover .png beside each node.
dag-builder writes <output>.part and publishes it to a new <output>. --resume, --overwrite and ContinuationMode are gone; --cache reuses the nodes of an interrupted build whose inputs.tmp matches, including the level range and root node, so each run stands alone. The index is also checkpointed every two minutes, SIGINT/SIGTERM keep the .part, and debug meshes go to a new <output>-debug. dag-convert-debug fails on an existing output instead of overwriting it. With no users left, StorageSettings::allow_overwrite is removed and saving an existing node is always an error.
The tools now refuse an existing output, so the script no longer creates output directories or passes --resume/--overwrite. An interrupted run's <output>.part is moved to <output>-cache.part and passed as --cache.
cli_support::log_level_names replaces the seven copies of the --verbosity map, and Log::add_file and log_assertion_failures replace the log-file and assertion-handler code duplicated in the rf-builder and rf-merger mains.
The generic .part, publication, cache record and fingerprint behaviour is tested once in store_snapshot.cpp, which takes over the fingerprint and destination variants. The raster, rf and sf suites keep only what their own code does: record contents, stopping and cache linking. Also drops the cut format and stop signal tests, and renames octree/storage/create.h to snapshot.h.
publish removed inputs.tmp before renaming, so a .part whose publication failed could no longer serve as a cache. The record is now removed from the published snapshot after the rename; failing to remove it only logs a warning, as a published snapshot is never a cache.
Linking an unchanged subtree did not check for stop requests or checkpoint the index, so a stop during a large subtree could be ignored until publication. The merger's poll now runs before each linked leaf.
--output out/ was rejected, because the normalized path has an empty filename, and partial_path made out/.part of it. store::snapshot:: normalized strips the separator for every snapshot path, and raster open/create, the rf log paths and the dag debug path use it too.
The names now match io::byte::read and io::byte::write. No behaviour change.
io::byte::write and io::envelope::write no longer have overloads that silently overwrite; every caller passes a WriteMode. All callers keep overwriting, so there is no behaviour change.
Encoding a node (copies, serialization, zstd, hashing) ran one node at a time: on the coordinator in rf_builder and rf_merger, and under the exclusive lock of ThreadSafeStorage in sf_builder and dag_builder. With zstd-best the merger was effectively single-threaded. Storage::save is now split into write_payload, which only reads the layout and codec and may run concurrently, and commit, which records the node in the index. Workers write the payloads; the coordinator, or the exclusive lock, only commits. Reads see a node only after commit. Codecs create payload files exclusively, so a second write of a key, concurrent or later, fails with AlreadyExists and never truncates a file hard-linked into another snapshot. Storage is always indexed now; the unindexed mode and the unused remove operations are gone. Failed saves stop dag_builder. The RF merger cancellation test now stops after the first written tile instead of after a fixed number of polls, which failed when the worker was slow.
vector_mask now offers three functions on Expected, so they chain: load only reads the polygons and rejects broken input, simplify removes vertices with CGAL's topology-preserving polyline simplification, and transform reprojects while subdividing edges until the result is within the tolerance of the curved edges. Tolerances are in metres, converted to SRS units on a coarse grid. The previous API moves unchanged to vector_mask_deprecated.h until sf-merger and rf-builder migrate.
get_logger created the logger lazily on the first call without synchronisation. Since payloads are written on the worker threads, the first log calls can come from several threads at once; each saw no logger, and all but one failed with "logger with name 'LOG' already exists". The tools hid this by calling Log::init in main, before starting threads. The logger is now a function-local static, created once on first use. Log::init only sets the level, so it can also be called after something has logged.
The macro read and set a plain static bool, a data race when several threads reach the call site at once, which could log the warning more than once. The flag is now an atomic, claimed with exchange before logging.
LogBackoff incremented a plain counter, a data race when several threads reach the call site at once, and the macro read the count again for the message, so another thread's increment could show up there. The counter is now atomic, and count_occurrence counts and returns the occurrence number to log in one step, so each power-of-two occurrence is logged exactly once and with its own number.
A failed save, cache link or debug mesh write called exit(1) on a worker thread. That skipped the other workers and the final index save, and could interrupt another worker's checkpoint, which rewrites the index in place. A failure now stops the level like a stop request: active tasks finish their nodes, queued ones are skipped, and build_level returns the first error. build_levels saves the index before returning it, so the nodes committed before the failure are kept in the incomplete snapshot.
The octree store wrote its index and metadata in place, truncating them first, so a crash during a checkpoint could leave an empty or partial index in the incomplete snapshot. The raster index and the RF merger statistics already wrote a temporary file and renamed it, each with its own copy of the code, but without syncing it. WriteMode::Replace writes and fsyncs <path>.tmp, then renames it over path, so path holds the old or the new bytes even after a crash or power loss, and hard links to the old file keep the old bytes. All three now use it. The glTF writer, which opens its file itself, rejects Replace as unsupported.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds persistent raster storage. tools to import, merge, and inspect attributed raster snapshots are in another repo. existing SF and DAG applications are moved onto a shared storage foundation.