[JanZachmann]

Summary

• write_partition compares the partition file with the image in 4 KiB blocks, only where one of them holds data, and writes only the blocks that differ, instead of running fallocate -d. It also writes a block that is data in the partition file but a hole in the image, e.g. a zero block of a file that e2cp or mcopy wrote into free space. fallocate, dd and sync are removed from the Docker image.
• The image is copied with sparse::copy_file, and a partition is read out of the image with the same sparse copy instead of dd conv=sparse: it finds the data with SEEK_DATA/SEEK_HOLE, keeps the holes and copies each data range in a loop. Decompressing an image keeps zero blocks as holes, reads a file made of several streams (e.g. from pbzip2), and fails on a truncated or corrupt stream. A compressed image is decompressed straight into the temp dir, without a copy first. Images named .gz and .bz2 are decompressed too; before, the output file had the name of the source, so creating it emptied the source. Compressing fails when the end of the stream cannot be written.
• omnect-cli writes the bmap of -b itself: it maps the ranges of the input bmap, or the whole image without one, plus every block the image file stores as data, with a sha256 per range. No zeros are written into the image, so it stays sparse, and bmaptool is no longer needed at runtime: -b now works in the Docker image, and bmap-tools is removed from the Debian dependencies and the Docker image. The bmap is copied next to the image after the image was written back.
• The input bmap is the file given with the new --input-bmap argument (requires -b), else <image>.bmap or the bmap of the uncompressed image next to it (e.g. x.wic.bmap for x.wic.xz, as the Yocto build creates it). It is only read with -b. The command fails if the input bmap is for an image of another size, if a range does not match its sha256 checksum, or if the bmap has no range checksums. The new --no-input-bmap (requires -b) skips it, and the bmap then maps the whole image. The input bmap of an uncompressed image is checked before the image is copied.
• The created bmap is named <output image>.bmap and written next to the output image, e.g. x.wic.bmap for x.wic and x.wic.xz.bmap for x.wic.xz, where bmaptool finds it. So with -p it no longer replaces the bmap of the input image, which stays unchanged.
• The output image is the input image, for a compressed image without the compression extension, plus the -p extension, as before. The new -o/--output-image writes it to another path, and the input image stays unchanged. With -p, the -o name must end in .xz, .bz2 or .gz, the extension that -p writes; without -p it must not have a compression extension. A command now fails instead of overwriting an existing file that is not the input image (a path given with -o is overwritten), replacing the bmap of an image that stays (e.g. x.wic.bmap for -i x.wic.xz without -p, also with --input-bmap or --no-input-bmap), or writing an image without -b while <output image>.bmap is next to it, or for a compressed output image the bmap of the uncompressed image, which bmaptool also uses. That bmap would no longer match. So for several commands on one image, use -b on each.
• The output image and the bmap are written to a new file in the same directory and renamed over the old one, so a failed write, e.g. on a full disk, no longer destroys an image that is changed in place. A symlink is followed, and the old file keeps its permissions. A symlink whose target does not exist yet is followed too. The directory needs write access and space for one more image, which is checked before the command runs, and the new file belongs to the user who runs omnect-cli. The image is replaced before its bmap: if the image step fails, the old image keeps its bmap, and if the bmap step fails, the old bmap is removed. An output image that is a mount point, e.g. a file mounted into a container, cannot be replaced by a rename, so its new content is staged in TMPDIR and copied into it; if that copy fails, the staged file is kept and the error names it. Ctrl-C, SIGTERM or SIGHUP removes every temp dir of a command before the process stops; a signal that is ignored, e.g. SIGHUP under nohup, is not handled.
• copy-from-image no longer writes the image back. It only reads, so an error can no longer truncate the source, and for x.wic.xz it no longer leaves x.wic next to it. It reads an uncompressed image directly, without a temp copy.
• The partition files are written to their own temp dir. Before, they were named <num>.img next to the image, so an image named e.g. 1.img was overwritten by its partition 1. The temp dirs follow TMPDIR.
• set-device-certificate and set-edge-ca-certificate write the new certificate and key to a temp dir instead of under fixed names next to the input image, so two runs on one base image no longer share them, a read-only image directory works, and no private key is left there.
• Partitions of a GPT with 4096-byte sectors are read at the right offset; before, the offsets were always counted in 512-byte sectors. A partition that does not fit into the image, e.g. from a corrupt GPT, is refused, so its offset cannot overflow.
• Known limitations, tracked as follow-up tasks: the created bmap maps every block that the image file stores as data, so for an image that is not sparse it maps the whole image, also with an input bmap. For a partition that does not start on a 4 KiB boundary, a block at the edge of each data run is written back, so the image is less sparse. In both cases the image content is correct. An uncompressed output image is copied twice, into TMPDIR and into the staged file.
• Breaking: --generate-bmap-file is renamed to --output-bmap, so the two options read as a pair. The short option -b stays. The name conflicts above fail where they overwrote files before. -p bzip2 writes <image>.bz2 instead of <image>.bzip2, and -p gzip writes <image>.gz instead of <image>.gzip; bmaptool reads neither of the old names. An input named .bzip2 or .gzip is still read, but with -p the result is written next to it as .bz2 or .gz, so a second run on the same input fails because that file exists. -o needs the short name.
• roxmltree is added to read the bmap, rustix for SEEK_DATA/SEEK_HOLE and statx, signal-hook to remove temp dirs on a stop signal (its signal-hook-registry was already in the tree through tokio), and libc for sigaction (already in the tree); libfs is removed. The test bmap testfiles/image.wic.bmap is now in git (*.bmap is in .gitignore), with checksums that match the test images. The version is bumped to 0.29.0, and ctutils to 0.4.5 for RUSTSEC-2026-0385.

Reason

A sparse image, e.g. one that xz decompressed into a file, or one after fallocate -d, has holes wherever the image is zero, and a bmap skips them. Some of these zeros must still be written: a bmap-based flash otherwise keeps what an earlier installation left there. An old ext4 journal can be replayed after a power cut and apply an old transaction to the new filesystem; old GPT entries break the entry array CRC; old FAT sectors and directory entries show up as used clusters or stale files. On an rpi4-64 image, mapping only the ext4 journals still drops 188 blocks of the Yocto bmap: GPT primary and backup entries, 69 blocks of boot, and 16–24 blocks on every ext4 partition.

The Yocto bmap maps all of these ranges, because the build image holds real data there. Keeping its ranges mapped covers every format without knowing about file systems. From a sparse image alone, omnect-cli cannot tell such zero blocks from free space, so without an input bmap it maps the whole image. bmaptool copy reads a mapped range with plain reads, so a hole in such a range is written as zeros: the zeros only have to be in the bmap, not allocated in the image file. This replaces the image-side approach discussed in omnect/meta-omnect#701.

With default-features = false, libfs made byte copies, so every image came back fully allocated, and the decompress wrote zero blocks as data. fallocate -d hid both. The sparse copy uses SEEK_DATA/SEEK_HOLE directly: libfs::next_sparse_segments seeks both files to the same offset, so it cannot read a partition out of the image, and libfs::copy_file copies each data segment with a single copy_file_range call, which moves at most 2 GiB, and ignores a short copy.

All omnect partitions have a fixed size, so builds of one machine have the same image size, and the bmap of another build (e.g. of an older download next to a new image) passes a size check. The sha256 of each range tells the builds apart; bmaptool copy runs the same check on flash.

omnect-cli derives every output name from the input name and finds the input bmap by name, so names can collide. omnect-cli runs in CI scripts, where nobody reads a warning, so each collision fails with an error that names the option to resolve it. A counter in the output name would not work: a script and the later bmaptool copy need a fixed name.

Writing only changed blocks also fixes a gap of the old dd conv=notrunc,sparse: a block that changed to all zeros was skipped and kept its old content; now it is written because it differs. In the same way, dd conv=sparse turned every zero block of a partition into a hole, so a zero block of a copied file that landed in free space stayed a hole in the image, and a bmap flash kept the old content there.

[JanZachmann]

On the nits in the review of 3da5a4d:

• README: 19bb6be describes the space TMPDIR needs: a copy of each partition the command reads or changes, and the uncompressed image for a command that changes an image or reads a compressed one.
• append_extension: 9126d55 uses std Path::with_added_extension in bmap_path and compressed_path.
• Bmap::read: 8f08735 checks the root element and refuses a missing version or a major version above 2, as bmaptool does.
• lib.rs:86, checking the checksums during the copy: not changed. f0ead21 checks the input bmap on the source before the copy, so a wrong bmap fails before a multi-GB copy. Checking during the copy would move that failure after it. Also, the copy uses copy_file_range, which does not read the data through user space. Hashing during the copy would need that.

[mlilien]

Thanks for the fixes. The three threads from the last round are resolved; I checked each against 6e9c9e1. Clippy (-D warnings) is clean, and unit and integration tests pass locally.

A few new points inline. One is outside the diff, so here:

src/lib.rs create_image_cert (around line 217): it writes the device certificate and private key under fixed names (device_cert_path.pem, device_key_path.key.pem, the edge_ca_* names) next to the input image and never deletes them. This is older than this PR, but -o makes "one base image, many device outputs" the documented workflow, so it matters more now:

• Two jobs running identity set-device-certificate -i base.wic.xz -o devA.wic and ... -o devB.wic in the same workspace at the same time: B overwrites the files while A is still decompressing, so devA.wic can get B's certificate and key.
• With a read-only input directory (-i /ro/base.wic.xz -o /out/dev.wic), the command fails at fs::write.
• A device private key is left next to the base image, even when check_conflicts rejects the run.

Writing the cert/key into a TempDir would fix all three. Fine as a follow-up if you prefer.

[mlilien]

bmap_path_of duplicates the public omnect_cli::file::bmap::bmap_path, and four tests (lines 964, 988, 1035, 1092) still build the path inline with format!("{}.bmap", ...). If the naming rule changes, the tests keep the old one. Using bmap_path everywhere avoids that.

[JanZachmann]

Done in f918204. bmap_path_of is gone, and all inline format!("{}.bmap") paths, plus one with_extension("bmap"), use bmap::bmap_path.

[mlilien]

Almost: tests/integration_tests.rs:1482 (check_bmap_generation_wic_xz_without_input_bmap) still uses image_path_wic_xz.with_extension("bmap") instead of bmap_path(<output image>).

[JanZachmann]

Fixed in aaa1c8a. check_bmap_generation_wic_xz_without_input_bmap now uses bmap_path of the output image image.wic.

Checked: cargo fmt --check, cargo clippy --all-targets -- -D warnings, cargo test (108 unit, 38 integration, 1 ignored), cargo audit (only RUSTSEC-2025-0141, bincode via mbrman, not from this change).

[JanZachmann]

On create_image_cert: fixed in this PR, in ddb60e9. The new certificate and key are written to a temp dir, so parallel runs no longer share them, a read-only input directory works, and no key is left next to the image. 47c05a5 also removes the dir on a stop signal. check_set_device_cert_est and check_set_edge_ca_cert check that no cert or key file is left next to the image.

[mlilien]

Thanks for the fixes. I resolved the previous threads except the bmap_path one (one leftover, see reply). New points inline.

[mlilien]

Signals::new installs the handler for SIGHUP, SIGINT and SIGTERM without checking whether a signal is set to SIG_IGN. signal-hook-registry skips the ignored previous action but still runs ours, which removes the dirs and exits via emulate_default_handler.

Example: nohup omnect-cli file copy-to-image -i x.wic.xz -p xz -b ... &, then the SSH session closes. Before this PR SIGHUP was ignored and the command finished. Now a hangup during the long xz compress deletes the staged image and the process exits with 129. Same for background jobs in non-interactive shells, which ignore SIGINT.

Fix: query the current disposition with sigaction(sig, NULL, &old) and skip signals that are SIG_IGN.

[JanZachmann]

Fixed in acf2cb3. Before it registers the handler, start_handler reads the current disposition of each stop signal with sigaction(sig, NULL, &old) and leaves out the ones set to SIG_IGN. is_ignored_reads_the_disposition covers the check. I ran it under nohup and sent SIGHUP during the -p xz compress: the command finished with exit 0 and wrote the image. Without nohup, SIGHUP still stops it with exit 129 and removes the staging dir. libc is new as a direct dependency; it was already in the tree.

[mlilien]

commit removes the existing bmap before the image is replaced. If image.rename() or copy_into_dest() then fails, or a stop signal arrives between the two steps, the old image stays and its bmap is gone.

Example: in-place -i x.wic -b where x.wic.bmap is the Yocto/portal input bmap. Rename fails with EPERM (sticky shared dir, x.wic owned by another user), the mount-point copy fails (ENOSPC/EIO), or Ctrl-C lands between remove_file and rename. x.wic.bmap is deleted, and the next -b run silently maps the whole image, so the zero-range information is lost for good.

Fix: replace the image first and then the bmap, or move the old bmap aside and restore it if the image step fails.

[JanZachmann]

Fixed in 13dfac7. commit now replaces the image first and then the bmap. If the image step fails, nothing is changed, so the old image keeps its bmap. If the bmap step fails, the old bmap is removed, because it no longer matches. One window is left: a signal or a crash between the two steps leaves the new image with the old bmap. Its range checksums no longer match, so bmaptool copy and the next -b run refuse it, and the error is not silent. commit_keeps_the_old_bmap_when_the_image_rename_fails covers the new case.

[mlilien]

Partition offsets are now computed in-process without overflow checks: entry.starting_lba * scale here, and partition_info.start * SECTOR_SIZE / image_offset + len in read_partition and write_partition. Before, dd did this arithmetic. There is no [profile] section, so release builds wrap instead of panicking.

With a corrupt or crafted GPT whose CRCs are valid (gptman doesn't check entry LBAs against last_usable_lba) and starting_lba >= 2^55, the offset wraps, the src_offset + len <= src_len check passes, and write_partition writes at the wrapped offset, e.g. over the GPT. Please use checked_mul/checked_add and fail.

[JanZachmann]

Fixed in 95c0918. get_partition_data computes the sectors with checked_mul and fails unless (start + count) * SECTOR_SIZE fits in u64 and ends inside the image file. This covers GPT and MBR, and the later byte offsets in read_partition and write_partition stay below that end. get_partition_data_rejects_an_overflowing_offset writes a GPT entry at LBA 2^55 with valid CRCs (through GPTHeader::write_into, because GPT::write_into refuses it), and get_partition_data_rejects_a_partition_after_the_image_end covers a cut-off image.

[mlilien]

resolve() uses path.exists(), which is false for a dangling symlink. The link path itself then becomes the destination, and Staged::write renames the new file over the link. That contradicts "A symlink is followed". The !output_image.exists() check in check_conflicts also lets a dangling derived output through.

Example: -o out.wic where out.wic -> /data/out.wic doesn't exist yet. The image is staged next to the link and replaces it with a regular file; /data/out.wic is never created. Before this PR, File::create followed the link and created the target.

[JanZachmann]

Fixed in 626c64f. resolve follows a symlink whose target does not exist yet, also through a chain, with a relative target taken relative to the link, up to the Linux limit of 40 links. In check_conflicts, a dangling derived output now counts as existing. staged_commit_creates_the_target_of_a_dangling_symlink, check_conflicts_rejects_a_dangling_derived_output and resolve_stops_at_a_symlink_loop cover it.

[mlilien]

The temp image keeps the input file name in a dir it shares with configure_hostname, which writes hostname and hosts next to the image via get_file_path. The PR fixed the <num>.img collision but not this one. identity set-config -i hosts (or -i hostname) overwrites the temp image, and the next partition read fails with "image is neither valid GPT nor MBR". Older than this PR and only a confusing error, so fine as a follow-up.

[JanZachmann]

Tracked as a follow-up.

[mlilien]

matches_extension still accepts .gzip and .bzip2. So -i x.wic.gzip -p gzip keeps x.wic.gzip, and -o y.wic.bzip2 -p bzip2 is accepted, although the -p output was renamed to .gz/.bz2 because bmaptool reads neither old name. A pipeline with an existing x.wic.gzip running -p gzip -b gets x.wic.gzip plus x.wic.gzip.bmap; bmaptool copy won't decompress it and the flash fails, with no warning. I saw 961f95b makes this deliberate, but maybe at least warn?

[JanZachmann]

Changed in dd1f171 and 32d03c9: with -p, -o must end in .gz or .bz2, and the error suggests the short name. An input named x.wic.gzip with -p gzip is written as x.wic.gz. The long names are still read as input. The README and the breaking notes in the description say so.

[mlilien]

For uncompressed output, every data block is copied twice: into TMPDIR (temp_image) and again into the staged file (Staged::write → sparse::copy_file). An in-place -i x.wic on an 8 GiB, mostly-data image needs room for two image copies (three for a mount point). Running the command directly on the staged file and then committing would save one copy and the TMPDIR space. Fine as a follow-up.

[JanZachmann]

Tracked as a follow-up.

[mlilien]

check_conflicts checks that the output file is writable, but not its directory, which needs write access for .omnect-cli-* and the rename. For a mount-point output it doesn't check TMPDIR either. So -i /ro/x.wic.xz in a read-only dir decompresses the full image and runs the command before tempdir_in(/ro) fails. An access(dir, W_OK) check here would fail up front.

[JanZachmann]

Fixed in 7447910. check_conflicts checks with access(W_OK) that the directory where the staged file goes is writable: the directory of the output image and of its bmap, or TMPDIR for a mount point. check_conflicts_rejects_a_read_only_output_dir covers it; it accepts the result when the tests run as root.

[mlilien]

"stays in TMPDIR" isn't right when the mount point is only found by the EBUSY/EXDEV rename fallback (kernel < 5.8, no STATX_ATTR_MOUNT_ROOT). Then the staged file is in .omnect-cli-* next to the image. The log names the right path, but the README sends people to the wrong place.

[JanZachmann]

Fixed in 59a6cf9: "If this copy fails or is stopped, the new image stays, and the log names it."

[mlilien]

Leftover from the bmap_path_of thread: this still uses image_path_wic_xz.with_extension("bmap") instead of bmap_path(<output image>). It only works because stripping .xz and adding .bmap happens to equal bmap_path(image.wic).

[JanZachmann]

Fixed in aaa1c8a.