generate CVMFS filebundle prefetch files - #111
Merged
Merged
Conversation
New bits_helpers/preload_bundle.py turns a package's .bits-preload/*.paths sidecars into spec-conformant .cvmfsbundle-<trigger> JSON: resolves each opened file to its owning package's CVMFS path, lists repo-root-absolute deps (trigger excluded), removes the sidecar dir. Pure + unit-tested; not yet wired into publish.
Regenerate .cvmfsbundle specs from a trigger straced on an already-deployed CVMFS tree: opens are already /cvmfs-absolute, so repo-root is a mount strip (no relocation, no cross-package map). Pure + unit-tested; drives the coming 'bits preload' tool.
… parse, tar) Pure, unit-tested steps for the post-publish tool: detect/extract a recipe's Preload() triggers by parsing cvmfs_preload calls (no recipe sourcing), locate a package under a deployed CVMFS path, parse strace open/openat output, and assemble staged .cvmfsbundle-* files into one tar. Orchestration + wiring next.
parse_strace_opens keeps only successful opens (ret >= 0), dropping the loader's ENOENT probes in tls/, glibc-hwcaps/ and arch subdirs; to_repo_absolute normpath's the path so the BITS_ARCH_PREFIX="." '/./' does not leak into deps.
Add 'preload' to _HASH_EXCLUDED_META_KEYS so a recipe can carry its CVMFS filebundle test list (consumed post-publish by 'bits preload') without editing it forcing a rebuild.
…ry helpers Read tests from a recipe preload: section (replaces the Preload() bash parser); load config/preload.yaml (arch scalar/list or discover; packages bare/mapping with test overrides); resolve config-over-recipe; bundle_exists skip check; discover_archs / package_versions / package_dir via cvmfs_inspect.
Config-driven sweep over arch x package x version x test: strace each recipe test (native or --docker), build .cvmfsbundle-* specs, one tar. Idempotent (skip existing unless --update), arch discovery, injectable tracer for tests. Wire 'bits preload' in the wrapper. --publish deferred to the prepub --tar path.
…repub publish_overlay_tars stages each tar at path='' (files land at their repo- relative locations, e.g. bits preload .cvmfsbundle-*) through the existing stage_tar/submit_staged path; distinct job_id_base per tar; caller's tar kept. 'bits preload' prints the ready-to-run 'bits cvmfs-publish --tar' command.
…ndependent) Add a cvmfs: key to preload.yaml and make --cvmfs optional (CLI overrides config, error if neither). With per-package tests: in the config no recipe is consulted, so 'bits preload --config preload.yaml' works with no bits recipes.
The wrapper pre-parses a global --config (bits.rc) that swallowed 'bits preload --config <yaml>', leaving the value as a stray positional. Handle 'preload' from raw ARGV before that loop so its own --config reaches preload_cmd.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
bits preload, a post-publish tool that generates CVMFS filebundle prefetchfiles (
.cvmfsbundle-<exe>, per the CVMFS spec) for already-deployed packages.For each package test it straces the deployed binary on the tree, turns the
opened
/cvmfsfiles into a spec-conformant bundle next to the trigger, and tarsthem. Opening the trigger then prefetches its whole dependency closure in one
round trip instead of many.
It runs after publish, over the deployed tree — so it needs no rebuild and
works on software published long ago. Because the binaries are already at their
final
/cvmfspaths, dependency paths are exact (a mount-strip), with norelocation and no cross-package resolution.
What's included
bits preload— config-driven sweep over arch × package × version × test.Idempotent (skips a version whose
.cvmfsbundle-<exe>exists,--updatetoregenerate), discovers arches/versions from the tree (reusing
cvmfs_inspect),traces natively or in a container (
--docker).preload:section (hash-excluded — seebelow), or a fully self-contained
preload.yaml(cvmfs+arch+packagestests) for users who don't use bits recipes.preloadis hash-excluded (_HASH_EXCLUDED_META_KEYS): recipes can carrytheir tests with no rebuild.
bits cvmfs-publish --tar FILE…— publishes the bundle tar(s) as aroot-overlay through the existing prepub
stage_tar/submit_stagedpath.Usage