`--aggressive-cleanup` is supposed to delete the BUILD and INSTALLROOT directories right?

[akritkbehera]

I started an CI job using --aggressive-cleanup but it doesn't delete the directories after a package is built and node runs out of the storage leading to failure.

[pbuncic]

There are two mechanisms, for two situations:

bits clean [--aggressive-cleanup] [-n] is symlink-reachability GC of a single workDir/arch (clean.py). It keeps whatever a -latest symlink still points to and deletes the rest. Good for a one-shot reclaim; -n/--dry-run prints what it would delete first.

bits cleanup (the newer cleanup.py) is for a persistent workDir that survives across many CI jobs. It tracks every installed package with a sentinel file at workDir/.packages/// whose mtime is bumped each time the package is built or pulled in as a dependency (transitive touching, so a still-needed dep isn't evicted). It has two combinable modes plus a retention sweep:

--max-age N — evict packages whose sentinel mtime is older than N days (meant as a nightly cron).
--min-free G — when the workDir filesystem drops below G GiB free, evict least-recently-used packages (oldest sentinel first) until the threshold is met (meant as a pre-build hook).
--retain — a manifest-rooted sweep across all archs: keep the newest --keep-builds build manifests per arch plus certified-but-unpublished content, and evict anything already uploaded to the store and in the signed manifest and recorded as published to CVMFS.

In bits-console, the pipeline (bits-console/.gitlab/cvmfs-prepub-publish.yml) does not use --aggressive-cleanup; the runners keep a persistent workDir and rely on bits cleanup:

Pre-build hook (line 378): bits cleanup --min-free "${CACHE_MIN_FREE_GB:-50}" --disk-pressure-only || true — before each build, if the workDir FS has under 50 GiB free, evict LRU packages. || true so cleanup never fails the job.

In principle, bits-console can be custimized and run bits build pipeline for different experiments. I did not test it with CMS since you have a somewhat special setup that I do not fully understand. If you are interested, let me know.

Finally, I am about to merge significant changes to make re-use of previously deployed packages on CVMFS for subsequent builds. I'll tag current version so that you can go back to working version in case of problems.