Feat/images max size

CHANGELOG.md

@@ -1,5 +1,12 @@
 # Changelog
 
+## v1.5.0
+
+- Added `images --max-size <size>`, the mirror of `--min-size`: images that would free more are evicted only once nothing else is left. Re-pulling a large image is what keeps a launch waiting, while a small one is back in about a second; and deleting costs per file and stalls creates for the length of one removal, so several small removals are gentler than one large one. Size is what removal would free, the bytes in layers no other image shares, as of the start of the pass. Images it reports carry `, over max-size`.
+- Added `images --stale-after <duration>`, which needs `--lru`: an image unused for longer is stale, held back by neither `--min-size` nor `--max-size`, and goes by age alone. Without it, `--max-size` keeps a large image nothing uses any more, such as a replaced build, until nothing smaller is left. Off by default, so `--min-size` behaves as before unless it is set. Like `--lru`, it keeps its last value when repeated. A record that shows no use at all within the duration is taken to be no longer kept: that pass treats nothing as stale and warns, rather than letting every image go by age alone.
+- The image use record now starts with a `# began <unix seconds>` line, which releases before 1.5.0 skip. An image stamped 0, there when the record began, ages from that time for `--stale-after`, so a new or deleted record does not make every image stale at once. A record from an earlier release is taken to begin when this release first saves it.
+- `images` refuses a `--target` at or above `--threshold`, or a `--max-size` below `--min-size`, at startup. Before, each pass logged the error and the process exited successfully, evicting nothing.
+
 ## v1.4.0
 
 - Added `images --lru <path>`, which evicts unused images least recently used first rather than largest first. Docker records no last-used time for an image (the Engine API has none, and a pull does not set `LastTagTime`), so the order comes from a record kept by the container sweep: `containers --record-image-use <path>` stamps the image of every container the sweep lists as in use now. It reads the container list the sweep fetches anyway, so it adds no call to the daemon; the one exception is the run that creates the record, which lists images once and stamps every image already on the host as older than anything seen since. An image the record has never seen counts as used just now, since it most likely arrived for a launch whose container does not exist yet. Ties are broken by largest reclaimable size, as the default order does.

Cargo.toml

@@ -5,7 +5,7 @@ edition = "2024"
 license = "MIT OR Apache-2.0"
 name = "docker-reaper"
 repository = "https://github.com/picoCTF/docker-reaper"
-version = "1.4.0"
+version = "1.5.0"
 
 # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
 

README.md

@@ -86,7 +86,9 @@ Key flags for `docker-reaper images`:
 - `--disk-path <path>`: Filesystem path to measure disk usage on. Defaults to the Docker daemon's root directory (`docker_root_dir`). Note: when targeting a remote daemon via `DOCKER_HOST`, `--disk-path` must be explicitly specified because disk measurement operates on local storage.
 - `-f, --filter <name=value>`: Only reap images matching Docker Engine-supported filters (can be specified multiple times).
 - `--lru <path>`: Evict least recently used first, by the record at this path (see below).
-- `--min-size <size>`: Evict images that would free less than this (e.g. `256MiB`, `1G`; binary units) only once no larger candidate is left. Deleting an image holds dockerd's image and layer store locks until its files are gone, stalling every container create and image pull on the host meanwhile, so a removal should free something worth that. Default `0`, which leaves the order alone.
+- `--min-size <size>`: Evict images that would free less than this (e.g. `256MiB`, `1G`; binary units) only once no larger candidate is left, so a pass reaches its target in fewer removals. Fewer is not necessarily gentler, since a removal costs time per file rather than per byte (see [Keeping large images](#keeping-large-images)). Default `0`, which leaves the order alone.
+- `--max-size <size>`: Evict images that would free more than this only once nothing else is left; the mirror of `--min-size` (see below). Must be at least `--min-size`.
+- `--stale-after <duration>`: With `--lru`, an image unused for longer than this is stale: neither `--min-size` nor `--max-size` holds it back, so it goes by age alone. Off by default.
 
 Images are selected and evicted largest-unique-size first (reclaimable bytes not shared with other images) until disk usage drops below the target percentage. Non-forced removals skip images that gain containers mid-run.
 
@@ -116,19 +118,60 @@ $ docker-reaper images --threshold 80 --target 70 --lru /var/lib/docker-reaper/i
   and that pass orders by size alone; a line in it that cannot be parsed costs only that
   line. With `--min-size`, size comes first: every image that would free at least that much
   goes before any that would not, so a large image used a minute ago, or pulled just now,
-  goes before a small one unused for weeks.
+  goes before a small one unused for weeks, unless `--stale-after` makes that one stale.
 
-Repeating `--record-image-use`, `--lru` or `--data-root` keeps the last value, so a
-deployment can append them to a command that may already carry them.
+Repeating `--record-image-use`, `--lru`, `--stale-after` or `--data-root` keeps the last
+value, so a deployment can append them to a command that may already carry them.
 
 Sampling is as frequent as the container sweep runs, so a container that comes and goes
 between two runs is not seen. The record is a text file, `<unix seconds> <image id>` per
-line; deleting it starts it over.
+line, after a first line `# began <unix seconds>` saying when the record started; releases
+before 1.5.0 skip that line. Deleting the record starts it over.
 
 One case it cannot see: an image removed by something other than this sweep, then pulled
 again before the next eviction pass, keeps the stamp it had. Noticing the removal in between
 would take an image list every run. After removing images by hand, delete the record.
 
+#### Keeping large images
+
+Pulling an image back costs time in proportion to its size, since dockerd unpacks each
+layer on one core, so a launch that needs a multi-GB image again waits tens of seconds
+while a small one is back in about a second. Deleting costs per file rather than per byte
+or per image, and stalls creates for one removal at a time, so several small removals are
+gentler than one large one.
+
+`--max-size <size>` acts on both: images that would free more than the size go only once
+every other candidate is gone. Size, here and for `--min-size`, is what removing the image
+frees: the bytes in layers no other image shares (the UNIQUE SIZE of `docker system df
+-v`), which is also what a re-pull would have to fetch and unpack.
+
+On its own it also keeps a large image that nothing uses any more, such as a replaced build
+or a retired challenge, until nothing smaller is left. With `--lru`, `--stale-after
+<duration>` ages those out: an image unused for longer goes by age alone, whatever its
+size. The order is then:
+
+1. stale images, oldest first;
+2. the rest, oldest first;
+3. images under `--min-size`, oldest first;
+4. images over `--max-size`, oldest first.
+
+Without `--lru` there is no age, and each group goes largest first. An image the record
+has never seen counts as just used, so a large pull for a launch that has no container yet
+is never stale. One stamped 0, there when the record began, ages from when it began, so a
+new or deleted record makes nothing stale for `--stale-after`; a record from before 1.5.0
+is taken to begin when this release first saves it. A record that shows no use at all
+within `--stale-after` is most likely no longer being kept, so that pass treats nothing as
+stale and says so, rather than letting every image go by age alone.
+
+Sizes are those at the start of a pass. Removing one image can leave another as the only
+one using layers they shared, which then would free more, and cost more to pull back, than
+when the pass began; it keeps its place until the next pass. Listing images again after
+every removal would catch it, at the cost of an image list per removal.
+
+```bash
+$ docker-reaper images --threshold 89 --target 84 --lru /var/lib/docker-reaper/image-use --max-size 1G --stale-after 72h
+```
+
 ### Orphaned containerd shim sweep
 
 Each running container has a `containerd-shim-runc-v2` process. If a container is removed

src/main.rs

@@ -13,7 +13,8 @@ use bollard::Docker;
 use clap::{Args, Parser, Subcommand};
 use reaper::{
     Filter, ReapContainersConfig, ReapImagesConfig, ReapNetworksConfig, ReapShimsConfig,
-    ReapVolumesConfig, reap_containers, reap_images, reap_networks, reap_shims, reap_volumes,
+    ReapVolumesConfig, check_image_bounds, reap_containers, reap_images, reap_networks, reap_shims,
+    reap_volumes,
 };
 use std::path::PathBuf;
 use tokio::time::{Duration, sleep};
@@ -118,6 +119,7 @@ struct VolumesArgs {
 }
 
 #[derive(Debug, Args)]
+#[command(after_help = "Note: <duration> values accept Go-style duration strings (e.g. 72h)")]
 struct ImagesArgs {
     /// Only reap when the measured filesystem is at least this full (percent).
     #[arg(long, value_name = "percent", default_value_t = 80, value_parser = parse_percent)]
@@ -147,6 +149,20 @@ struct ImagesArgs {
     /// candidate is left. Units are binary: K, M and G mean KiB, MiB and GiB.
     #[arg(long, value_name = "size", value_parser = parse_size, default_value = "0")]
     min_size: u64,
+    /// Evict images that would free more than this (e.g. 1G) only once nothing else is left:
+    /// they are the slow ones to pull back for a launch. Units as for --min-size.
+    #[arg(long, value_name = "size", value_parser = parse_size)]
+    max_size: Option<u64>,
+    /// With --lru, an image unused for longer than this is stale: --min-size and --max-size
+    /// no longer hold it back, so it goes by age alone.
+    #[arg(
+        long,
+        value_name = "duration",
+        value_parser = parse_duration,
+        requires = "lru",
+        overrides_with = "stale_after"
+    )]
+    stale_after: Option<Duration>,
 }
 
 #[derive(Debug, Args)]
@@ -252,6 +268,12 @@ async fn main() -> Result<(), anyhow::Error> {
         );
     }
 
+    // A pass only logs its errors, so bounds no pass can use would leave a unit evicting
+    // nothing while it reports success.
+    if let Commands::Images(ref args) = global_args.command {
+        check_image_bounds(args.threshold, args.target, args.min_size, args.max_size)?;
+    }
+
     // Orphaned shims are found by reading this machine's process table, which says
     // nothing about a daemon running elsewhere.
     // DOCKER_CERT_PATH matters as much as DOCKER_HOST: the branch below prefers it and
@@ -331,6 +353,8 @@ async fn main() -> Result<(), anyhow::Error> {
                     filters: &args.filters,
                     lru: args.lru.clone(),
                     min_size: args.min_size,
+                    max_size: args.max_size,
+                    stale_after: args.stale_after,
                 };
                 reap_images(&docker, &config).await
             }