From c038d064126bf8bd3c0c779761366d5eeb0d9927 Mon Sep 17 00:00:00 2001 From: Roberto D'Auria Date: Mon, 29 Jun 2026 16:51:21 +0200 Subject: [PATCH 01/25] Replace Docker Compose with native Debian package + hardened systemd Convert the autonode deployment from docker-compose to an mlab-node Debian package whose components run as hardened systemd services (systemd sandboxing instead of Docker isolation). No Docker at build or run time. - .build/fetch-binaries.sh: extract component binaries from the pinned M-Lab images via skopeo (no Docker daemon); fail the build on musl-linked binaries. - systemd/: mlab-node.target plus per-component services and oneshots (schema generation, uuid.prefix, BBR, metadata). Atomic lifecycle: every component BindsTo register-node, so if registration goes down the whole stack goes down; uniform Restart=always. - Per-service sandboxing (ProtectSystem=strict, dropped capabilities, RestrictAddressFamilies, etc.); CAP_NET_BIND_SERVICE for ndt-server, CAP_NET_RAW for scamper, DynamicUser for node-exporter. - debian/: native debhelper packaging (control, rules, install, sysusers, tmpfiles, postinst, conffiles). - config/: /etc/mlab env templates, verify.pub, modules-load.d + sysctl.d BBR. - Remove docker-compose.yml and env; rewrite README for the package workflow. Filesystem moves to FHS: binaries /usr/lib/mlab, config /etc/mlab, state /var/lib/mlab, sockets/metadata /run/mlab. --- .build/fetch-binaries.sh | 143 +++++++ .gitignore | 18 +- README.md | 99 ++++- config/api-key.env | 3 + config/locate/verify.pub | 1 + config/mlab-node.env | 44 +++ config/modules-load.d/mlab-node.conf | 3 + config/sysctl.d/90-mlab-node.conf | 4 + debian/changelog | 8 + debian/control | 31 ++ debian/copyright | 32 ++ debian/mlab-node.install | 47 +++ debian/mlab-node.postinst | 15 + debian/mlab-node.sysusers | 4 + debian/mlab-node.tmpfiles | 15 + debian/rules | 23 ++ debian/source/format | 1 + docker-compose.yml | 348 ------------------ env | 93 ----- html/index.html | 6 + scripts/gen-uuid-prefix | 9 + scripts/wait-ready | 15 + scripts/wait-socket | 13 + scripts/write-metadata | 10 + systemd/mlab-node-bbr.service | 13 + systemd/mlab-node-heartbeat.service | 54 +++ systemd/mlab-node-jostler.service | 66 ++++ systemd/mlab-node-metadata.service | 25 ++ systemd/mlab-node-ndt-server.service | 73 ++++ systemd/mlab-node-node-exporter.service | 52 +++ systemd/mlab-node-register-ready.service | 25 ++ systemd/mlab-node-register.service | 59 +++ systemd/mlab-node-schemas-annotation2.service | 21 ++ systemd/mlab-node-schemas-ndt7.service | 22 ++ systemd/mlab-node-schemas-traceroute.service | 23 ++ systemd/mlab-node-traceroute-caller.service | 65 ++++ systemd/mlab-node-uuid-annotator.service | 58 +++ systemd/mlab-node-uuidprefix.service | 22 ++ systemd/mlab-node.target | 16 + 39 files changed, 1133 insertions(+), 446 deletions(-) create mode 100755 .build/fetch-binaries.sh create mode 100644 config/api-key.env create mode 100644 config/locate/verify.pub create mode 100644 config/mlab-node.env create mode 100644 config/modules-load.d/mlab-node.conf create mode 100644 config/sysctl.d/90-mlab-node.conf create mode 100644 debian/changelog create mode 100644 debian/control create mode 100644 debian/copyright create mode 100644 debian/mlab-node.install create mode 100755 debian/mlab-node.postinst create mode 100644 debian/mlab-node.sysusers create mode 100644 debian/mlab-node.tmpfiles create mode 100755 debian/rules create mode 100644 debian/source/format delete mode 100644 docker-compose.yml delete mode 100644 env create mode 100644 html/index.html create mode 100755 scripts/gen-uuid-prefix create mode 100755 scripts/wait-ready create mode 100755 scripts/wait-socket create mode 100755 scripts/write-metadata create mode 100644 systemd/mlab-node-bbr.service create mode 100644 systemd/mlab-node-heartbeat.service create mode 100644 systemd/mlab-node-jostler.service create mode 100644 systemd/mlab-node-metadata.service create mode 100644 systemd/mlab-node-ndt-server.service create mode 100644 systemd/mlab-node-node-exporter.service create mode 100644 systemd/mlab-node-register-ready.service create mode 100644 systemd/mlab-node-register.service create mode 100644 systemd/mlab-node-schemas-annotation2.service create mode 100644 systemd/mlab-node-schemas-ndt7.service create mode 100644 systemd/mlab-node-schemas-traceroute.service create mode 100644 systemd/mlab-node-traceroute-caller.service create mode 100644 systemd/mlab-node-uuid-annotator.service create mode 100644 systemd/mlab-node-uuidprefix.service create mode 100644 systemd/mlab-node.target diff --git a/.build/fetch-binaries.sh b/.build/fetch-binaries.sh new file mode 100755 index 0000000..36f518d --- /dev/null +++ b/.build/fetch-binaries.sh @@ -0,0 +1,143 @@ +#!/bin/bash +# +# fetch-binaries.sh extracts the component binaries from the pinned M-Lab +# container images and stages them under ./binaries/ so they can be shipped in +# the mlab-node Debian package. It uses skopeo to pull the image filesystems +# without a running Docker daemon: no Docker is required to build the package, +# and none is required at runtime. +# +# Required host tools (build-time only): skopeo, jq, tar, file. +# +# The binaries are version-pinned to the same image tags used by the historical +# docker-compose deployment, so the package ships the exact binaries M-Lab has +# tested. Each extracted ELF is checked: a statically linked or glibc-linked +# binary is fine on Debian; a musl-linked (Alpine) binary is NOT and aborts the +# build, because it would fail to run on the target. +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)" +OUT_DIR="${REPO_DIR}/binaries" + +# Image versions. These mirror the tags pinned in the original docker-compose.yml. +# NOTE: the annotation2 schema generator intentionally comes from an older +# uuid-annotator image (v0.5.8) than the running annotator (v0.5.10), matching +# the compose configuration. +IMG_NDT_SERVER="measurementlab/ndt-server:v0.25.2" +IMG_HEARTBEAT="measurementlab/heartbeat:v0.19.1" +IMG_UUID_ANNOTATOR="measurementlab/uuid-annotator:v0.5.10" +IMG_UUID_ANNOTATOR_SCHEMA="measurementlab/uuid-annotator:v0.5.8" +IMG_JOSTLER="measurementlab/jostler:v1.1.4" +IMG_TRACEROUTE="measurementlab/traceroute-caller:v0.12.0" +IMG_REGISTER="measurementlab/autojoin-register:v0.2.13" +IMG_NODE_EXPORTER="quay.io/prometheus/node-exporter:v1.9.0" + +# extraction requests: "image|search-names|dest-name" +# search-names is a comma-separated list of candidate basenames to locate in the +# image rootfs (first match wins), to tolerate differing install paths. +REQUESTS=( + "${IMG_NDT_SERVER}|ndt-server|ndt-server" + "${IMG_NDT_SERVER}|generate-schemas|generate-schemas-ndt7" + "${IMG_HEARTBEAT}|heartbeat|heartbeat" + "${IMG_UUID_ANNOTATOR}|uuid-annotator|uuid-annotator" + "${IMG_UUID_ANNOTATOR_SCHEMA}|generate-schemas|generate-schemas-annotation2" + "${IMG_JOSTLER}|jostler|jostler" + "${IMG_TRACEROUTE}|traceroute-caller|traceroute-caller" + "${IMG_TRACEROUTE}|generate-schemas|generate-schemas-traceroute" + "${IMG_TRACEROUTE}|scamper|scamper" + "${IMG_REGISTER}|register,autojoin-register|autojoin-register" + "${IMG_NODE_EXPORTER}|node_exporter,node-exporter|node-exporter" +) + +for tool in skopeo jq tar file; do + command -v "${tool}" >/dev/null 2>&1 || { + echo "ERROR: required build tool '${tool}' not found in PATH" >&2 + exit 1 + } +done + +WORK_DIR="$(mktemp -d)" +trap 'rm -rf "${WORK_DIR}"' EXIT + +# unpack_image IMAGE DEST_ROOTFS +# Copies the image to a local OCI/dir layout with skopeo and extracts every +# layer (in manifest order) into DEST_ROOTFS to reconstruct the image rootfs. +declare -A UNPACKED=() +unpack_image() { + local image="$1" rootfs="$2" + local imgdir="${WORK_DIR}/img/$(echo "${image}" | tr '/:' '__')" + + if [ -n "${UNPACKED[${image}]:-}" ]; then + return 0 + fi + + echo ">> pulling ${image}" + mkdir -p "${imgdir}" "${rootfs}" + # Force a single linux/amd64 instance so the dir: manifest is an image + # manifest (with .layers) even when the source is a multi-arch index. + skopeo --override-os linux --override-arch amd64 \ + copy --quiet "docker://${image}" "dir:${imgdir}" + + # Layer blobs are listed (in order) in the manifest; each is a gzipped tar. + local layer + while read -r layer; do + local blob="${imgdir}/${layer#sha256:}" + [ -f "${blob}" ] || { echo "ERROR: missing layer blob ${blob}" >&2; exit 1; } + # Extract over the rootfs; ignore whiteout/permission quirks (we only need + # to read out a handful of executables afterwards). + tar -xf "${blob}" -C "${rootfs}" 2>/dev/null || true + done < <(jq -r '.layers[].digest' "${imgdir}/manifest.json") + + UNPACKED[${image}]=1 +} + +# check_binary PATH — abort if the ELF is musl-linked (won't run on Debian). +check_binary() { + local bin="$1" info + info="$(file -L "${bin}")" + case "${info}" in + *"statically linked"*) ;; # portable, OK + *"dynamically linked"*) + if echo "${info}" | grep -q "ld-musl"; then + echo "ERROR: ${bin} is musl-linked (Alpine); it will not run on Debian." >&2 + echo " Build this component from source or use a Debian package instead." >&2 + exit 1 + fi + ;; # glibc dynamic, OK (dh_shlibdeps handles deps) + *"ELF"*) ;; + *) + echo "ERROR: ${bin} does not look like an executable: ${info}" >&2 + exit 1 + ;; + esac + echo " ${bin##*/}: ${info#*: }" +} + +rm -rf "${OUT_DIR}" +mkdir -p "${OUT_DIR}" + +for req in "${REQUESTS[@]}"; do + IFS='|' read -r image names dest <<<"${req}" + rootfs="${WORK_DIR}/rootfs/$(echo "${image}" | tr '/:' '__')" + unpack_image "${image}" "${rootfs}" + + found="" + IFS=',' read -ra candidates <<<"${names}" + for name in "${candidates[@]}"; do + # Prefer an executable regular file; search the whole rootfs. + found="$(find "${rootfs}" -type f -name "${name}" -perm -u+x 2>/dev/null | head -n1 || true)" + [ -z "${found}" ] && found="$(find "${rootfs}" -type f -name "${name}" 2>/dev/null | head -n1 || true)" + [ -n "${found}" ] && break + done + + if [ -z "${found}" ]; then + echo "ERROR: could not find any of [${names}] in ${image}" >&2 + exit 1 + fi + + install -D -m 0755 "${found}" "${OUT_DIR}/${dest}" + check_binary "${OUT_DIR}/${dest}" +done + +echo +echo "Staged $(ls -1 "${OUT_DIR}" | wc -l | tr -d ' ') binaries in ${OUT_DIR}" diff --git a/.gitignore b/.gitignore index 2eea525..9408389 100644 --- a/.gitignore +++ b/.gitignore @@ -1 +1,17 @@ -.env \ No newline at end of file +.env + +# Binaries extracted from container images at build time (see +# .build/fetch-binaries.sh). Not checked in. +/binaries/ + +# Debian build artifacts. +/debian/mlab-node/ +/debian/.debhelper/ +/debian/files +/debian/*.debhelper +/debian/*.debhelper.log +/debian/*.substvars +/debian/debhelper-build-stamp +*.deb +*.buildinfo +*.changes \ No newline at end of file diff --git a/README.md b/README.md index c343eaf..1e24fc1 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,96 @@ -# autonode +# autonode (mlab-node) -Documentation on how to set up a host-managed M-Lab server is located in [the -Wiki of this -repository](https://github.com/m-lab/autonode/wiki/Host%E2%80%90managed-Deployments). +Native Debian packaging of the M-Lab "autonode" Network Diagnostic Tool (NDT) +measurement stack. The components run as hardened **systemd** services, confined +with systemd's own sandboxing directives — there is no Docker at build or run +time. + +This replaces the previous Docker Compose deployment. Documentation on +host-managed M-Lab servers is in the +[Wiki](https://github.com/m-lab/autonode/wiki/Host%E2%80%90managed-Deployments). + +## What's in the package + +The `mlab-node` package installs and runs, as systemd services: + +| Service | Unit | +|---|---| +| Node registration (Autojoin API) | `mlab-node-register.service` | +| NDT7 measurement server | `mlab-node-ndt-server.service` | +| Heartbeat (Locate API) | `mlab-node-heartbeat.service` | +| UUID annotator | `mlab-node-uuid-annotator.service` | +| Jostler (GCS archival) | `mlab-node-jostler.service` | +| Traceroute caller (scamper) | `mlab-node-traceroute-caller.service` | +| Prometheus node exporter | `mlab-node-node-exporter.service` | + +plus one-shot units for schema generation, the UUID prefix, BBR, and external-IP +metadata. The whole stack is controlled atomically through `mlab-node.target`; +`register-node` is the keystone — if it goes down, the entire stack is brought +down with it. + +The component binaries are extracted from M-Lab's pinned container images at +build time (see `.build/fetch-binaries.sh`). + +## File layout + +| Path | Contents | +|---|---| +| `/usr/lib/mlab/` | component binaries and helper scripts | +| `/etc/mlab/mlab-node.env` | configuration (non-secret) | +| `/etc/mlab/api-key.env` | M-Lab API key (`0640 root:mlab-node`) | +| `/etc/mlab/locate/verify.pub` | Locate token verification key | +| `/var/lib/mlab/node/` | registration data + service-account credential | +| `/var/lib/mlab/schemas/` | generated JSON schemas + uuid.prefix | +| `/var/lib/mlab/resultsdir/` | measurement results awaiting upload | +| `/var/lib/mlab/autocert/` | Let's Encrypt certificate cache | +| `/run/mlab/sockets/` | tcpinfo / ipservice unix sockets | +| `/usr/share/mlab/html/` | static content served by ndt-server | + +## Building the package + +Requires `dpkg-buildpackage`, debhelper, and (for binary extraction) +`skopeo`, `jq`, and `file`: + +```sh +dpkg-buildpackage -us -uc -b +``` + +The build pulls the pinned component binaries from the M-Lab images via skopeo +(no Docker daemon) and stages them under `binaries/`. The build fails if any +extracted binary is musl-linked (it would not run on Debian). + +## Installing and configuring + +```sh +sudo apt install ./mlab-node_1.0.0_amd64.deb + +# Configure the node, then add the API key. +sudoedit /etc/mlab/mlab-node.env +sudoedit /etc/mlab/api-key.env + +# Start the whole stack (it is also enabled to start on boot). +sudo systemctl start mlab-node.target +``` + +Required settings in `/etc/mlab/mlab-node.env`: `ORGANIZATION`, `IATA`, +`INTERFACE_NAME`, `INTERFACE_MAXRATE`, `UPLINK`, `IPV4`, `TYPE` (and `IPV6` if +the node is reachable over IPv6). `API_KEY` goes in `/etc/mlab/api-key.env`. + +## Operating + +```sh +systemctl status mlab-node.target +systemctl list-dependencies mlab-node.target +journalctl -u mlab-node-ndt-server -f + +# Stop / restart the entire stack. +sudo systemctl stop mlab-node.target +sudo systemctl restart mlab-node.target +``` + +## Prometheus metrics + +Exposed on the host (all `network_mode: host` equivalents are now native): +9990 ndt-server, 9991 jostler, 9992 uuid-annotator, 9993 heartbeat, +9994 traceroute-caller, 9995 node-exporter. ndt-server serves measurements on +80 (cleartext) and 443 (TLS). diff --git a/config/api-key.env b/config/api-key.env new file mode 100644 index 0000000..46f7d6b --- /dev/null +++ b/config/api-key.env @@ -0,0 +1,3 @@ +# M-Lab API key for this node. Keep this file secret. +# Installed as 0640 root:mlab-node. Set the value provided by M-Lab below. +API_KEY= diff --git a/config/locate/verify.pub b/config/locate/verify.pub new file mode 100644 index 0000000..a2ea853 --- /dev/null +++ b/config/locate/verify.pub @@ -0,0 +1 @@ +{"use":"sig","kty":"OKP","kid":"locate_20200409","crv":"Ed25519","alg":"EdDSA","x":"1tS1d-dd2B-VRBTWzOaq7zUngKDyV409K-o42LN2nx8"} diff --git a/config/mlab-node.env b/config/mlab-node.env new file mode 100644 index 0000000..e0d08c7 --- /dev/null +++ b/config/mlab-node.env @@ -0,0 +1,44 @@ +# Configuration for the M-Lab node (mlab-node) services. +# This file is sourced by the systemd units as an EnvironmentFile. Edit the +# values below to match your deployment, then start the stack with: +# systemctl start mlab-node.target +# +# The API key is kept separately in /etc/mlab/api-key.env. + +# ORGANIZATION is the organization name assigned by M-Lab to your deployment. +ORGANIZATION= + +# IATA is the 3-letter, lowercase IATA code of the nearest international +# airport. Estimate it from your latitude/longitude: +# https://autojoin-dot-mlab-autojoin.appspot.com/autojoin/v0/lookup?lat=&lon= +IATA= + +# PROBABILITY is the probability (0-1) that this node is selected by M-Lab's +# load balancer. Default is 0.1 (10%). +PROBABILITY=0.1 + +# INTERFACE_NAME is the network interface to which INTERFACE_MAXRATE applies. +INTERFACE_NAME= + +# UPLINK is the uplink speed, an integer followed by "g" (e.g. "1g", "10g", +# "25g"). Minimum is 1g; 10g or greater is recommended. +UPLINK= + +# INTERFACE_MAXRATE is the rate (bit/s) above which ndt-server refuses new +# connections. Recommended: 1G -> 150000000, 10G -> 7000000000. +INTERFACE_MAXRATE= + +# IPV4 is the external IPv4 address this node is reachable at. Required. +IPV4= + +# IPV6 is the external IPv6 address this node is reachable at. Optional; set it +# to make the node accessible over IPv6. +IPV6= + +# TYPE is "physical" (bare-metal dedicated to M-Lab) or "virtual" (a VM). +TYPE= + +# ### Static configuration. Do NOT change these. ### +PROJECT=mlab-autojoin +LOCATE_URL=locate.measurementlab.net +AUTH_URL=auth.mlab-oti.measurementlab.net diff --git a/config/modules-load.d/mlab-node.conf b/config/modules-load.d/mlab-node.conf new file mode 100644 index 0000000..da42f81 --- /dev/null +++ b/config/modules-load.d/mlab-node.conf @@ -0,0 +1,3 @@ +# ndt-server uses the BBR congestion control algorithm, which requires the +# tcp_bbr kernel module to be loaded at boot. +tcp_bbr diff --git a/config/sysctl.d/90-mlab-node.conf b/config/sysctl.d/90-mlab-node.conf new file mode 100644 index 0000000..0985521 --- /dev/null +++ b/config/sysctl.d/90-mlab-node.conf @@ -0,0 +1,4 @@ +# Enable BBR congestion control for ndt-server. Requires the tcp_bbr module +# (loaded via /usr/lib/modules-load.d/mlab-node.conf). +net.core.default_qdisc = fq +net.ipv4.tcp_congestion_control = bbr diff --git a/debian/changelog b/debian/changelog new file mode 100644 index 0000000..5f356bd --- /dev/null +++ b/debian/changelog @@ -0,0 +1,8 @@ +mlab-node (1.0.0) unstable; urgency=medium + + * Initial release: native Debian packaging of the M-Lab autonode stack. + Replaces the Docker Compose deployment with hardened systemd services. + Component binaries are extracted from the pinned M-Lab container images at + build time; no Docker is required at build or run time. + + -- Measurement Lab Mon, 29 Jun 2026 00:00:00 +0000 diff --git a/debian/control b/debian/control new file mode 100644 index 0000000..e1442e5 --- /dev/null +++ b/debian/control @@ -0,0 +1,31 @@ +Source: mlab-node +Section: net +Priority: optional +Maintainer: Measurement Lab +Build-Depends: + debhelper-compat (= 13), + skopeo, + jq, + file, +Standards-Version: 4.6.2 +Homepage: https://github.com/m-lab/autonode +Rules-Requires-Root: no + +Package: mlab-node +Architecture: amd64 +Depends: + ${shlibs:Depends}, + ${misc:Depends}, + curl, + kmod, +Description: M-Lab node (autonode) measurement stack + Native systemd packaging of the M-Lab "autonode" Network Diagnostic Tool + (NDT) measurement platform, replacing the former Docker Compose deployment. + . + It runs ndt-server and its sidecar services - node registration, heartbeat, + uuid-annotator, jostler, traceroute-caller (with scamper) and a Prometheus + node exporter - as hardened systemd services confined with systemd's own + sandboxing directives. No Docker is required at build or run time. + . + The component binaries are extracted from M-Lab's pinned container images at + build time. The whole stack is controlled atomically through mlab-node.target. diff --git a/debian/copyright b/debian/copyright new file mode 100644 index 0000000..e89a326 --- /dev/null +++ b/debian/copyright @@ -0,0 +1,32 @@ +Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/ +Upstream-Name: autonode +Source: https://github.com/m-lab/autonode + +Files: * +Copyright: Measurement Lab +License: Apache-2.0 + +Files: binaries/* +Copyright: Measurement Lab and the Prometheus Authors +Comment: + These binaries are extracted at build time from M-Lab's released container + images (and the Prometheus node_exporter image). They are not part of the + source in this repository. Each component is licensed under its own upstream + license; see the respective M-Lab and Prometheus projects. +License: Apache-2.0 + +License: Apache-2.0 + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + . + https://www.apache.org/licenses/LICENSE-2.0 + . + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + . + On Debian systems, the full text of the Apache License version 2.0 can be + found in the file /usr/share/common-licenses/Apache-2.0. diff --git a/debian/mlab-node.install b/debian/mlab-node.install new file mode 100644 index 0000000..68a06bc --- /dev/null +++ b/debian/mlab-node.install @@ -0,0 +1,47 @@ +# Component binaries (extracted from M-Lab images by .build/fetch-binaries.sh). +binaries/ndt-server usr/lib/mlab +binaries/heartbeat usr/lib/mlab +binaries/uuid-annotator usr/lib/mlab +binaries/jostler usr/lib/mlab +binaries/traceroute-caller usr/lib/mlab +binaries/scamper usr/lib/mlab +binaries/autojoin-register usr/lib/mlab +binaries/node-exporter usr/lib/mlab +binaries/generate-schemas-ndt7 usr/lib/mlab +binaries/generate-schemas-annotation2 usr/lib/mlab +binaries/generate-schemas-traceroute usr/lib/mlab + +# Helper scripts. +scripts/gen-uuid-prefix usr/lib/mlab +scripts/write-metadata usr/lib/mlab +scripts/wait-ready usr/lib/mlab +scripts/wait-socket usr/lib/mlab + +# systemd units. +systemd/mlab-node.target lib/systemd/system +systemd/mlab-node-register.service lib/systemd/system +systemd/mlab-node-register-ready.service lib/systemd/system +systemd/mlab-node-bbr.service lib/systemd/system +systemd/mlab-node-uuidprefix.service lib/systemd/system +systemd/mlab-node-metadata.service lib/systemd/system +systemd/mlab-node-schemas-ndt7.service lib/systemd/system +systemd/mlab-node-schemas-annotation2.service lib/systemd/system +systemd/mlab-node-schemas-traceroute.service lib/systemd/system +systemd/mlab-node-ndt-server.service lib/systemd/system +systemd/mlab-node-heartbeat.service lib/systemd/system +systemd/mlab-node-uuid-annotator.service lib/systemd/system +systemd/mlab-node-jostler.service lib/systemd/system +systemd/mlab-node-traceroute-caller.service lib/systemd/system +systemd/mlab-node-node-exporter.service lib/systemd/system + +# Configuration (files under /etc are registered as conffiles automatically). +config/mlab-node.env etc/mlab +config/api-key.env etc/mlab +config/locate/verify.pub etc/mlab/locate + +# Kernel module + sysctl drop-ins for BBR. +config/modules-load.d/mlab-node.conf usr/lib/modules-load.d +config/sysctl.d/90-mlab-node.conf usr/lib/sysctl.d + +# Static content served by ndt-server. +html/index.html usr/share/mlab/html diff --git a/debian/mlab-node.postinst b/debian/mlab-node.postinst new file mode 100755 index 0000000..b06fbc1 --- /dev/null +++ b/debian/mlab-node.postinst @@ -0,0 +1,15 @@ +#!/bin/sh +set -e + +#DEBHELPER# + +# Restrict the API key file to the service group. Runs after the debhelper +# snippets above have created the mlab-node user/group (sysusers). +if [ "$1" = "configure" ]; then + if getent group mlab-node >/dev/null 2>&1; then + chown root:mlab-node /etc/mlab/api-key.env || true + chmod 0640 /etc/mlab/api-key.env || true + fi +fi + +exit 0 diff --git a/debian/mlab-node.sysusers b/debian/mlab-node.sysusers new file mode 100644 index 0000000..678bca4 --- /dev/null +++ b/debian/mlab-node.sysusers @@ -0,0 +1,4 @@ +# Shared system user/group for the M-Lab node services. A single identity owns +# the service-account credential and the shared unix sockets, which are the only +# things the components share. node-exporter runs as a DynamicUser instead. +u mlab-node - "M-Lab node services" /var/lib/mlab /usr/sbin/nologin diff --git a/debian/mlab-node.tmpfiles b/debian/mlab-node.tmpfiles new file mode 100644 index 0000000..fe654f0 --- /dev/null +++ b/debian/mlab-node.tmpfiles @@ -0,0 +1,15 @@ +# Persistent state under /var/lib/mlab and runtime dirs under /run/mlab. +# setgid (2xxx) so files and sockets created here inherit the mlab-node group. + +# Persistent state. +d /var/lib/mlab 0755 mlab-node mlab-node - +d /var/lib/mlab/node 2750 mlab-node mlab-node - +d /var/lib/mlab/schemas 2755 mlab-node mlab-node - +d /var/lib/mlab/resultsdir 2755 mlab-node mlab-node - +d /var/lib/mlab/resultsdir/ndt 2755 mlab-node mlab-node - +d /var/lib/mlab/autocert 2750 mlab-node mlab-node - + +# Runtime (tmpfs, recreated each boot). +d /run/mlab 0755 mlab-node mlab-node - +d /run/mlab/sockets 2770 mlab-node mlab-node - +d /run/mlab/metadata 2755 mlab-node mlab-node - diff --git a/debian/rules b/debian/rules new file mode 100755 index 0000000..d2c316e --- /dev/null +++ b/debian/rules @@ -0,0 +1,23 @@ +#!/usr/bin/make -f + +%: + dh $@ + +# Extract the component binaries from the pinned container images (via skopeo, +# no Docker daemon) unless they are already staged. This runs as the package +# "build" step; the binaries are then installed via debian/mlab-node.install. +override_dh_auto_build: + if [ ! -e binaries/ndt-server ]; then ./.build/fetch-binaries.sh; fi + +# Install the target so it is enabled (auto-start on boot) but not started at +# install time - the operator must configure /etc/mlab/*.env first. The other +# units carry no [Install] section and are pulled in by the target, so they are +# only registered for daemon-reload, not enabled or started individually. +override_dh_installsystemd: + dh_installsystemd --no-start --no-enable + dh_installsystemd --no-start --name=mlab-node mlab-node.target + +# The shipped binaries are M-Lab's released, tested artifacts; do not strip or +# otherwise rewrite them. +override_dh_strip: +override_dh_dwz: diff --git a/debian/source/format b/debian/source/format new file mode 100644 index 0000000..89ae9db --- /dev/null +++ b/debian/source/format @@ -0,0 +1 @@ +3.0 (native) diff --git a/docker-compose.yml b/docker-compose.yml deleted file mode 100644 index 206973e..0000000 --- a/docker-compose.yml +++ /dev/null @@ -1,348 +0,0 @@ -services: - # Register this node with the Autojoin API. A successful registration will - # assign a DNS name for this node based on the public IPs, their ASN, - # organiztion, and IATA code. As well, Register distributes other credentials - # needed to upload and archive measurement data. The register command contacts - # the Autojoin API every hour. - register-node: - # NOTE: register should not be restarted on failure. - image: measurementlab/autojoin-register:v0.2.13 - profiles: [check-config, ndt] - pull_policy: always - network_mode: host - volumes: - - ./autonode:/autonode - restart: always - command: - - -endpoint=https://autojoin-dot-${PROJECT}.appspot.com/autojoin/v0/node/register - - -key=${API_KEY} - - -service=ndt - - -iata=${IATA} - - -output=/autonode - - -healthcheck-addr=:8001 - - -ports=9990,9991,9992,9993,9994,9995 - - -probability=${PROBABILITY} - - -ipv4=${IPV4} - - -ipv6=${IPV6} - - -type=${TYPE} - - -uplink=${UPLINK} - healthcheck: - test: ["CMD", "curl", "-f", "http://localhost:8001/ready"] - interval: 3s - timeout: 5s - retries: 10 - - # ndt-server is the public facing measurement service. Only ndt7 is enabled. - ndt-server: - image: measurementlab/ndt-server:v0.25.2 - profiles: [ndt] - network_mode: host - cap_add: - - NET_BIND_SERVICE - volumes: - - ./certs:/certs - - ./html:/html - - ./schemas:/schemas - - ./resultsdir:/resultsdir - - ./autonode:/autonode - - ./autocert:/autocert - depends_on: - generate-schemas-ndt7: - condition: service_completed_successfully - check-bbr: - condition: service_completed_successfully - register-node: - condition: service_healthy - restart: always - # NOTE: All containers will use the host network. All the following ports - # must not be in use. - ports: - # ndt-server TLS and non-TLS ports. - - target: 443 - published: 443 - protocol: tcp - - target: 80 - published: 80 - protocol: tcp - # ndt-server prometheus. - - target: 9990 - published: 9990 - protocol: tcp - # jostler prometheus. - - target: 9991 - published: 9991 - protocol: tcp - # annotator prometheus. - - target: 9992 - published: 9992 - protocol: tcp - # heartbeat prometheus. - - target: 9993 - published: 9993 - protocol: tcp - # traceroute-caller prometheus. - - target: 9994 - published: 9994 - protocol: tcp - # Generate the UUID prefix before starting ndt-server. This runs on every - # container start (including restart:always restarts after host reboot), - # ensuring a fresh prefix with the current boot time. The prefix uses btime - # from /proc/stat, which is a deterministic integer — no coordination file - # or separate container needed. - entrypoint: - - /bin/sh - - -c - - | - btime=$$(awk '/^btime / {print $$2}' /proc/stat) && - printf '%s_%s' "$$(hostname)" "$$btime" > /schemas/uuid.prefix && - exec /ndt-server "$$@" - - -- - command: - - -uuid-prefix-file=/schemas/uuid.prefix - - -datadir=/resultsdir/ndt - - -ndt7_addr=:443 - - -ndt7_addr_cleartext=:80 - # Allow ndt7 data to be autoloaded. - - -compress-results=false - # TODO(host): confirm device name. - - -txcontroller.device=${INTERFACE_NAME} - - -txcontroller.max-rate=${INTERFACE_MAXRATE} - - -prometheusx.listen-address=:9990 - # Add server metadata. - - -label=type=${TYPE} - - -label=deployment=byos - - -label=managed=none - - -label=loadbalanced=false - - -label=org=${ORGANIZATION} - - -label=external-ip=${IPV4} - - -label=external-ipv6=${IPV6} - # Effectively disable ndt5. - - -ndt5_addr=127.0.0.1:3002 - - -ndt5_ws_addr=127.0.0.1:3001 - - -tcpinfo.eventsocket=/schemas/events.sock - - -autocert.enabled=true - - -autocert.hostname=@/autonode/hostname - - -token.machine=@/autonode/hostname - - -token.verify-key=/locate/verify.pub - - -ndt7.token.required=true - configs: - - source: locate-verify-key - target: /locate/verify.pub - healthcheck: - test: ["CMD", "wget", "-O/dev/null", "-q", "http://localhost:80/"] - interval: 3s - timeout: 5s - retries: 10 - - # heartbeat reports health and liveness messages to M-Lab's Locate API every - # 10sec. The Locate API is responsible for directing clients to nearby healthy - # servers. - heartbeat: - image: measurementlab/heartbeat:v0.19.1 - profiles: [ndt] - volumes: - - ./autonode:/autonode - depends_on: - ndt-server: - condition: service_started - register-node: - condition: service_healthy - restart: always - command: - - -prometheusx.listen-address=:9993 - - -experiment=ndt - - -hostname=@/autonode/hostname - - -registration-url=file:///autonode/registration.json - - -heartbeat-url=wss://${LOCATE_URL}/v2/platform/heartbeat-jwt - - -api-key=${API_KEY} - - -token-exchange-url=https://${AUTH_URL}/v0/token/autojoin - - -services=ndt/ndt7=ws:///ndt/v7/download,ws:///ndt/v7/upload,wss:///ndt/v7/download,wss:///ndt/v7/upload - network_mode: host - - # uuid-annotator records network and geographic annotations based on the - # client and server IP so each NDT measurement is annotated in BigQuery. - uuid-annotator: - image: measurementlab/uuid-annotator:v0.5.10 - profiles: [ndt] - volumes: - - ./resultsdir:/resultsdir - - ./schemas:/schemas - - ./certs:/certs - - ./autonode:/autonode - depends_on: - ndt-server: - condition: service_started - register-node: - condition: service_healthy - network_mode: host - environment: - - GOOGLE_APPLICATION_CREDENTIALS=/autonode/service-account-autojoin.json - restart: always - # NOTE: all database URLs are required. - command: - - -prometheusx.listen-address=:9992 - - -datadir=/resultsdir/ndt/annotation2 - - -tcpinfo.eventsocket=/schemas/events.sock - - -ipservice.sock=/schemas/uuid-annotator.sock - - -maxmind.url=gs://downloader-${PROJECT}/Maxmind/current/GeoLite2-City.tar.gz - - -routeview-v4.url=gs://downloader-${PROJECT}/RouteViewIPv4/current/routeview.pfx2as.gz - - -routeview-v6.url=gs://downloader-${PROJECT}/RouteViewIPv6/current/routeview.pfx2as.gz - - -siteinfo.url=file:///autonode/annotation.json - - -hostname=@/autonode/hostname - - # jostler archives the raw data from ndt-server and the uuid-annotator into - # JSONL "bundles". These bundles are uploaded to GCS using credentials from - # register. Jostler depends on the local schemas being up to date or backward - # compatible with upstream schemas. - jostler: - image: measurementlab/jostler:v1.1.4 - profiles: [ndt] - volumes: - - ./resultsdir:/resultsdir - - ./schemas:/schemas - - ./certs:/certs - - ./autonode:/autonode - network_mode: host - depends_on: - generate-schemas-ndt7: - condition: service_completed_successfully - generate-schemas-annotation2: - condition: service_completed_successfully - generate-schemas-traceroute: - condition: service_completed_successfully - register-node: - condition: service_healthy - restart: always - environment: - - GOOGLE_APPLICATION_CREDENTIALS=/autonode/service-account-autojoin.json - # NOTE: jostler should not restart on exit. - command: - - -mlab-node-name=@/autonode/hostname - # NOTE: the ndt7 schema must already exist in the target bucket. - - -gcs-bucket=archive-${PROJECT} - - -gcs-data-dir=autoload/v2 - - -local-data-dir=/resultsdir - - -organization=${ORGANIZATION} - - -experiment=ndt - - -datatype=ndt7 - - -datatype-schema-file=ndt7:/schemas/ndt7.json - - -datatype=annotation2 - - -datatype-schema-file=annotation2:/schemas/annotation2.json - - -datatype=scamper2 - - -datatype-schema-file=scamper2:/schemas/scamper2.json - - -datatype=hopannotation2 - - -datatype-schema-file=hopannotation2:/schemas/hopannotation2.json - - -bundle-size-max=20971520 - - -bundle-age-max=1h - - -missed-age=2h - - -missed-interval=5m - - -extensions=.json - - -upload-schema=false - - -verbose - - -prometheusx.listen-address=:9991 - - traceroute-caller: - image: measurementlab/traceroute-caller:v0.12.0 - profiles: [ndt] - network_mode: host - volumes: - - ./resultsdir:/resultsdir - - ./schemas:/schemas - - ./metadata:/metadata - depends_on: - generate-schemas-traceroute: - condition: service_completed_successfully - register-node: - condition: service_healthy - ndt-server: - condition: service_healthy - pull_policy: always - restart: always - configs: - - source: metadata-external-ipv6 - target: /metadata/external-ipv6 - - source: metadata-external-ip - target: /metadata/external-ip - command: - - -prometheusx.listen-address=:9994 - - -traceroute-output=/resultsdir/ndt/scamper2 - - -hopannotation-output=/resultsdir/ndt/hopannotation2 - - -tcpinfo.eventsocket=/schemas/events.sock - - -IPCacheTimeout=10m - - -IPCacheUpdatePeriod=1m - - -scamper.timeout=30m - - -scamper.waitprobe=15 - - -scamper.ptr=true - - -ipservice.sock=/schemas/uuid-annotator.sock - - -anonymize.ip=none - - -scamper.trace-type=regular - - -output.format=json - - node-exporter: - image: quay.io/prometheus/node-exporter:v1.9.0 - profiles: [ndt] - container_name: node-exporter - restart: always - network_mode: host - pid: host - volumes: - - /:/host:ro,rslave - command: - - --web.listen-address=:9995 - - --path.rootfs=/host - - --collector.disable-defaults - - --collector.cpu - - --collector.diskstats - - --collector.filesystem - - --collector.loadavg - - --collector.meminfo - - --collector.netdev - - --collector.stat - - --collector.time - - # Generate the schemas needed by jostler to assess that the data is backward - # compatible with upstream schemas and safe to upload. - generate-schemas-ndt7: - network_mode: host - image: measurementlab/ndt-server:v0.25.2 - volumes: - - ./schemas:/schemas - entrypoint: - - /generate-schemas - - -ndt7=/schemas/ndt7.json - - generate-schemas-annotation2: - network_mode: host - image: measurementlab/uuid-annotator:v0.5.8 - volumes: - - ./schemas:/schemas - entrypoint: - - /generate-schemas - - -ann2=/schemas/annotation2.json - - generate-schemas-traceroute: - network_mode: host - image: measurementlab/traceroute-caller:v0.12.0 - volumes: - - ./schemas:/schemas - entrypoint: - - /generate-schemas - - -scamper2=/schemas/scamper2.json - - -hopannotation2=/schemas/hopannotation2.json - - # Check that the tcp_bbr module is available on the host. This is required - # for ndt-server to use the BBR congestion control algorithm. - check-bbr: - image: bash:latest - command: ["sh", "-c", "lsmod | grep tcp_bbr"] - -configs: - locate-verify-key: - content: | - {"use":"sig","kty":"OKP","kid":"locate_20200409","crv":"Ed25519","alg":"EdDSA","x":"1tS1d-dd2B-VRBTWzOaq7zUngKDyV409K-o42LN2nx8"} - metadata-external-ip: - content: |- - ${IPV4} - metadata-external-ipv6: - content: |- - ${IPV6} diff --git a/env b/env deleted file mode 100644 index 63b325d..0000000 --- a/env +++ /dev/null @@ -1,93 +0,0 @@ -# This is an example .env file for the ndt-fullstack docker compose file. -# -# ### External directories ### -# -# These folders are mounted by the various containers in the fullstack NDT -# Docker compose. If they do not exist, they are created the first time the -# containers start, owned by the docker daemon's user (generally root). -# -# The only folder that MUST exist and contain the service account key file -# (see below) is ./certs. This file must also be readable by Docker's user. -# -# All other folders are created and managed by the containers in the Docker -# compose. -# -# ./html: Content served by ndt-server. Can be used to provide your own -# speedtest UI. -# -# ./certs: Contains the service account key file. This key file (provided by -# M-Lab) is expected to be placed at certs/service-account-autojoin.json. -# This file must be present for several components in the docker compose to -# work properly. -# -# ./resultsdir: Contains the test results waiting to be uploaded. -# -# ./schemas: Contains the JSON schemas for the test results. -# -# ./autocert: Contains the automatically-generated LetsEncrypt TLS certificates -# so that they can be reused across docker compose restarts. -# -# ./autonode: Contains the configuration files for the NDT server and all the -# sidecar containers, provided by the Autojoin API on node registration. -# -# ./metadata: Contains configuration files used by some services such as -# traceroute-caller to read external IP addresses in some environments. - -# ### Configuration variables ### -# Change the following settings according to your environment. - -# ORGANIZATION is the organization name assigned by M-Lab to your deployment. -ORGANIZATION= - -# API_KEY is the API key provided by M-Lab. -API_KEY= - -# IATA is the 3-letter, lowercase IATA code of the nearest international -# airport. You can use the following service to estimate it, providing your -# server's latitude and longitude: -# https://autojoin-dot-mlab-autojoin.appspot.com/autojoin/v0/lookup?lat=&lon= -IATA= - -# PROBABILITY is the probability that this node will be selected by M-Lab's -# load balancer. Tweak it according to how much traffic this node can handle. -# A probability of 1 (100%) means that this node will be equally as likely to -# be selected as any other M-Lab node in the same metro. Default is 0.1 (10%). -PROBABILITY=0.1 - -# INTERFACE_NAME is the name of the interface to apply INTERFACE_MAXRATE to. -INTERFACE_NAME= - -# UPLINK is the speed of the uplink the M-Lab machine is connected to. The value -# must be expressed as an integer followed by the letter "g" (e.g., "1g", "10g", -# "25g"), where "g" represents Gbps (gigabits per second). The minimum uplink -# M-lab requires is 1g, but 10g or greater is recommended. -UPLINK= - -# INTERFACE_MAXRATE is the rate (in bit/s) after which the NDT server will -# refuse new connections. Set it according to this node's uplink defined in -# UPLINK. -# Recommended values: -# - 1G: INTERFACE_MAXRATE=150000000 -# - 10G: INTERFACE_MAXRATE=7000000000 -INTERFACE_MAXRATE= - -# IPV4 is the external IPv4 address that this node will be reachable at. -# This is required. -IPV4= - -# IPV6 is the external IPv6 address that this node will be reachable at. -# If you want you node to be accessible over IPv6, this must be set. -IPV6= - -# TYPE is the type of machine that will be running the M-Lab software. The only -# valid values are "physical" or "virtual". This helps M-Lab understand the -# environment where the software is running. In other words, is the machine a -# bare-metal machine dedicated to M-Lab, or a VM? -TYPE= - -# ### Static configuration variables ### -# ### Do NOT change these. ### - -PROJECT=mlab-autojoin -LOCATE_URL=locate.measurementlab.net -AUTH_URL=auth.mlab-oti.measurementlab.net diff --git a/html/index.html b/html/index.html new file mode 100644 index 0000000..4dbf4f1 --- /dev/null +++ b/html/index.html @@ -0,0 +1,6 @@ + +M-Lab NDT server +

M-Lab NDT server

+

This is an M-Lab Network +Diagnostic Tool (NDT) measurement server. See the +autonode project for details.

diff --git a/scripts/gen-uuid-prefix b/scripts/gen-uuid-prefix new file mode 100755 index 0000000..a38a2d9 --- /dev/null +++ b/scripts/gen-uuid-prefix @@ -0,0 +1,9 @@ +#!/bin/sh +# Generate the ndt-server UUID prefix from the kernel boot time and hostname. +# Runs once per boot (btime only changes across reboots). Replaces the inline +# /bin/sh entrypoint the ndt-server container used under docker-compose. +set -eu + +prefix_file="/var/lib/mlab/schemas/uuid.prefix" +btime="$(awk '/^btime / {print $2}' /proc/stat)" +printf '%s_%s' "$(hostname)" "${btime}" > "${prefix_file}" diff --git a/scripts/wait-ready b/scripts/wait-ready new file mode 100755 index 0000000..f4e38d4 --- /dev/null +++ b/scripts/wait-ready @@ -0,0 +1,15 @@ +#!/bin/sh +# Poll the register-node readiness endpoint until it reports ready. This gates +# every dependent service on a successful node registration (replacing the +# docker-compose `depends_on: condition: service_healthy`). +set -eu + +url="${1:-http://localhost:8001/ready}" +for _ in $(seq 1 60); do + if curl -fsS -o /dev/null "${url}"; then + exit 0 + fi + sleep 2 +done +echo "mlab-node: ${url} did not become ready in time" >&2 +exit 1 diff --git a/scripts/wait-socket b/scripts/wait-socket new file mode 100755 index 0000000..faf7068 --- /dev/null +++ b/scripts/wait-socket @@ -0,0 +1,13 @@ +#!/bin/sh +# Wait for a unix socket to appear before starting. `After=` only guarantees the +# producing service's process has started, not that it has created its socket; +# this closes that race for the tcpinfo/ipservice sockets. +set -eu + +sock="${1:?usage: wait-socket SOCKET_PATH}" +for _ in $(seq 1 60); do + [ -S "${sock}" ] && exit 0 + sleep 1 +done +echo "mlab-node: socket ${sock} did not appear in time" >&2 +exit 1 diff --git a/scripts/write-metadata b/scripts/write-metadata new file mode 100755 index 0000000..13a682f --- /dev/null +++ b/scripts/write-metadata @@ -0,0 +1,10 @@ +#!/bin/sh +# Write the external IP metadata files consumed by traceroute-caller. Replaces +# the docker-compose "configs" that injected ${IPV4}/${IPV6} as files. +# IPV4/IPV6 come from the mlab-node EnvironmentFile. +set -eu + +: "${IPV4:=}" +: "${IPV6:=}" +printf '%s' "${IPV4}" > /run/mlab/metadata/external-ip +printf '%s' "${IPV6}" > /run/mlab/metadata/external-ipv6 diff --git a/systemd/mlab-node-bbr.service b/systemd/mlab-node-bbr.service new file mode 100644 index 0000000..bae61be --- /dev/null +++ b/systemd/mlab-node-bbr.service @@ -0,0 +1,13 @@ +[Unit] +Description=Ensure tcp_bbr kernel module is loaded for ndt-server +Documentation=https://github.com/m-lab/autonode +PartOf=mlab-node.target +After=systemd-modules-load.service +# Replaces the docker-compose check-bbr container. modprobe both loads the +# module (covering a fresh install before the next reboot) and fails loudly if +# the kernel cannot provide it. modules-load.d also loads it at boot. + +[Service] +Type=oneshot +RemainAfterExit=yes +ExecStart=/sbin/modprobe tcp_bbr diff --git a/systemd/mlab-node-heartbeat.service b/systemd/mlab-node-heartbeat.service new file mode 100644 index 0000000..19038e5 --- /dev/null +++ b/systemd/mlab-node-heartbeat.service @@ -0,0 +1,54 @@ +[Unit] +Description=M-Lab heartbeat (reports liveness to the Locate API) +Documentation=https://github.com/m-lab/autonode +PartOf=mlab-node.target +Requires=mlab-node-register-ready.service +After=mlab-node-register-ready.service +# Soft dependency on ndt-server (compose used service_started): order after it +# but do not tear heartbeat down if ndt-server restarts. +Wants=mlab-node-ndt-server.service +After=mlab-node-ndt-server.service +# Atomic lifecycle: keystone is register-node. +BindsTo=mlab-node-register.service + +[Service] +Type=exec +EnvironmentFile=/etc/mlab/mlab-node.env +EnvironmentFile=/etc/mlab/api-key.env +ExecStart=/usr/lib/mlab/heartbeat \ + -prometheusx.listen-address=:9993 \ + -experiment=ndt \ + -hostname=@/var/lib/mlab/node/hostname \ + -registration-url=file:///var/lib/mlab/node/registration.json \ + -heartbeat-url=wss://${LOCATE_URL}/v2/platform/heartbeat-jwt \ + -api-key=${API_KEY} \ + -token-exchange-url=https://${AUTH_URL}/v0/token/autojoin \ + -services=ndt/ndt7=ws:///ndt/v7/download,ws:///ndt/v7/upload,wss:///ndt/v7/download,wss:///ndt/v7/upload +Restart=always +RestartSec=5 + +# --- hardening --- +User=mlab-node +Group=mlab-node +NoNewPrivileges=yes +ProtectSystem=strict +ProtectHome=yes +ProtectProc=invisible +ProtectKernelTunables=yes +ProtectKernelModules=yes +ProtectKernelLogs=yes +ProtectControlGroups=yes +ProtectClock=yes +ProtectHostname=yes +PrivateTmp=yes +PrivateDevices=yes +RestrictNamespaces=yes +RestrictRealtime=yes +RestrictSUIDSGID=yes +LockPersonality=yes +MemoryDenyWriteExecute=yes +RemoveIPC=yes +CapabilityBoundingSet= +RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX +SystemCallFilter=@system-service +SystemCallErrorNumber=EPERM diff --git a/systemd/mlab-node-jostler.service b/systemd/mlab-node-jostler.service new file mode 100644 index 0000000..4c01036 --- /dev/null +++ b/systemd/mlab-node-jostler.service @@ -0,0 +1,66 @@ +[Unit] +Description=M-Lab jostler (archives results to GCS) +Documentation=https://github.com/m-lab/autonode +PartOf=mlab-node.target +# Requires all schemas to be generated (service_completed_successfully) and a +# successful registration. +Requires=mlab-node-register-ready.service mlab-node-schemas-ndt7.service mlab-node-schemas-annotation2.service mlab-node-schemas-traceroute.service +After=mlab-node-register-ready.service mlab-node-schemas-ndt7.service mlab-node-schemas-annotation2.service mlab-node-schemas-traceroute.service +BindsTo=mlab-node-register.service + +[Service] +Type=exec +EnvironmentFile=/etc/mlab/mlab-node.env +Environment=GOOGLE_APPLICATION_CREDENTIALS=/var/lib/mlab/node/service-account-autojoin.json +ExecStart=/usr/lib/mlab/jostler \ + -mlab-node-name=@/var/lib/mlab/node/hostname \ + -gcs-bucket=archive-${PROJECT} \ + -gcs-data-dir=autoload/v2 \ + -local-data-dir=/var/lib/mlab/resultsdir \ + -organization=${ORGANIZATION} \ + -experiment=ndt \ + -datatype=ndt7 \ + -datatype-schema-file=ndt7:/var/lib/mlab/schemas/ndt7.json \ + -datatype=annotation2 \ + -datatype-schema-file=annotation2:/var/lib/mlab/schemas/annotation2.json \ + -datatype=scamper2 \ + -datatype-schema-file=scamper2:/var/lib/mlab/schemas/scamper2.json \ + -datatype=hopannotation2 \ + -datatype-schema-file=hopannotation2:/var/lib/mlab/schemas/hopannotation2.json \ + -bundle-size-max=20971520 \ + -bundle-age-max=1h \ + -missed-age=2h \ + -missed-interval=5m \ + -extensions=.json \ + -upload-schema=false \ + -verbose \ + -prometheusx.listen-address=:9991 +Restart=always +RestartSec=5 + +# --- hardening --- +User=mlab-node +Group=mlab-node +ReadWritePaths=/var/lib/mlab/resultsdir +NoNewPrivileges=yes +ProtectSystem=strict +ProtectHome=yes +ProtectProc=invisible +ProtectKernelTunables=yes +ProtectKernelModules=yes +ProtectKernelLogs=yes +ProtectControlGroups=yes +ProtectClock=yes +ProtectHostname=yes +PrivateTmp=yes +PrivateDevices=yes +RestrictNamespaces=yes +RestrictRealtime=yes +RestrictSUIDSGID=yes +LockPersonality=yes +MemoryDenyWriteExecute=yes +RemoveIPC=yes +CapabilityBoundingSet= +RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX +SystemCallFilter=@system-service +SystemCallErrorNumber=EPERM diff --git a/systemd/mlab-node-metadata.service b/systemd/mlab-node-metadata.service new file mode 100644 index 0000000..8e387c4 --- /dev/null +++ b/systemd/mlab-node-metadata.service @@ -0,0 +1,25 @@ +[Unit] +Description=Write external IP metadata files for traceroute-caller +Documentation=https://github.com/m-lab/autonode +PartOf=mlab-node.target +RequiresMountsFor=/run/mlab +# Replaces the docker-compose "configs" that injected ${IPV4}/${IPV6} as the +# /metadata/external-ip[v6] files. Regenerated each boot (the files live on the +# /run tmpfs). + +[Service] +Type=oneshot +RemainAfterExit=yes +EnvironmentFile=/etc/mlab/mlab-node.env +ExecStart=/usr/lib/mlab/write-metadata + +# --- hardening --- +User=mlab-node +Group=mlab-node +ReadWritePaths=/run/mlab/metadata +NoNewPrivileges=yes +ProtectSystem=strict +ProtectHome=yes +PrivateTmp=yes +RestrictAddressFamilies=AF_UNIX +CapabilityBoundingSet= diff --git a/systemd/mlab-node-ndt-server.service b/systemd/mlab-node-ndt-server.service new file mode 100644 index 0000000..3910ec3 --- /dev/null +++ b/systemd/mlab-node-ndt-server.service @@ -0,0 +1,73 @@ +[Unit] +Description=M-Lab ndt-server (NDT7 measurement service) +Documentation=https://github.com/m-lab/autonode +PartOf=mlab-node.target +# Start ordering / gating (replaces docker-compose depends_on): +# register-ready -> registration succeeded (service_healthy) +# schemas-ndt7 -> ndt7 schema generated (service_completed_successfully) +# uuidprefix/bbr -> prerequisites generated/loaded +Requires=mlab-node-register-ready.service mlab-node-schemas-ndt7.service mlab-node-uuidprefix.service mlab-node-bbr.service +After=mlab-node-register-ready.service mlab-node-schemas-ndt7.service mlab-node-uuidprefix.service mlab-node-bbr.service +# Atomic lifecycle: ndt-server cannot run without an active registration. +BindsTo=mlab-node-register.service + +[Service] +Type=exec +EnvironmentFile=/etc/mlab/mlab-node.env +# Clear any stale tcpinfo socket left by a previous instance before binding. +ExecStartPre=/bin/rm -f /run/mlab/sockets/events.sock +ExecStart=/usr/lib/mlab/ndt-server \ + -uuid-prefix-file=/var/lib/mlab/schemas/uuid.prefix \ + -datadir=/var/lib/mlab/resultsdir/ndt \ + -ndt7_addr=:443 \ + -ndt7_addr_cleartext=:80 \ + -compress-results=false \ + -txcontroller.device=${INTERFACE_NAME} \ + -txcontroller.max-rate=${INTERFACE_MAXRATE} \ + -prometheusx.listen-address=:9990 \ + -label=type=${TYPE} \ + -label=deployment=byos \ + -label=managed=none \ + -label=loadbalanced=false \ + -label=org=${ORGANIZATION} \ + -label=external-ip=${IPV4} \ + -label=external-ipv6=${IPV6} \ + -ndt5_addr=127.0.0.1:3002 \ + -ndt5_ws_addr=127.0.0.1:3001 \ + -tcpinfo.eventsocket=/run/mlab/sockets/events.sock \ + -autocert.enabled=true \ + -autocert.hostname=@/var/lib/mlab/node/hostname \ + -token.machine=@/var/lib/mlab/node/hostname \ + -token.verify-key=/etc/mlab/locate/verify.pub \ + -ndt7.token.required=true +Restart=always +RestartSec=5 + +# --- hardening --- +User=mlab-node +Group=mlab-node +# Bind the privileged ports 80/443 without running as root. +AmbientCapabilities=CAP_NET_BIND_SERVICE +CapabilityBoundingSet=CAP_NET_BIND_SERVICE +ReadWritePaths=/var/lib/mlab/resultsdir /var/lib/mlab/autocert /run/mlab/sockets +NoNewPrivileges=yes +ProtectSystem=strict +ProtectHome=yes +ProtectProc=invisible +ProtectKernelTunables=yes +ProtectKernelModules=yes +ProtectKernelLogs=yes +ProtectControlGroups=yes +ProtectClock=yes +ProtectHostname=yes +PrivateTmp=yes +RestrictNamespaces=yes +RestrictRealtime=yes +RestrictSUIDSGID=yes +LockPersonality=yes +MemoryDenyWriteExecute=yes +RemoveIPC=yes +# AF_NETLINK is required by -txcontroller to read the interface transmit rate. +RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK +SystemCallFilter=@system-service +SystemCallErrorNumber=EPERM diff --git a/systemd/mlab-node-node-exporter.service b/systemd/mlab-node-node-exporter.service new file mode 100644 index 0000000..79119d1 --- /dev/null +++ b/systemd/mlab-node-node-exporter.service @@ -0,0 +1,52 @@ +[Unit] +Description=Prometheus node exporter for the M-Lab node +Documentation=https://github.com/m-lab/autonode +PartOf=mlab-node.target +After=network-online.target +Wants=network-online.target +# Part of the atomic stack: brought down with the keystone register-node. +BindsTo=mlab-node-register.service + +[Service] +Type=exec +# Running natively on the host, so the default rootfs (/) is correct; the +# --path.rootfs=/host indirection used under Docker is dropped. +ExecStart=/usr/lib/mlab/node-exporter \ + --web.listen-address=:9995 \ + --collector.disable-defaults \ + --collector.cpu \ + --collector.diskstats \ + --collector.filesystem \ + --collector.loadavg \ + --collector.meminfo \ + --collector.netdev \ + --collector.stat \ + --collector.time +Restart=always +RestartSec=5 + +# --- hardening --- +# node-exporter shares nothing with the rest of the stack and needs no state, so +# it runs under an ephemeral DynamicUser. It only reads world-readable /proc and +# /sys, so it needs no capabilities and no write access. +DynamicUser=yes +NoNewPrivileges=yes +ProtectSystem=strict +ProtectHome=yes +ProtectProc=default +ProcSubset=all +ProtectKernelTunables=yes +ProtectKernelModules=yes +ProtectKernelLogs=yes +ProtectControlGroups=yes +ProtectClock=yes +PrivateTmp=yes +RestrictNamespaces=yes +RestrictRealtime=yes +RestrictSUIDSGID=yes +LockPersonality=yes +MemoryDenyWriteExecute=yes +CapabilityBoundingSet= +RestrictAddressFamilies=AF_INET AF_INET6 +SystemCallFilter=@system-service +SystemCallErrorNumber=EPERM diff --git a/systemd/mlab-node-register-ready.service b/systemd/mlab-node-register-ready.service new file mode 100644 index 0000000..86fcb63 --- /dev/null +++ b/systemd/mlab-node-register-ready.service @@ -0,0 +1,25 @@ +[Unit] +Description=Wait for M-Lab node registration to be ready +Documentation=https://github.com/m-lab/autonode +PartOf=mlab-node.target +Requires=mlab-node-register.service +After=mlab-node-register.service +BindsTo=mlab-node-register.service +# This oneshot gates every dependent component on a successful registration +# (the register-node :8001/ready endpoint), guaranteeing the hostname, +# registration.json and service-account files exist before anything reads them. + +[Service] +Type=oneshot +RemainAfterExit=yes +ExecStart=/usr/lib/mlab/wait-ready http://localhost:8001/ready + +# --- hardening --- +User=mlab-node +Group=mlab-node +NoNewPrivileges=yes +ProtectSystem=strict +ProtectHome=yes +PrivateTmp=yes +RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX +CapabilityBoundingSet= diff --git a/systemd/mlab-node-register.service b/systemd/mlab-node-register.service new file mode 100644 index 0000000..61bb819 --- /dev/null +++ b/systemd/mlab-node-register.service @@ -0,0 +1,59 @@ +[Unit] +Description=M-Lab node registration (Autojoin API) +Documentation=https://github.com/m-lab/autonode +PartOf=mlab-node.target +After=network-online.target +Wants=network-online.target +# register-node is the keystone of the stack: every other component BindsTo= +# this unit, so if registration goes down the whole stack is brought down with +# it (it cannot run without a valid registration). A genuine crash-loop is +# bounded by the start limit below, at which point the stack stays down. +StartLimitIntervalSec=600 +StartLimitBurst=5 + +[Service] +Type=exec +EnvironmentFile=/etc/mlab/mlab-node.env +EnvironmentFile=/etc/mlab/api-key.env +ExecStart=/usr/lib/mlab/autojoin-register \ + -endpoint=https://autojoin-dot-${PROJECT}.appspot.com/autojoin/v0/node/register \ + -key=${API_KEY} \ + -service=ndt \ + -iata=${IATA} \ + -output=/var/lib/mlab/node \ + -healthcheck-addr=:8001 \ + -ports=9990,9991,9992,9993,9994,9995 \ + -probability=${PROBABILITY} \ + -ipv4=${IPV4} \ + -ipv6=${IPV6} \ + -type=${TYPE} \ + -uplink=${UPLINK} +Restart=always +RestartSec=10 + +# --- hardening --- +User=mlab-node +Group=mlab-node +ReadWritePaths=/var/lib/mlab/node +NoNewPrivileges=yes +ProtectSystem=strict +ProtectHome=yes +ProtectProc=invisible +ProtectKernelTunables=yes +ProtectKernelModules=yes +ProtectKernelLogs=yes +ProtectControlGroups=yes +ProtectClock=yes +ProtectHostname=yes +PrivateTmp=yes +PrivateDevices=yes +RestrictNamespaces=yes +RestrictRealtime=yes +RestrictSUIDSGID=yes +LockPersonality=yes +MemoryDenyWriteExecute=yes +RemoveIPC=yes +CapabilityBoundingSet= +RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX +SystemCallFilter=@system-service +SystemCallErrorNumber=EPERM diff --git a/systemd/mlab-node-schemas-annotation2.service b/systemd/mlab-node-schemas-annotation2.service new file mode 100644 index 0000000..9848bc0 --- /dev/null +++ b/systemd/mlab-node-schemas-annotation2.service @@ -0,0 +1,21 @@ +[Unit] +Description=Generate annotation2 JSON schema +Documentation=https://github.com/m-lab/autonode +PartOf=mlab-node.target +# Replaces the generate-schemas-annotation2 init container. + +[Service] +Type=oneshot +RemainAfterExit=yes +ExecStart=/usr/lib/mlab/generate-schemas-annotation2 -ann2=/var/lib/mlab/schemas/annotation2.json + +# --- hardening --- +User=mlab-node +Group=mlab-node +ReadWritePaths=/var/lib/mlab/schemas +NoNewPrivileges=yes +ProtectSystem=strict +ProtectHome=yes +PrivateTmp=yes +RestrictAddressFamilies=AF_UNIX +CapabilityBoundingSet= diff --git a/systemd/mlab-node-schemas-ndt7.service b/systemd/mlab-node-schemas-ndt7.service new file mode 100644 index 0000000..ffbbd3e --- /dev/null +++ b/systemd/mlab-node-schemas-ndt7.service @@ -0,0 +1,22 @@ +[Unit] +Description=Generate ndt7 JSON schema +Documentation=https://github.com/m-lab/autonode +PartOf=mlab-node.target +# Replaces the generate-schemas-ndt7 init container. jostler validates uploads +# against these schemas; ndt-server also requires the ndt7 schema. + +[Service] +Type=oneshot +RemainAfterExit=yes +ExecStart=/usr/lib/mlab/generate-schemas-ndt7 -ndt7=/var/lib/mlab/schemas/ndt7.json + +# --- hardening --- +User=mlab-node +Group=mlab-node +ReadWritePaths=/var/lib/mlab/schemas +NoNewPrivileges=yes +ProtectSystem=strict +ProtectHome=yes +PrivateTmp=yes +RestrictAddressFamilies=AF_UNIX +CapabilityBoundingSet= diff --git a/systemd/mlab-node-schemas-traceroute.service b/systemd/mlab-node-schemas-traceroute.service new file mode 100644 index 0000000..9e27f09 --- /dev/null +++ b/systemd/mlab-node-schemas-traceroute.service @@ -0,0 +1,23 @@ +[Unit] +Description=Generate scamper2 and hopannotation2 JSON schemas +Documentation=https://github.com/m-lab/autonode +PartOf=mlab-node.target +# Replaces the generate-schemas-traceroute init container. + +[Service] +Type=oneshot +RemainAfterExit=yes +ExecStart=/usr/lib/mlab/generate-schemas-traceroute \ + -scamper2=/var/lib/mlab/schemas/scamper2.json \ + -hopannotation2=/var/lib/mlab/schemas/hopannotation2.json + +# --- hardening --- +User=mlab-node +Group=mlab-node +ReadWritePaths=/var/lib/mlab/schemas +NoNewPrivileges=yes +ProtectSystem=strict +ProtectHome=yes +PrivateTmp=yes +RestrictAddressFamilies=AF_UNIX +CapabilityBoundingSet= diff --git a/systemd/mlab-node-traceroute-caller.service b/systemd/mlab-node-traceroute-caller.service new file mode 100644 index 0000000..cff2135 --- /dev/null +++ b/systemd/mlab-node-traceroute-caller.service @@ -0,0 +1,65 @@ +[Unit] +Description=M-Lab traceroute-caller (runs scamper traceroutes) +Documentation=https://github.com/m-lab/autonode +PartOf=mlab-node.target +Requires=mlab-node-register-ready.service mlab-node-schemas-traceroute.service mlab-node-metadata.service +After=mlab-node-register-ready.service mlab-node-schemas-traceroute.service mlab-node-metadata.service +# Compose required ndt-server to be healthy (traceroute-caller connects to its +# events.sock); order after it and wait for the socket in ExecStartPre. +Wants=mlab-node-ndt-server.service +After=mlab-node-ndt-server.service +BindsTo=mlab-node-register.service + +[Service] +Type=exec +EnvironmentFile=/etc/mlab/mlab-node.env +# scamper is shipped alongside traceroute-caller; make it discoverable on PATH. +Environment=PATH=/usr/lib/mlab:/usr/sbin:/usr/bin:/sbin:/bin +ExecStartPre=/usr/lib/mlab/wait-socket /run/mlab/sockets/events.sock +ExecStartPre=/usr/lib/mlab/wait-socket /run/mlab/sockets/uuid-annotator.sock +ExecStart=/usr/lib/mlab/traceroute-caller \ + -prometheusx.listen-address=:9994 \ + -traceroute-output=/var/lib/mlab/resultsdir/ndt/scamper2 \ + -hopannotation-output=/var/lib/mlab/resultsdir/ndt/hopannotation2 \ + -tcpinfo.eventsocket=/run/mlab/sockets/events.sock \ + -IPCacheTimeout=10m \ + -IPCacheUpdatePeriod=1m \ + -scamper.timeout=30m \ + -scamper.waitprobe=15 \ + -scamper.ptr=true \ + -ipservice.sock=/run/mlab/sockets/uuid-annotator.sock \ + -anonymize.ip=none \ + -scamper.trace-type=regular \ + -output.format=json +Restart=always +RestartSec=5 + +# --- hardening --- +User=mlab-node +Group=mlab-node +# scamper needs raw sockets to send/receive probes; the child inherits the +# ambient capability from traceroute-caller. +AmbientCapabilities=CAP_NET_RAW +CapabilityBoundingSet=CAP_NET_RAW +ReadWritePaths=/var/lib/mlab/resultsdir +NoNewPrivileges=yes +ProtectSystem=strict +ProtectHome=yes +ProtectProc=invisible +ProtectKernelTunables=yes +ProtectKernelModules=yes +ProtectKernelLogs=yes +ProtectControlGroups=yes +ProtectClock=yes +ProtectHostname=yes +PrivateTmp=yes +RestrictNamespaces=yes +RestrictRealtime=yes +RestrictSUIDSGID=yes +LockPersonality=yes +# Relaxed for the bundled scamper C binary. +MemoryDenyWriteExecute=no +RemoveIPC=yes +RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK AF_PACKET +SystemCallFilter=@system-service +SystemCallErrorNumber=EPERM diff --git a/systemd/mlab-node-uuid-annotator.service b/systemd/mlab-node-uuid-annotator.service new file mode 100644 index 0000000..ae5d621 --- /dev/null +++ b/systemd/mlab-node-uuid-annotator.service @@ -0,0 +1,58 @@ +[Unit] +Description=M-Lab uuid-annotator (network/geo annotations) +Documentation=https://github.com/m-lab/autonode +PartOf=mlab-node.target +Requires=mlab-node-register-ready.service +After=mlab-node-register-ready.service +# Soft dependency on ndt-server (compose used service_started); the annotator +# connects to ndt-server's tcpinfo events.sock. +Wants=mlab-node-ndt-server.service +After=mlab-node-ndt-server.service +BindsTo=mlab-node-register.service + +[Service] +Type=exec +EnvironmentFile=/etc/mlab/mlab-node.env +Environment=GOOGLE_APPLICATION_CREDENTIALS=/var/lib/mlab/node/service-account-autojoin.json +# Wait for ndt-server's events socket, and clear any stale annotator socket. +ExecStartPre=/usr/lib/mlab/wait-socket /run/mlab/sockets/events.sock +ExecStartPre=/bin/rm -f /run/mlab/sockets/uuid-annotator.sock +ExecStart=/usr/lib/mlab/uuid-annotator \ + -prometheusx.listen-address=:9992 \ + -datadir=/var/lib/mlab/resultsdir/ndt/annotation2 \ + -tcpinfo.eventsocket=/run/mlab/sockets/events.sock \ + -ipservice.sock=/run/mlab/sockets/uuid-annotator.sock \ + -maxmind.url=gs://downloader-${PROJECT}/Maxmind/current/GeoLite2-City.tar.gz \ + -routeview-v4.url=gs://downloader-${PROJECT}/RouteViewIPv4/current/routeview.pfx2as.gz \ + -routeview-v6.url=gs://downloader-${PROJECT}/RouteViewIPv6/current/routeview.pfx2as.gz \ + -siteinfo.url=file:///var/lib/mlab/node/annotation.json \ + -hostname=@/var/lib/mlab/node/hostname +Restart=always +RestartSec=5 + +# --- hardening --- +User=mlab-node +Group=mlab-node +ReadWritePaths=/var/lib/mlab/resultsdir /run/mlab/sockets +NoNewPrivileges=yes +ProtectSystem=strict +ProtectHome=yes +ProtectProc=invisible +ProtectKernelTunables=yes +ProtectKernelModules=yes +ProtectKernelLogs=yes +ProtectControlGroups=yes +ProtectClock=yes +ProtectHostname=yes +PrivateTmp=yes +PrivateDevices=yes +RestrictNamespaces=yes +RestrictRealtime=yes +RestrictSUIDSGID=yes +LockPersonality=yes +MemoryDenyWriteExecute=yes +RemoveIPC=yes +CapabilityBoundingSet= +RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX +SystemCallFilter=@system-service +SystemCallErrorNumber=EPERM diff --git a/systemd/mlab-node-uuidprefix.service b/systemd/mlab-node-uuidprefix.service new file mode 100644 index 0000000..92be4f1 --- /dev/null +++ b/systemd/mlab-node-uuidprefix.service @@ -0,0 +1,22 @@ +[Unit] +Description=Generate ndt-server UUID prefix from boot time +Documentation=https://github.com/m-lab/autonode +PartOf=mlab-node.target +# Runs once per boot; btime (from /proc/stat) only changes across reboots. +# Replaces the inline shell entrypoint the ndt-server container used. + +[Service] +Type=oneshot +RemainAfterExit=yes +ExecStart=/usr/lib/mlab/gen-uuid-prefix + +# --- hardening --- +User=mlab-node +Group=mlab-node +ReadWritePaths=/var/lib/mlab/schemas +NoNewPrivileges=yes +ProtectSystem=strict +ProtectHome=yes +PrivateTmp=yes +RestrictAddressFamilies=AF_UNIX +CapabilityBoundingSet= diff --git a/systemd/mlab-node.target b/systemd/mlab-node.target new file mode 100644 index 0000000..fe1da6a --- /dev/null +++ b/systemd/mlab-node.target @@ -0,0 +1,16 @@ +[Unit] +Description=M-Lab node (autonode) measurement stack +Documentation=https://github.com/m-lab/autonode +# The target is the single handle for the whole stack, replacing +# `docker compose up/down`. Starting it pulls in every component; stopping it +# stops them all (each component is PartOf= this target). +Wants=mlab-node-register.service +Wants=mlab-node-ndt-server.service +Wants=mlab-node-heartbeat.service +Wants=mlab-node-uuid-annotator.service +Wants=mlab-node-jostler.service +Wants=mlab-node-traceroute-caller.service +Wants=mlab-node-node-exporter.service + +[Install] +WantedBy=multi-user.target From 32a2206e1fc4a3de3e9c0009d1e5afe2e0b3f29b Mon Sep 17 00:00:00 2001 From: Roberto D'Auria Date: Tue, 30 Jun 2026 14:54:22 +0200 Subject: [PATCH 02/25] fetch-binaries: handle musl outlier via musl dependency, not abort Real extraction on Ubuntu 24.04 showed 9/11 binaries static, scamper glibc-dynamic, and only generate-schemas-ndt7 musl-linked (needs just libc.musl-x86_64.so.1). Stop aborting on musl: warn instead, add Depends: musl, and exclude that one binary from dh_shlibdeps (no musl shlibs info) while keeping strict resolution for scamper's glibc libs. Harden temp cleanup against root-owned image entries. --- .DS_Store | Bin 0 -> 8196 bytes .build/fetch-binaries.sh | 13 ++++++++----- debian/control | 1 + debian/rules | 6 ++++++ 4 files changed, 15 insertions(+), 5 deletions(-) create mode 100644 .DS_Store diff --git a/.DS_Store b/.DS_Store new file mode 100644 index 0000000000000000000000000000000000000000..f1a1ab67596f1a1f84da061e894de2fd09a0fb49 GIT binary patch literal 8196 zcmeHM%}*0S6n}#VTTxjQ6pSVtdoh71A8ItlPzppFKOmGK_*HhdLs?m7n%xBiLejII z^saZWo=m)W^y1Z<@lVjJCw=o#p)HtrF&bx*ncv%)_j^0@Yi4KP763?gr8of44FDEa zfp!}@*O@~pyC}0#Y3K#{90)K-7cxH=Y?Q!m_ zQPUa)i~|3q0^<4LU=^6rSWze+9oWbe0I?anWkDPB0LgJQW;9k5N-D-w*#ps3qEif_ z=s4~Ocf^dwib6#PqUb>Mor%s+h`c-U9O(`uqtLWQ0i!^p0wQ;B0UtOpSg6b2#a7B< zJ^Ni2Np686Q1H;x1-2vC7rw85C;4E57VIN+u2Z!5aJy@Iy^KDc+Z{68wTMzC<4r6q z#59@RVb2xp_Slbj7*6syNCM90?#HNgE}_dgw!ra@tElrtSq#N9G>Q=$IJ0wZ6uGQK zv+EB+*Q4PC%km`Q%feGc^EsQ0sE9Nhpe@#t9{#GNOlNLXjWf!yA*~F28||!G8cF({ zKQHZR#u-he(wRv;WI->Cz%7`8CD?#x@B&`JTX+wj;0t^st)z#XCcWf5xk`q~b&?}@ ziBA?uKprW6Q}u^FrQ>NJF&Z^L^lCn0A!jA*7o28Tgano$tla?QG>+}uEUj=8axeqfX4%Q&C7&O6%0dmUo*q>ZuV zMt`|uqa#@?)zO&ISWzgI!$zLMi01r9T}&|w{7VY#Q/dev/null || true; rm -rf "${WORK_DIR}" 2>/dev/null || true' EXIT # unpack_image IMAGE DEST_ROOTFS # Copies the image to a local OCI/dir layout with skopeo and extracts every @@ -91,7 +93,10 @@ unpack_image() { UNPACKED[${image}]=1 } -# check_binary PATH — abort if the ELF is musl-linked (won't run on Debian). +# check_binary PATH — classify the ELF. glibc-dynamic and static binaries run on +# Debian as-is (dh_shlibdeps resolves shared-lib deps). musl-linked binaries run +# only with the `musl` package installed, so they are flagged (the package +# declares the dependency) rather than rejected. A non-ELF file is fatal. check_binary() { local bin="$1" info info="$(file -L "${bin}")" @@ -99,9 +104,7 @@ check_binary() { *"statically linked"*) ;; # portable, OK *"dynamically linked"*) if echo "${info}" | grep -q "ld-musl"; then - echo "ERROR: ${bin} is musl-linked (Alpine); it will not run on Debian." >&2 - echo " Build this component from source or use a Debian package instead." >&2 - exit 1 + echo " WARNING: ${bin##*/} is musl-linked; requires the 'musl' package at runtime." >&2 fi ;; # glibc dynamic, OK (dh_shlibdeps handles deps) *"ELF"*) ;; diff --git a/debian/control b/debian/control index e1442e5..f72a757 100644 --- a/debian/control +++ b/debian/control @@ -18,6 +18,7 @@ Depends: ${misc:Depends}, curl, kmod, + musl, Description: M-Lab node (autonode) measurement stack Native systemd packaging of the M-Lab "autonode" Network Diagnostic Tool (NDT) measurement platform, replacing the former Docker Compose deployment. diff --git a/debian/rules b/debian/rules index d2c316e..1c147e0 100755 --- a/debian/rules +++ b/debian/rules @@ -21,3 +21,9 @@ override_dh_installsystemd: # otherwise rewrite them. override_dh_strip: override_dh_dwz: + +# scamper and the static Go binaries are resolved normally. generate-schemas-ndt7 +# is the lone musl-linked binary; dpkg-shlibdeps has no shlibs info for musl, so +# exclude it here - its runtime requirement is declared as Depends: musl. +override_dh_shlibdeps: + dh_shlibdeps -Xgenerate-schemas-ndt7 From c733a15285216924d05d8901d44a452480c5ced7 Mon Sep 17 00:00:00 2001 From: Roberto D'Auria Date: Tue, 30 Jun 2026 15:30:23 +0200 Subject: [PATCH 03/25] generate-schemas-ndt7: build from source with CGO, drop musl approach The ndt-server image builds cmd/generate-schemas on Alpine with CGO (musl-dev), so the extracted copy is musl-linked. cmd/generate-schemas is a nested Go module that pins ndt-server@v0.20.17, whose bbr package needs CGO on Linux (so a static build fails). Instead, build that nested module from the pinned source with CGO on the glibc build host -> a glibc-dynamic binary (NEEDED: libc.so.6 only, resolved by base libc6) that produces a schema byte-identical to the image's. - fetch-binaries.sh: stop extracting generate-schemas-ndt7; shallow-clone ndt-server@v0.25.2 and 'go build' the nested cmd/generate-schemas with CGO_ENABLED=1 and a pinned GOTOOLCHAIN. Require go+git. - control: drop Depends: musl; add golang-go/git/ca-certificates/gcc/libc6-dev to Build-Depends. - rules: drop the dh_shlibdeps -X exclude (binary is now glibc-resolvable). --- .build/fetch-binaries.sh | 30 ++++++++++++++++++++++++++++-- debian/control | 6 +++++- debian/rules | 6 ------ 3 files changed, 33 insertions(+), 9 deletions(-) diff --git a/.build/fetch-binaries.sh b/.build/fetch-binaries.sh index b3c2c44..67f6040 100755 --- a/.build/fetch-binaries.sh +++ b/.build/fetch-binaries.sh @@ -32,12 +32,22 @@ IMG_TRACEROUTE="measurementlab/traceroute-caller:v0.12.0" IMG_REGISTER="measurementlab/autojoin-register:v0.2.13" IMG_NODE_EXPORTER="quay.io/prometheus/node-exporter:v1.9.0" +# generate-schemas-ndt7 is built from source rather than extracted: the +# ndt-server image builds it on Alpine with CGO (musl-dev), so the shipped copy +# is musl-linked. We rebuild the same pinned source with CGO on the glibc build +# host, yielding a glibc-dynamic binary (NEEDED: libc.so.6) that runs natively. +# NDT_GO_TOOLCHAIN must satisfy the `go` directive in that version's go.mod; the +# system go fetches it on demand (the bare "go 1.25" directive otherwise mis- +# resolves to a non-existent "go1.25" download). +NDT_SERVER_SRC="https://github.com/m-lab/ndt-server.git" +NDT_SERVER_VERSION="v0.25.2" +NDT_GO_TOOLCHAIN="go1.25.11" + # extraction requests: "image|search-names|dest-name" # search-names is a comma-separated list of candidate basenames to locate in the # image rootfs (first match wins), to tolerate differing install paths. REQUESTS=( "${IMG_NDT_SERVER}|ndt-server|ndt-server" - "${IMG_NDT_SERVER}|generate-schemas|generate-schemas-ndt7" "${IMG_HEARTBEAT}|heartbeat|heartbeat" "${IMG_UUID_ANNOTATOR}|uuid-annotator|uuid-annotator" "${IMG_UUID_ANNOTATOR_SCHEMA}|generate-schemas|generate-schemas-annotation2" @@ -49,7 +59,7 @@ REQUESTS=( "${IMG_NODE_EXPORTER}|node_exporter,node-exporter|node-exporter" ) -for tool in skopeo jq tar file; do +for tool in skopeo jq tar file go git; do command -v "${tool}" >/dev/null 2>&1 || { echo "ERROR: required build tool '${tool}' not found in PATH" >&2 exit 1 @@ -142,5 +152,21 @@ for req in "${REQUESTS[@]}"; do check_binary "${OUT_DIR}/${dest}" done +# Build generate-schemas-ndt7 from source (glibc-dynamic, CGO on), since the +# ndt-server image ships it musl-linked. cmd/generate-schemas is a nested Go +# module inside the ndt-server repo, so build from within that directory. +echo ">> building generate-schemas-ndt7 from ${NDT_SERVER_SRC}@${NDT_SERVER_VERSION}" +SRC_DIR="${WORK_DIR}/ndt-server" +git clone --quiet --depth 1 --branch "${NDT_SERVER_VERSION}" "${NDT_SERVER_SRC}" "${SRC_DIR}" +( + cd "${SRC_DIR}/cmd/generate-schemas" + CGO_ENABLED=1 \ + GOTOOLCHAIN="${NDT_GO_TOOLCHAIN}" \ + GOPATH="${WORK_DIR}/go" \ + GOCACHE="${WORK_DIR}/gocache" \ + go build -trimpath -o "${OUT_DIR}/generate-schemas-ndt7" . +) +check_binary "${OUT_DIR}/generate-schemas-ndt7" + echo echo "Staged $(ls -1 "${OUT_DIR}" | wc -l | tr -d ' ') binaries in ${OUT_DIR}" diff --git a/debian/control b/debian/control index f72a757..6d9fb81 100644 --- a/debian/control +++ b/debian/control @@ -7,6 +7,11 @@ Build-Depends: skopeo, jq, file, + golang-go, + git, + ca-certificates, + gcc, + libc6-dev, Standards-Version: 4.6.2 Homepage: https://github.com/m-lab/autonode Rules-Requires-Root: no @@ -18,7 +23,6 @@ Depends: ${misc:Depends}, curl, kmod, - musl, Description: M-Lab node (autonode) measurement stack Native systemd packaging of the M-Lab "autonode" Network Diagnostic Tool (NDT) measurement platform, replacing the former Docker Compose deployment. diff --git a/debian/rules b/debian/rules index 1c147e0..d2c316e 100755 --- a/debian/rules +++ b/debian/rules @@ -21,9 +21,3 @@ override_dh_installsystemd: # otherwise rewrite them. override_dh_strip: override_dh_dwz: - -# scamper and the static Go binaries are resolved normally. generate-schemas-ndt7 -# is the lone musl-linked binary; dpkg-shlibdeps has no shlibs info for musl, so -# exclude it here - its runtime requirement is declared as Depends: musl. -override_dh_shlibdeps: - dh_shlibdeps -Xgenerate-schemas-ndt7 From af6b2e699c062f9b7fea3cdb329d606576ba3d68 Mon Sep 17 00:00:00 2001 From: Roberto D'Auria Date: Tue, 30 Jun 2026 15:37:18 +0200 Subject: [PATCH 04/25] debian/rules: explicitly run dh_installsysusers On debhelper 13.14 (Ubuntu 24.04) dh_installsysusers is not in the default dh sequence, so the mlab-node system user was never created. Invoke it via execute_after_dh_install (before dh_installtmpfiles) so the user exists before the tmpfiles dirs are chowned to it. --- debian/rules | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/debian/rules b/debian/rules index d2c316e..d284aca 100755 --- a/debian/rules +++ b/debian/rules @@ -3,6 +3,14 @@ %: dh $@ +# dh_installsysusers is not part of the default dh sequence on debhelper 13.14 +# (unlike dh_installtmpfiles), so invoke it explicitly to create the mlab-node +# system user from debian/mlab-node.sysusers. Running it here (after dh_install, +# before dh_installtmpfiles) orders the postinst so the user exists before the +# tmpfiles directories are chowned to it. +execute_after_dh_install: + dh_installsysusers + # Extract the component binaries from the pinned container images (via skopeo, # no Docker daemon) unless they are already staged. This runs as the package # "build" step; the binaries are then installed via debian/mlab-node.install. From 1c6b2d7f17e532a6649faab77652d857e079dc63 Mon Sep 17 00:00:00 2001 From: Roberto D'Auria Date: Tue, 30 Jun 2026 16:27:10 +0200 Subject: [PATCH 05/25] uuid-annotator: ship bundled IPInfo asnames CSV and pass -asname.url The uuid-annotator image provides the AS-names data via 'ENV ASNAME_URL file:///data/asnames.ipinfo.csv', which the binary picks up through flagx.ArgsFromEnv. systemd does not inherit image ENVs, so without it the binary panics (asname.url is required). Extract /data/asnames.ipinfo.csv from the pinned image, install it to /usr/share/mlab, and pass -asname.url=file:// to it from the unit. Also add AF_NETLINK to the uuid-annotator sandbox (needed to enumerate local addresses). --- .build/fetch-binaries.sh | 13 +++++++++++++ debian/mlab-node.install | 4 ++++ systemd/mlab-node-uuid-annotator.service | 5 ++++- 3 files changed, 21 insertions(+), 1 deletion(-) diff --git a/.build/fetch-binaries.sh b/.build/fetch-binaries.sh index 67f6040..c4655f3 100755 --- a/.build/fetch-binaries.sh +++ b/.build/fetch-binaries.sh @@ -152,6 +152,19 @@ for req in "${REQUESTS[@]}"; do check_binary "${OUT_DIR}/${dest}" done +# Ship the IPInfo AS-names CSV bundled in the uuid-annotator image. The image +# provides it via `ENV ASNAME_URL file:///data/asnames.ipinfo.csv`, which the +# binary reads through flagx.ArgsFromEnv (asname.url <- ASNAME_URL). systemd does +# not inherit image ENVs, so we ship the same file and pass it via -asname.url. +ua_rootfs="${WORK_DIR}/rootfs/$(echo "${IMG_UUID_ANNOTATOR}" | tr '/:' '__')" +asname_src="$(find "${ua_rootfs}" -type f -name 'asnames.ipinfo.csv' 2>/dev/null | head -n1)" +if [ -z "${asname_src}" ]; then + echo "ERROR: asnames.ipinfo.csv not found in ${IMG_UUID_ANNOTATOR}" >&2 + exit 1 +fi +install -D -m 0644 "${asname_src}" "${OUT_DIR}/asnames.ipinfo.csv" +echo " asnames.ipinfo.csv: $(wc -c <"${OUT_DIR}/asnames.ipinfo.csv" | tr -d ' ') bytes" + # Build generate-schemas-ndt7 from source (glibc-dynamic, CGO on), since the # ndt-server image ships it musl-linked. cmd/generate-schemas is a nested Go # module inside the ndt-server repo, so build from within that directory. diff --git a/debian/mlab-node.install b/debian/mlab-node.install index 68a06bc..3964942 100644 --- a/debian/mlab-node.install +++ b/debian/mlab-node.install @@ -45,3 +45,7 @@ config/sysctl.d/90-mlab-node.conf usr/lib/sysctl.d # Static content served by ndt-server. html/index.html usr/share/mlab/html + +# IPInfo AS-names CSV (bundled from the uuid-annotator image), read by the +# uuid-annotator via -asname.url. +binaries/asnames.ipinfo.csv usr/share/mlab diff --git a/systemd/mlab-node-uuid-annotator.service b/systemd/mlab-node-uuid-annotator.service index ae5d621..1e827bc 100644 --- a/systemd/mlab-node-uuid-annotator.service +++ b/systemd/mlab-node-uuid-annotator.service @@ -26,6 +26,7 @@ ExecStart=/usr/lib/mlab/uuid-annotator \ -routeview-v4.url=gs://downloader-${PROJECT}/RouteViewIPv4/current/routeview.pfx2as.gz \ -routeview-v6.url=gs://downloader-${PROJECT}/RouteViewIPv6/current/routeview.pfx2as.gz \ -siteinfo.url=file:///var/lib/mlab/node/annotation.json \ + -asname.url=file:///usr/share/mlab/asnames.ipinfo.csv \ -hostname=@/var/lib/mlab/node/hostname Restart=always RestartSec=5 @@ -53,6 +54,8 @@ LockPersonality=yes MemoryDenyWriteExecute=yes RemoveIPC=yes CapabilityBoundingSet= -RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX +# AF_NETLINK is required to enumerate the host's local addresses (the annotator +# reads them via netlink to annotate the server side of each measurement). +RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK SystemCallFilter=@system-service SystemCallErrorNumber=EPERM From 369524247bfe8f47f8d7c353bb56c911fe9e2e1d Mon Sep 17 00:00:00 2001 From: Roberto D'Auria Date: Tue, 30 Jun 2026 16:41:34 +0200 Subject: [PATCH 06/25] traceroute-caller: point -scamper.bin at the shipped scamper The binary defaults -scamper.bin to /usr/local/bin/scamper; ours is at /usr/lib/mlab/scamper. Set the flag explicitly and drop the PATH override (traceroute-caller uses the absolute flag, not a PATH lookup). --- systemd/mlab-node-traceroute-caller.service | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/systemd/mlab-node-traceroute-caller.service b/systemd/mlab-node-traceroute-caller.service index cff2135..8dff50b 100644 --- a/systemd/mlab-node-traceroute-caller.service +++ b/systemd/mlab-node-traceroute-caller.service @@ -13,11 +13,10 @@ BindsTo=mlab-node-register.service [Service] Type=exec EnvironmentFile=/etc/mlab/mlab-node.env -# scamper is shipped alongside traceroute-caller; make it discoverable on PATH. -Environment=PATH=/usr/lib/mlab:/usr/sbin:/usr/bin:/sbin:/bin ExecStartPre=/usr/lib/mlab/wait-socket /run/mlab/sockets/events.sock ExecStartPre=/usr/lib/mlab/wait-socket /run/mlab/sockets/uuid-annotator.sock ExecStart=/usr/lib/mlab/traceroute-caller \ + -scamper.bin=/usr/lib/mlab/scamper \ -prometheusx.listen-address=:9994 \ -traceroute-output=/var/lib/mlab/resultsdir/ndt/scamper2 \ -hopannotation-output=/var/lib/mlab/resultsdir/ndt/hopannotation2 \ From d1056af7199401858bfb82dd9d61345da875b911 Mon Sep 17 00:00:00 2001 From: Roberto D'Auria Date: Tue, 7 Jul 2026 15:26:14 +0200 Subject: [PATCH 07/25] build-binaries: build components from source at pinned tags, drop image extraction Replace fetch-binaries.sh (skopeo-based extraction from the released container images) with build-binaries.sh, which shallow-clones each component repository at the same pinned release tags and compiles from source. This gives verifiable provenance (git tag + go.sum + the Go checksum database), removes the skopeo/jq machinery and all musl special-casing, and produces glibc-compatible binaries by construction. Every Go component builds with CGO_ENABLED=0 (static) except ndt-server and its schema generator, whose bbr package requires cgo on Linux; those two are glibc-dynamic. Version ldflags mirror each upstream image build so Prometheus build-info metrics keep reporting versions. scamper is built from the snapshot tarball vendored by traceroute-caller, with --disable-shared so it ships self-contained. heartbeat builds from m-lab/locate (cmd/heartbeat), where it actually lives. --- .build/build-binaries.sh | 196 +++++++++++++++++++++++++++++++++++++++ .build/fetch-binaries.sh | 185 ------------------------------------ .gitignore | 4 +- README.md | 16 ++-- debian/changelog | 4 +- debian/control | 10 +- debian/mlab-node.install | 4 +- debian/rules | 12 +-- 8 files changed, 222 insertions(+), 209 deletions(-) create mode 100755 .build/build-binaries.sh delete mode 100755 .build/fetch-binaries.sh diff --git a/.build/build-binaries.sh b/.build/build-binaries.sh new file mode 100755 index 0000000..3ed111a --- /dev/null +++ b/.build/build-binaries.sh @@ -0,0 +1,196 @@ +#!/bin/bash +# +# build-binaries.sh builds the component binaries from source at the upstream +# release tags pinned below and stages them under ./binaries/ so they can be +# shipped in the mlab-node Debian package. Building from source (rather than +# extracting binaries from the released container images) gives verifiable +# provenance — git tag + go.sum + the Go checksum database — and produces +# glibc-compatible binaries by construction, with no Docker or image tooling +# involved at any point. +# +# Required host tools (build-time only): git, go, gcc, make, file. +# The build host must be linux/amd64 (the package is Architecture: amd64, and +# the vendored scamper C code is compiled natively). +# +# Every Go component builds with CGO_ENABLED=0 (static) except ndt-server and +# its schema generator: ndt-server's bbr package requires cgo on Linux, so +# those two build with CGO_ENABLED=1 and come out glibc-dynamic (fine on the +# Debian target; dh_shlibdeps declares the libc dependency). The Go toolchain +# is pinned (GOTOOLCHAIN below) so builds do not depend on the build host's go +# version; the pinned toolchain is fetched on demand by the system go. Version +# ldflags mirror what each upstream image build stamps, so Prometheus +# build-info metrics keep reporting versions. +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)" +OUT_DIR="${REPO_DIR}/binaries" + +# Component versions. These mirror the image tags pinned in the historical +# docker-compose deployment, so the package builds the exact sources M-Lab has +# tested. +# NOTE: heartbeat lives in the m-lab/locate repository (cmd/heartbeat); the +# measurementlab/heartbeat image tags track locate's tags. +# NOTE: the annotation2 schema generator intentionally comes from an older +# uuid-annotator tag (v0.5.8) than the running annotator (v0.5.10), matching +# the compose configuration. +VER_NDT_SERVER="v0.25.2" +VER_HEARTBEAT="v0.19.1" +VER_UUID_ANNOTATOR="v0.5.10" +VER_UUID_ANNOTATOR_SCHEMA="v0.5.8" +VER_JOSTLER="v1.1.4" +VER_TRACEROUTE="v0.12.0" +VER_AUTOJOIN="v0.2.13" +VER_NODE_EXPORTER="v1.9.0" + +# scamper is C, not Go; traceroute-caller vendors the exact snapshot it is +# tested against, so we build that tarball rather than Debian's scamper. +SCAMPER_SNAPSHOT="scamper-cvs-20230302" + +# Pinned Go toolchain used for every component. Newer toolchains build older +# modules fine (Go 1 compatibility); pinning one version keeps builds +# deterministic across build hosts. It must also satisfy the highest `go` +# directive among the pinned sources (ndt-server's bare "go 1.25" otherwise +# mis-resolves to a non-existent "go1.25" toolchain download on older hosts). +GO_TOOLCHAIN="go1.25.11" + +for tool in git go gcc make file; do + command -v "${tool}" >/dev/null 2>&1 || { + echo "ERROR: required build tool '${tool}' not found in PATH" >&2 + exit 1 + } +done + +if [ "$(uname -sm)" != "Linux x86_64" ]; then + echo "ERROR: this script must run on a linux/amd64 build host (got: $(uname -sm))" >&2 + exit 1 +fi + +WORK_DIR="$(mktemp -d)" +trap 'rm -rf "${WORK_DIR}"' EXIT + +# Hermetic Go state: do not touch (or depend on) the build host's module +# cache. CGO is disabled by default (overridden per-build where required); +# -trimpath aids reproducibility. +export CGO_ENABLED=0 +export GOTOOLCHAIN="${GO_TOOLCHAIN}" +export GOPATH="${WORK_DIR}/gopath" +export GOCACHE="${WORK_DIR}/gocache" +export GOFLAGS="-trimpath" + +# clone_at_tag URL TAG DEST — shallow clone of a repo at a release tag. +# The clone (not a tarball download) provides the commit hash for version +# stamping and, for traceroute-caller, the vendored scamper tarball. +clone_at_tag() { + local url="$1" tag="$2" dest="$3" + [ -d "${dest}" ] && return 0 + echo ">> cloning ${url}@${tag}" + git clone --quiet --depth 1 --branch "${tag}" "${url}" "${dest}" +} + +# short_commit DIR — abbreviated hash of the checked-out commit, as upstream +# image builds pass to -X github.com/m-lab/go/prometheusx.GitShortCommit. +short_commit() { + git -C "$1" log -1 --format=%h +} + +# build_go SRC_DIR REL_PKG_DIR OUT_NAME [LDFLAGS] +# Builds the main package at SRC_DIR/REL_PKG_DIR into OUT_DIR/OUT_NAME. +# Building from within the package directory transparently handles both +# regular packages and nested Go modules (ndt-server's cmd/generate-schemas). +# Respects CGO_ENABLED from the environment (0 by default, see above). +build_go() { + local src="$1" rel="$2" out="$3" ldflags="${4:-}" + echo ">> building ${out} from ${src##*/}/${rel#.} (CGO_ENABLED=${CGO_ENABLED})" + ( + cd "${src}/${rel}" + go build ${ldflags:+-ldflags "${ldflags}"} -o "${OUT_DIR}/${out}" . + ) + check_binary "${OUT_DIR}/${out}" +} + +# check_binary PATH — the Go binaries must be static ELF executables; scamper +# is glibc-dynamic (dh_shlibdeps resolves its shared-lib deps). Anything that +# is not ELF is fatal. +check_binary() { + local bin="$1" info + info="$(file -L "${bin}")" + case "${info}" in + *"ELF"*) ;; + *) + echo "ERROR: ${bin} does not look like an ELF executable: ${info}" >&2 + exit 1 + ;; + esac + echo " ${bin##*/}: ${info#*: }" +} + +rm -rf "${OUT_DIR}" +mkdir -p "${OUT_DIR}" + +SRC="${WORK_DIR}/src" +clone_at_tag https://github.com/m-lab/ndt-server.git "${VER_NDT_SERVER}" "${SRC}/ndt-server" +clone_at_tag https://github.com/m-lab/locate.git "${VER_HEARTBEAT}" "${SRC}/locate" +clone_at_tag https://github.com/m-lab/uuid-annotator.git "${VER_UUID_ANNOTATOR}" "${SRC}/uuid-annotator" +clone_at_tag https://github.com/m-lab/uuid-annotator.git "${VER_UUID_ANNOTATOR_SCHEMA}" "${SRC}/uuid-annotator-schema" +clone_at_tag https://github.com/m-lab/jostler.git "${VER_JOSTLER}" "${SRC}/jostler" +clone_at_tag https://github.com/m-lab/traceroute-caller.git "${VER_TRACEROUTE}" "${SRC}/traceroute-caller" +clone_at_tag https://github.com/m-lab/autojoin.git "${VER_AUTOJOIN}" "${SRC}/autojoin" +clone_at_tag https://github.com/prometheus/node_exporter.git "${VER_NODE_EXPORTER}" "${SRC}/node_exporter" + +# Per-component ldflags replicate each upstream image build (see the +# Dockerfiles at the pinned tags); without them the binaries report empty +# versions in logs and Prometheus build-info metrics. +# +# ndt-server (and its schema generator, which imports the same packages) +# requires cgo for the bbr package: with CGO_ENABLED=0 the build fails with +# "undefined: enableBBR". +CGO_ENABLED=1 build_go "${SRC}/ndt-server" . ndt-server \ + "-X github.com/m-lab/ndt-server/version.Version=${VER_NDT_SERVER} -X github.com/m-lab/go/prometheusx.GitShortCommit=$(short_commit "${SRC}/ndt-server")" +CGO_ENABLED=1 build_go "${SRC}/ndt-server" cmd/generate-schemas generate-schemas-ndt7 + +build_go "${SRC}/locate" cmd/heartbeat heartbeat \ + "-X github.com/m-lab/go/prometheusx.GitShortCommit=$(short_commit "${SRC}/locate")" + +build_go "${SRC}/uuid-annotator" . uuid-annotator \ + "-X github.com/m-lab/go/prometheusx.GitShortCommit=$(short_commit "${SRC}/uuid-annotator")" +build_go "${SRC}/uuid-annotator-schema" cmd/generate-schemas generate-schemas-annotation2 + +build_go "${SRC}/jostler" cmd/jostler jostler \ + "-X github.com/m-lab/go/prometheusx.GitShortCommit=$(short_commit "${SRC}/jostler") -X main.Version=${VER_JOSTLER} -X main.GitCommit=$(git -C "${SRC}/jostler" log -1 --format=%H)" + +build_go "${SRC}/traceroute-caller" . traceroute-caller \ + "-X github.com/m-lab/go/prometheusx.GitShortCommit=$(short_commit "${SRC}/traceroute-caller")" +build_go "${SRC}/traceroute-caller" cmd/generate-schemas generate-schemas-traceroute + +build_go "${SRC}/autojoin" cmd/register autojoin-register \ + "-s -w -X main.Version=${VER_AUTOJOIN}" + +build_go "${SRC}/node_exporter" . node-exporter \ + "-X github.com/prometheus/common/version.Version=${VER_NODE_EXPORTER#v} -X github.com/prometheus/common/version.Revision=$(git -C "${SRC}/node_exporter" log -1 --format=%H) -X github.com/prometheus/common/version.Branch=HEAD" + +# Build scamper from the snapshot vendored in traceroute-caller. The tarball +# ships a pre-generated ./configure (no autotools needed); --disable-shared +# links scamper's internal libraries statically so we ship a single +# self-contained binary (only libc is dynamic). +echo ">> building scamper from vendored ${SCAMPER_SNAPSHOT}" +tar -xzf "${SRC}/traceroute-caller/third_party/scamper/${SCAMPER_SNAPSHOT}.tar.gz" -C "${WORK_DIR}" +( + cd "${WORK_DIR}/${SCAMPER_SNAPSHOT}" + chmod +x ./configure + ./configure --disable-shared --prefix="${WORK_DIR}/scamper-prefix" >/dev/null + make -j"$(nproc)" >/dev/null + make install >/dev/null +) +install -m 0755 "${WORK_DIR}/scamper-prefix/bin/scamper" "${OUT_DIR}/scamper" +check_binary "${OUT_DIR}/scamper" + +# Ship the IPInfo AS-names CSV from the uuid-annotator repo. The container +# image passes it to the binary via `ENV ASNAME_URL file:///data/...`; systemd +# does not inherit image ENVs, so we ship the same file and pass it via +# -asname.url. +install -m 0644 "${SRC}/uuid-annotator/data/asnames.ipinfo.csv" "${OUT_DIR}/asnames.ipinfo.csv" +echo " asnames.ipinfo.csv: $(wc -c <"${OUT_DIR}/asnames.ipinfo.csv" | tr -d ' ') bytes" + +echo +echo "Staged $(ls -1 "${OUT_DIR}" | wc -l | tr -d ' ') artifacts in ${OUT_DIR}" diff --git a/.build/fetch-binaries.sh b/.build/fetch-binaries.sh deleted file mode 100755 index c4655f3..0000000 --- a/.build/fetch-binaries.sh +++ /dev/null @@ -1,185 +0,0 @@ -#!/bin/bash -# -# fetch-binaries.sh extracts the component binaries from the pinned M-Lab -# container images and stages them under ./binaries/ so they can be shipped in -# the mlab-node Debian package. It uses skopeo to pull the image filesystems -# without a running Docker daemon: no Docker is required to build the package, -# and none is required at runtime. -# -# Required host tools (build-time only): skopeo, jq, tar, file. -# -# The binaries are version-pinned to the same image tags used by the historical -# docker-compose deployment, so the package ships the exact binaries M-Lab has -# tested. Each extracted ELF is checked: a statically linked or glibc-linked -# binary is fine on Debian; a musl-linked (Alpine) binary is NOT and aborts the -# build, because it would fail to run on the target. -set -euo pipefail - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -REPO_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)" -OUT_DIR="${REPO_DIR}/binaries" - -# Image versions. These mirror the tags pinned in the original docker-compose.yml. -# NOTE: the annotation2 schema generator intentionally comes from an older -# uuid-annotator image (v0.5.8) than the running annotator (v0.5.10), matching -# the compose configuration. -IMG_NDT_SERVER="measurementlab/ndt-server:v0.25.2" -IMG_HEARTBEAT="measurementlab/heartbeat:v0.19.1" -IMG_UUID_ANNOTATOR="measurementlab/uuid-annotator:v0.5.10" -IMG_UUID_ANNOTATOR_SCHEMA="measurementlab/uuid-annotator:v0.5.8" -IMG_JOSTLER="measurementlab/jostler:v1.1.4" -IMG_TRACEROUTE="measurementlab/traceroute-caller:v0.12.0" -IMG_REGISTER="measurementlab/autojoin-register:v0.2.13" -IMG_NODE_EXPORTER="quay.io/prometheus/node-exporter:v1.9.0" - -# generate-schemas-ndt7 is built from source rather than extracted: the -# ndt-server image builds it on Alpine with CGO (musl-dev), so the shipped copy -# is musl-linked. We rebuild the same pinned source with CGO on the glibc build -# host, yielding a glibc-dynamic binary (NEEDED: libc.so.6) that runs natively. -# NDT_GO_TOOLCHAIN must satisfy the `go` directive in that version's go.mod; the -# system go fetches it on demand (the bare "go 1.25" directive otherwise mis- -# resolves to a non-existent "go1.25" download). -NDT_SERVER_SRC="https://github.com/m-lab/ndt-server.git" -NDT_SERVER_VERSION="v0.25.2" -NDT_GO_TOOLCHAIN="go1.25.11" - -# extraction requests: "image|search-names|dest-name" -# search-names is a comma-separated list of candidate basenames to locate in the -# image rootfs (first match wins), to tolerate differing install paths. -REQUESTS=( - "${IMG_NDT_SERVER}|ndt-server|ndt-server" - "${IMG_HEARTBEAT}|heartbeat|heartbeat" - "${IMG_UUID_ANNOTATOR}|uuid-annotator|uuid-annotator" - "${IMG_UUID_ANNOTATOR_SCHEMA}|generate-schemas|generate-schemas-annotation2" - "${IMG_JOSTLER}|jostler|jostler" - "${IMG_TRACEROUTE}|traceroute-caller|traceroute-caller" - "${IMG_TRACEROUTE}|generate-schemas|generate-schemas-traceroute" - "${IMG_TRACEROUTE}|scamper|scamper" - "${IMG_REGISTER}|register,autojoin-register|autojoin-register" - "${IMG_NODE_EXPORTER}|node_exporter,node-exporter|node-exporter" -) - -for tool in skopeo jq tar file go git; do - command -v "${tool}" >/dev/null 2>&1 || { - echo "ERROR: required build tool '${tool}' not found in PATH" >&2 - exit 1 - } -done - -WORK_DIR="$(mktemp -d)" -# Image layers can contain root-owned, mode-000 entries (e.g. var/empty); make -# the tree writable before removing it so cleanup never fails the script. -trap 'chmod -R u+rwX "${WORK_DIR}" 2>/dev/null || true; rm -rf "${WORK_DIR}" 2>/dev/null || true' EXIT - -# unpack_image IMAGE DEST_ROOTFS -# Copies the image to a local OCI/dir layout with skopeo and extracts every -# layer (in manifest order) into DEST_ROOTFS to reconstruct the image rootfs. -declare -A UNPACKED=() -unpack_image() { - local image="$1" rootfs="$2" - local imgdir="${WORK_DIR}/img/$(echo "${image}" | tr '/:' '__')" - - if [ -n "${UNPACKED[${image}]:-}" ]; then - return 0 - fi - - echo ">> pulling ${image}" - mkdir -p "${imgdir}" "${rootfs}" - # Force a single linux/amd64 instance so the dir: manifest is an image - # manifest (with .layers) even when the source is a multi-arch index. - skopeo --override-os linux --override-arch amd64 \ - copy --quiet "docker://${image}" "dir:${imgdir}" - - # Layer blobs are listed (in order) in the manifest; each is a gzipped tar. - local layer - while read -r layer; do - local blob="${imgdir}/${layer#sha256:}" - [ -f "${blob}" ] || { echo "ERROR: missing layer blob ${blob}" >&2; exit 1; } - # Extract over the rootfs; ignore whiteout/permission quirks (we only need - # to read out a handful of executables afterwards). - tar -xf "${blob}" -C "${rootfs}" 2>/dev/null || true - done < <(jq -r '.layers[].digest' "${imgdir}/manifest.json") - - UNPACKED[${image}]=1 -} - -# check_binary PATH — classify the ELF. glibc-dynamic and static binaries run on -# Debian as-is (dh_shlibdeps resolves shared-lib deps). musl-linked binaries run -# only with the `musl` package installed, so they are flagged (the package -# declares the dependency) rather than rejected. A non-ELF file is fatal. -check_binary() { - local bin="$1" info - info="$(file -L "${bin}")" - case "${info}" in - *"statically linked"*) ;; # portable, OK - *"dynamically linked"*) - if echo "${info}" | grep -q "ld-musl"; then - echo " WARNING: ${bin##*/} is musl-linked; requires the 'musl' package at runtime." >&2 - fi - ;; # glibc dynamic, OK (dh_shlibdeps handles deps) - *"ELF"*) ;; - *) - echo "ERROR: ${bin} does not look like an executable: ${info}" >&2 - exit 1 - ;; - esac - echo " ${bin##*/}: ${info#*: }" -} - -rm -rf "${OUT_DIR}" -mkdir -p "${OUT_DIR}" - -for req in "${REQUESTS[@]}"; do - IFS='|' read -r image names dest <<<"${req}" - rootfs="${WORK_DIR}/rootfs/$(echo "${image}" | tr '/:' '__')" - unpack_image "${image}" "${rootfs}" - - found="" - IFS=',' read -ra candidates <<<"${names}" - for name in "${candidates[@]}"; do - # Prefer an executable regular file; search the whole rootfs. - found="$(find "${rootfs}" -type f -name "${name}" -perm -u+x 2>/dev/null | head -n1 || true)" - [ -z "${found}" ] && found="$(find "${rootfs}" -type f -name "${name}" 2>/dev/null | head -n1 || true)" - [ -n "${found}" ] && break - done - - if [ -z "${found}" ]; then - echo "ERROR: could not find any of [${names}] in ${image}" >&2 - exit 1 - fi - - install -D -m 0755 "${found}" "${OUT_DIR}/${dest}" - check_binary "${OUT_DIR}/${dest}" -done - -# Ship the IPInfo AS-names CSV bundled in the uuid-annotator image. The image -# provides it via `ENV ASNAME_URL file:///data/asnames.ipinfo.csv`, which the -# binary reads through flagx.ArgsFromEnv (asname.url <- ASNAME_URL). systemd does -# not inherit image ENVs, so we ship the same file and pass it via -asname.url. -ua_rootfs="${WORK_DIR}/rootfs/$(echo "${IMG_UUID_ANNOTATOR}" | tr '/:' '__')" -asname_src="$(find "${ua_rootfs}" -type f -name 'asnames.ipinfo.csv' 2>/dev/null | head -n1)" -if [ -z "${asname_src}" ]; then - echo "ERROR: asnames.ipinfo.csv not found in ${IMG_UUID_ANNOTATOR}" >&2 - exit 1 -fi -install -D -m 0644 "${asname_src}" "${OUT_DIR}/asnames.ipinfo.csv" -echo " asnames.ipinfo.csv: $(wc -c <"${OUT_DIR}/asnames.ipinfo.csv" | tr -d ' ') bytes" - -# Build generate-schemas-ndt7 from source (glibc-dynamic, CGO on), since the -# ndt-server image ships it musl-linked. cmd/generate-schemas is a nested Go -# module inside the ndt-server repo, so build from within that directory. -echo ">> building generate-schemas-ndt7 from ${NDT_SERVER_SRC}@${NDT_SERVER_VERSION}" -SRC_DIR="${WORK_DIR}/ndt-server" -git clone --quiet --depth 1 --branch "${NDT_SERVER_VERSION}" "${NDT_SERVER_SRC}" "${SRC_DIR}" -( - cd "${SRC_DIR}/cmd/generate-schemas" - CGO_ENABLED=1 \ - GOTOOLCHAIN="${NDT_GO_TOOLCHAIN}" \ - GOPATH="${WORK_DIR}/go" \ - GOCACHE="${WORK_DIR}/gocache" \ - go build -trimpath -o "${OUT_DIR}/generate-schemas-ndt7" . -) -check_binary "${OUT_DIR}/generate-schemas-ndt7" - -echo -echo "Staged $(ls -1 "${OUT_DIR}" | wc -l | tr -d ' ') binaries in ${OUT_DIR}" diff --git a/.gitignore b/.gitignore index 9408389..0ac758f 100644 --- a/.gitignore +++ b/.gitignore @@ -1,7 +1,7 @@ .env -# Binaries extracted from container images at build time (see -# .build/fetch-binaries.sh). Not checked in. +# Binaries built from source at build time (see +# .build/build-binaries.sh). Not checked in. /binaries/ # Debian build artifacts. diff --git a/README.md b/README.md index 1e24fc1..8c4dab9 100644 --- a/README.md +++ b/README.md @@ -28,8 +28,9 @@ metadata. The whole stack is controlled atomically through `mlab-node.target`; `register-node` is the keystone — if it goes down, the entire stack is brought down with it. -The component binaries are extracted from M-Lab's pinned container images at -build time (see `.build/fetch-binaries.sh`). +The component binaries are built from source at build time, at the same +upstream release tags the pinned M-Lab container images were built from (see +`.build/build-binaries.sh`). ## File layout @@ -48,16 +49,17 @@ build time (see `.build/fetch-binaries.sh`). ## Building the package -Requires `dpkg-buildpackage`, debhelper, and (for binary extraction) -`skopeo`, `jq`, and `file`: +Requires `dpkg-buildpackage`, debhelper, `git`, `go`, and a C toolchain +(`gcc`, `make`) for scamper: ```sh dpkg-buildpackage -us -uc -b ``` -The build pulls the pinned component binaries from the M-Lab images via skopeo -(no Docker daemon) and stages them under `binaries/`. The build fails if any -extracted binary is musl-linked (it would not run on Debian). +The build clones each component repository at its pinned release tag, compiles +the Go binaries with `CGO_ENABLED=0` (static, glibc-independent) using a pinned +Go toolchain, builds the scamper snapshot vendored by traceroute-caller, and +stages everything under `binaries/`. ## Installing and configuring diff --git a/debian/changelog b/debian/changelog index 5f356bd..d06f686 100644 --- a/debian/changelog +++ b/debian/changelog @@ -2,7 +2,7 @@ mlab-node (1.0.0) unstable; urgency=medium * Initial release: native Debian packaging of the M-Lab autonode stack. Replaces the Docker Compose deployment with hardened systemd services. - Component binaries are extracted from the pinned M-Lab container images at - build time; no Docker is required at build or run time. + Component binaries are built from source at the pinned upstream release + tags at build time; no Docker is required at build or run time. -- Measurement Lab Mon, 29 Jun 2026 00:00:00 +0000 diff --git a/debian/control b/debian/control index 6d9fb81..8d97d39 100644 --- a/debian/control +++ b/debian/control @@ -4,13 +4,12 @@ Priority: optional Maintainer: Measurement Lab Build-Depends: debhelper-compat (= 13), - skopeo, - jq, file, - golang-go, + golang-go (>= 2:1.21~), git, ca-certificates, gcc, + make, libc6-dev, Standards-Version: 4.6.2 Homepage: https://github.com/m-lab/autonode @@ -32,5 +31,6 @@ Description: M-Lab node (autonode) measurement stack node exporter - as hardened systemd services confined with systemd's own sandboxing directives. No Docker is required at build or run time. . - The component binaries are extracted from M-Lab's pinned container images at - build time. The whole stack is controlled atomically through mlab-node.target. + The component binaries are built from source at build time, at the same + upstream release tags the pinned M-Lab container images were built from. The + whole stack is controlled atomically through mlab-node.target. diff --git a/debian/mlab-node.install b/debian/mlab-node.install index 3964942..c7816b0 100644 --- a/debian/mlab-node.install +++ b/debian/mlab-node.install @@ -1,4 +1,4 @@ -# Component binaries (extracted from M-Lab images by .build/fetch-binaries.sh). +# Component binaries (built from source by .build/build-binaries.sh). binaries/ndt-server usr/lib/mlab binaries/heartbeat usr/lib/mlab binaries/uuid-annotator usr/lib/mlab @@ -46,6 +46,6 @@ config/sysctl.d/90-mlab-node.conf usr/lib/sysctl.d # Static content served by ndt-server. html/index.html usr/share/mlab/html -# IPInfo AS-names CSV (bundled from the uuid-annotator image), read by the +# IPInfo AS-names CSV (from the uuid-annotator repo), read by the # uuid-annotator via -asname.url. binaries/asnames.ipinfo.csv usr/share/mlab diff --git a/debian/rules b/debian/rules index d284aca..583c49c 100755 --- a/debian/rules +++ b/debian/rules @@ -11,11 +11,11 @@ execute_after_dh_install: dh_installsysusers -# Extract the component binaries from the pinned container images (via skopeo, -# no Docker daemon) unless they are already staged. This runs as the package -# "build" step; the binaries are then installed via debian/mlab-node.install. +# Build the component binaries from source at the pinned upstream release +# tags unless they are already staged. This runs as the package "build" step; +# the binaries are then installed via debian/mlab-node.install. override_dh_auto_build: - if [ ! -e binaries/ndt-server ]; then ./.build/fetch-binaries.sh; fi + if [ ! -e binaries/ndt-server ]; then ./.build/build-binaries.sh; fi # Install the target so it is enabled (auto-start on boot) but not started at # install time - the operator must configure /etc/mlab/*.env first. The other @@ -25,7 +25,7 @@ override_dh_installsystemd: dh_installsystemd --no-start --no-enable dh_installsystemd --no-start --name=mlab-node mlab-node.target -# The shipped binaries are M-Lab's released, tested artifacts; do not strip or -# otherwise rewrite them. +# The shipped binaries are built from M-Lab's released, tested sources; do +# not strip or otherwise rewrite them. override_dh_strip: override_dh_dwz: From 6faf4b5de35b1eb6ccf440c24a5bb30541a589cb Mon Sep 17 00:00:00 2001 From: Roberto D'Auria Date: Tue, 7 Jul 2026 15:26:45 +0200 Subject: [PATCH 08/25] ci: build the package natively on amd64 and arm64 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add a GitHub Actions workflow that builds the mlab-node package on ubuntu-24.04 (amd64) and ubuntu-24.04-arm (arm64) runners — both native, no QEMU. Build dependencies come from debian/control via mk-build-deps so the workflow cannot drift from the package metadata. Packages are uploaded as workflow artifacts on every run and attached to the GitHub Release on v* tag pushes. Extend the package to Architecture: amd64 arm64 and relax the build-binaries.sh host check accordingly; all components compile natively for the host architecture. --- .build/build-binaries.sh | 16 ++++--- .github/workflows/build-package.yml | 73 +++++++++++++++++++++++++++++ debian/control | 2 +- 3 files changed, 84 insertions(+), 7 deletions(-) create mode 100644 .github/workflows/build-package.yml diff --git a/.build/build-binaries.sh b/.build/build-binaries.sh index 3ed111a..07b2c22 100755 --- a/.build/build-binaries.sh +++ b/.build/build-binaries.sh @@ -9,8 +9,9 @@ # involved at any point. # # Required host tools (build-time only): git, go, gcc, make, file. -# The build host must be linux/amd64 (the package is Architecture: amd64, and -# the vendored scamper C code is compiled natively). +# The build host must be linux on amd64 or arm64; the binaries (Go and the +# vendored scamper C code) are compiled natively for the host architecture, +# which is also the architecture of the resulting package. # # Every Go component builds with CGO_ENABLED=0 (static) except ndt-server and # its schema generator: ndt-server's bbr package requires cgo on Linux, so @@ -61,10 +62,13 @@ for tool in git go gcc make file; do } done -if [ "$(uname -sm)" != "Linux x86_64" ]; then - echo "ERROR: this script must run on a linux/amd64 build host (got: $(uname -sm))" >&2 - exit 1 -fi +case "$(uname -sm)" in + "Linux x86_64"|"Linux aarch64") ;; + *) + echo "ERROR: this script must run on a linux amd64/arm64 build host (got: $(uname -sm))" >&2 + exit 1 + ;; +esac WORK_DIR="$(mktemp -d)" trap 'rm -rf "${WORK_DIR}"' EXIT diff --git a/.github/workflows/build-package.yml b/.github/workflows/build-package.yml new file mode 100644 index 0000000..db0cae6 --- /dev/null +++ b/.github/workflows/build-package.yml @@ -0,0 +1,73 @@ +# Builds the mlab-node Debian package natively on amd64 and arm64. +# +# arm64 uses GitHub's native ubuntu-24.04-arm runners (free for public +# repositories) — no QEMU emulation, so the Go and scamper builds run at +# native speed on both architectures. +# +# Every push/PR build uploads the .deb as a workflow artifact; pushing a +# version tag (v*) additionally attaches the packages to the GitHub Release +# for that tag. +name: build-package + +on: + push: + branches: [main] + tags: ['v*'] + pull_request: + workflow_dispatch: + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +permissions: + contents: write # needed to attach .debs to releases on tag pushes + +jobs: + build: + name: build (${{ matrix.arch }}) + strategy: + fail-fast: false + matrix: + include: + - runner: ubuntu-24.04 + arch: amd64 + - runner: ubuntu-24.04-arm + arch: arm64 + runs-on: ${{ matrix.runner }} + steps: + - uses: actions/checkout@v4 + + # mk-build-deps installs exactly what debian/control declares in + # Build-Depends, so the workflow never drifts from the package metadata. + - name: Install build dependencies + run: | + sudo apt-get update + sudo apt-get install -y --no-install-recommends devscripts equivs + sudo mk-build-deps --install --remove \ + --tool 'apt-get -y --no-install-recommends' debian/control + + - name: Build package + run: dpkg-buildpackage -us -uc -b + + # dpkg-buildpackage writes its output to the parent directory. + - name: Collect artifacts + run: | + mkdir -p dist + mv ../mlab-node_*.deb ../mlab-node_*.buildinfo ../mlab-node_*.changes dist/ + ls -l dist/ + + - uses: actions/upload-artifact@v4 + with: + name: mlab-node-${{ matrix.arch }} + path: dist/ + if-no-files-found: error + + - name: Attach to release + if: startsWith(github.ref, 'refs/tags/') + env: + GH_TOKEN: ${{ github.token }} + run: | + gh release view "${GITHUB_REF_NAME}" >/dev/null 2>&1 \ + || gh release create "${GITHUB_REF_NAME}" --generate-notes + gh release upload "${GITHUB_REF_NAME}" dist/*.deb --clobber diff --git a/debian/control b/debian/control index 8d97d39..b6ae47e 100644 --- a/debian/control +++ b/debian/control @@ -16,7 +16,7 @@ Homepage: https://github.com/m-lab/autonode Rules-Requires-Root: no Package: mlab-node -Architecture: amd64 +Architecture: amd64 arm64 Depends: ${shlibs:Depends}, ${misc:Depends}, From b25eff469080905e03862921ab247bdf61a123ba Mon Sep 17 00:00:00 2001 From: Roberto D'Auria Date: Tue, 7 Jul 2026 15:28:54 +0200 Subject: [PATCH 09/25] ci: retrigger workflow From cf219598086b7db038558bd8fa2e41c64cd9e138 Mon Sep 17 00:00:00 2001 From: Roberto D'Auria Date: Tue, 7 Jul 2026 19:59:07 +0200 Subject: [PATCH 10/25] build-binaries: survive cleanup of the read-only Go module cache Go marks module cache files read-only, so the EXIT trap's plain rm -rf failed with permission errors and tanked the build exit code after all artifacts had been staged. Use -modcacherw and chmod in the trap. --- .build/build-binaries.sh | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/.build/build-binaries.sh b/.build/build-binaries.sh index e574c87..5fa9cb0 100755 --- a/.build/build-binaries.sh +++ b/.build/build-binaries.sh @@ -71,16 +71,19 @@ case "$(uname -sm)" in esac WORK_DIR="$(mktemp -d)" -trap 'rm -rf "${WORK_DIR}"' EXIT +# Go marks its module cache read-only; chmod before removal so cleanup never +# fails the build (belt and braces with -modcacherw below). +trap 'chmod -R u+rwX "${WORK_DIR}" 2>/dev/null || true; rm -rf "${WORK_DIR}" 2>/dev/null || true' EXIT # Hermetic Go state: do not touch (or depend on) the build host's module # cache. CGO is disabled by default (overridden per-build where required); -# -trimpath aids reproducibility. +# -trimpath aids reproducibility; -modcacherw keeps the throwaway module +# cache writable so it can be removed. export CGO_ENABLED=0 export GOTOOLCHAIN="${GO_TOOLCHAIN}" export GOPATH="${WORK_DIR}/gopath" export GOCACHE="${WORK_DIR}/gocache" -export GOFLAGS="-trimpath" +export GOFLAGS="-trimpath -modcacherw" # clone_at_tag URL TAG DEST — shallow clone of a repo at a release tag. # The clone (not a tarball download) provides the commit hash for version From 182e30ece367b895d12cb564d76b3897c82ff29f Mon Sep 17 00:00:00 2001 From: Roberto D'Auria Date: Thu, 16 Jul 2026 12:53:25 +0200 Subject: [PATCH 11/25] refactor: drop custom postinst, split out static.env, cache binary fetch --- .DS_Store | Bin 8196 -> 0 bytes .build/fetch-binaries.sh | 90 ++++++++++++-------- .gitignore | 3 +- README.md | 7 +- config/mlab-node.env | 5 -- config/static.env | 8 ++ debian/mlab-node.install | 22 ++--- debian/mlab-node.postinst | 15 ---- debian/mlab-node.tmpfiles | 5 ++ debian/rules | 15 ++-- scripts/wait-ready | 15 ---- scripts/wait-socket | 18 ++-- systemd/mlab-node-heartbeat.service | 1 + systemd/mlab-node-jostler.service | 1 + systemd/mlab-node-metadata.service | 1 - systemd/mlab-node-node-exporter.service | 5 +- systemd/mlab-node-register-ready.service | 7 +- systemd/mlab-node-register.service | 1 + systemd/mlab-node-traceroute-caller.service | 3 +- systemd/mlab-node-uuid-annotator.service | 1 + 20 files changed, 110 insertions(+), 113 deletions(-) delete mode 100644 .DS_Store create mode 100644 config/static.env delete mode 100755 debian/mlab-node.postinst delete mode 100755 scripts/wait-ready diff --git a/.DS_Store b/.DS_Store deleted file mode 100644 index f1a1ab67596f1a1f84da061e894de2fd09a0fb49..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 8196 zcmeHM%}*0S6n}#VTTxjQ6pSVtdoh71A8ItlPzppFKOmGK_*HhdLs?m7n%xBiLejII z^saZWo=m)W^y1Z<@lVjJCw=o#p)HtrF&bx*ncv%)_j^0@Yi4KP763?gr8of44FDEa zfp!}@*O@~pyC}0#Y3K#{90)K-7cxH=Y?Q!m_ zQPUa)i~|3q0^<4LU=^6rSWze+9oWbe0I?anWkDPB0LgJQW;9k5N-D-w*#ps3qEif_ z=s4~Ocf^dwib6#PqUb>Mor%s+h`c-U9O(`uqtLWQ0i!^p0wQ;B0UtOpSg6b2#a7B< zJ^Ni2Np686Q1H;x1-2vC7rw85C;4E57VIN+u2Z!5aJy@Iy^KDc+Z{68wTMzC<4r6q z#59@RVb2xp_Slbj7*6syNCM90?#HNgE}_dgw!ra@tElrtSq#N9G>Q=$IJ0wZ6uGQK zv+EB+*Q4PC%km`Q%feGc^EsQ0sE9Nhpe@#t9{#GNOlNLXjWf!yA*~F28||!G8cF({ zKQHZR#u-he(wRv;WI->Cz%7`8CD?#x@B&`JTX+wj;0t^st)z#XCcWf5xk`q~b&?}@ ziBA?uKprW6Q}u^FrQ>NJF&Z^L^lCn0A!jA*7o28Tgano$tla?QG>+}uEUj=8axeqfX4%Q&C7&O6%0dmUo*q>ZuV zMt`|uqa#@?)zO&ISWzgI!$zLMi01r9T}&|w{7VY#Q/dev/null)" = "${SCRIPT_HASH}" ]; then + echo "binaries/ already staged by this script version; skipping fetch" + exit 0 +fi + # Image versions. These mirror the tags pinned in the original docker-compose.yml. # NOTE: the annotation2 schema generator intentionally comes from an older # uuid-annotator image (v0.5.8) than the running annotator (v0.5.10), matching @@ -40,23 +51,29 @@ IMG_NODE_EXPORTER="quay.io/prometheus/node-exporter:v1.9.0" # system go fetches it on demand (the bare "go 1.25" directive otherwise mis- # resolves to a non-existent "go1.25" download). NDT_SERVER_SRC="https://github.com/m-lab/ndt-server.git" -NDT_SERVER_VERSION="v0.25.2" +NDT_SERVER_VERSION="${IMG_NDT_SERVER##*:}" NDT_GO_TOOLCHAIN="go1.25.11" -# extraction requests: "image|search-names|dest-name" +# extraction requests: "image|search-names|dest-name|type" # search-names is a comma-separated list of candidate basenames to locate in the -# image rootfs (first match wins), to tolerate differing install paths. +# image rootfs (first match wins), to tolerate differing install paths. type is +# "bin" (installed 0755, ELF-checked) or "data" (installed 0644, unchecked). REQUESTS=( - "${IMG_NDT_SERVER}|ndt-server|ndt-server" - "${IMG_HEARTBEAT}|heartbeat|heartbeat" - "${IMG_UUID_ANNOTATOR}|uuid-annotator|uuid-annotator" - "${IMG_UUID_ANNOTATOR_SCHEMA}|generate-schemas|generate-schemas-annotation2" - "${IMG_JOSTLER}|jostler|jostler" - "${IMG_TRACEROUTE}|traceroute-caller|traceroute-caller" - "${IMG_TRACEROUTE}|generate-schemas|generate-schemas-traceroute" - "${IMG_TRACEROUTE}|scamper|scamper" - "${IMG_REGISTER}|register,autojoin-register|autojoin-register" - "${IMG_NODE_EXPORTER}|node_exporter,node-exporter|node-exporter" + "${IMG_NDT_SERVER}|ndt-server|ndt-server|bin" + "${IMG_HEARTBEAT}|heartbeat|heartbeat|bin" + "${IMG_UUID_ANNOTATOR}|uuid-annotator|uuid-annotator|bin" + "${IMG_UUID_ANNOTATOR_SCHEMA}|generate-schemas|generate-schemas-annotation2|bin" + "${IMG_JOSTLER}|jostler|jostler|bin" + "${IMG_TRACEROUTE}|traceroute-caller|traceroute-caller|bin" + "${IMG_TRACEROUTE}|generate-schemas|generate-schemas-traceroute|bin" + "${IMG_TRACEROUTE}|scamper|scamper|bin" + "${IMG_REGISTER}|register,autojoin-register|autojoin-register|bin" + "${IMG_NODE_EXPORTER}|node_exporter,node-exporter|node-exporter|bin" + # The IPInfo AS-names CSV bundled in the uuid-annotator image. The image + # provides it via `ENV ASNAME_URL file:///data/asnames.ipinfo.csv`, which the + # binary reads through flagx.ArgsFromEnv (asname.url <- ASNAME_URL). systemd + # does not inherit image ENVs, so we ship the file and pass it via -asname.url. + "${IMG_UUID_ANNOTATOR}|asnames.ipinfo.csv|asnames.ipinfo.csv|data" ) for tool in skopeo jq tar file go git; do @@ -71,13 +88,16 @@ WORK_DIR="$(mktemp -d)" # the tree writable before removing it so cleanup never fails the script. trap 'chmod -R u+rwX "${WORK_DIR}" 2>/dev/null || true; rm -rf "${WORK_DIR}" 2>/dev/null || true' EXIT +# mangle IMAGE — filesystem-safe directory name for an image reference. +mangle() { echo "$1" | tr '/:' '__'; } + # unpack_image IMAGE DEST_ROOTFS # Copies the image to a local OCI/dir layout with skopeo and extracts every # layer (in manifest order) into DEST_ROOTFS to reconstruct the image rootfs. declare -A UNPACKED=() unpack_image() { local image="$1" rootfs="$2" - local imgdir="${WORK_DIR}/img/$(echo "${image}" | tr '/:' '__')" + local imgdir="${WORK_DIR}/img/$(mangle "${image}")" if [ -n "${UNPACKED[${image}]:-}" ]; then return 0 @@ -104,9 +124,10 @@ unpack_image() { } # check_binary PATH — classify the ELF. glibc-dynamic and static binaries run on -# Debian as-is (dh_shlibdeps resolves shared-lib deps). musl-linked binaries run -# only with the `musl` package installed, so they are flagged (the package -# declares the dependency) rather than rejected. A non-ELF file is fatal. +# Debian as-is (dh_shlibdeps resolves shared-lib deps). A musl-linked (Alpine) +# binary would not run on the target, so it aborts the build; rebuild such +# outliers from source instead (see generate-schemas-ndt7 below). A non-ELF +# file is fatal too. check_binary() { local bin="$1" info info="$(file -L "${bin}")" @@ -114,7 +135,9 @@ check_binary() { *"statically linked"*) ;; # portable, OK *"dynamically linked"*) if echo "${info}" | grep -q "ld-musl"; then - echo " WARNING: ${bin##*/} is musl-linked; requires the 'musl' package at runtime." >&2 + echo "ERROR: ${bin##*/} is musl-linked and would not run on the target;" >&2 + echo " rebuild it from source instead (see generate-schemas-ndt7)." >&2 + exit 1 fi ;; # glibc dynamic, OK (dh_shlibdeps handles deps) *"ELF"*) ;; @@ -130,8 +153,8 @@ rm -rf "${OUT_DIR}" mkdir -p "${OUT_DIR}" for req in "${REQUESTS[@]}"; do - IFS='|' read -r image names dest <<<"${req}" - rootfs="${WORK_DIR}/rootfs/$(echo "${image}" | tr '/:' '__')" + IFS='|' read -r image names dest type <<<"${req}" + rootfs="${WORK_DIR}/rootfs/$(mangle "${image}")" unpack_image "${image}" "${rootfs}" found="" @@ -148,23 +171,15 @@ for req in "${REQUESTS[@]}"; do exit 1 fi - install -D -m 0755 "${found}" "${OUT_DIR}/${dest}" - check_binary "${OUT_DIR}/${dest}" + if [ "${type}" = "bin" ]; then + install -D -m 0755 "${found}" "${OUT_DIR}/${dest}" + check_binary "${OUT_DIR}/${dest}" + else + install -D -m 0644 "${found}" "${OUT_DIR}/${dest}" + echo " ${dest}: $(wc -c <"${OUT_DIR}/${dest}" | tr -d ' ') bytes" + fi done -# Ship the IPInfo AS-names CSV bundled in the uuid-annotator image. The image -# provides it via `ENV ASNAME_URL file:///data/asnames.ipinfo.csv`, which the -# binary reads through flagx.ArgsFromEnv (asname.url <- ASNAME_URL). systemd does -# not inherit image ENVs, so we ship the same file and pass it via -asname.url. -ua_rootfs="${WORK_DIR}/rootfs/$(echo "${IMG_UUID_ANNOTATOR}" | tr '/:' '__')" -asname_src="$(find "${ua_rootfs}" -type f -name 'asnames.ipinfo.csv' 2>/dev/null | head -n1)" -if [ -z "${asname_src}" ]; then - echo "ERROR: asnames.ipinfo.csv not found in ${IMG_UUID_ANNOTATOR}" >&2 - exit 1 -fi -install -D -m 0644 "${asname_src}" "${OUT_DIR}/asnames.ipinfo.csv" -echo " asnames.ipinfo.csv: $(wc -c <"${OUT_DIR}/asnames.ipinfo.csv" | tr -d ' ') bytes" - # Build generate-schemas-ndt7 from source (glibc-dynamic, CGO on), since the # ndt-server image ships it musl-linked. cmd/generate-schemas is a nested Go # module inside the ndt-server repo, so build from within that directory. @@ -181,5 +196,6 @@ git clone --quiet --depth 1 --branch "${NDT_SERVER_VERSION}" "${NDT_SERVER_SRC}" ) check_binary "${OUT_DIR}/generate-schemas-ndt7" +printf '%s\n' "${SCRIPT_HASH}" > "${STAMP_FILE}" echo -echo "Staged $(ls -1 "${OUT_DIR}" | wc -l | tr -d ' ') binaries in ${OUT_DIR}" +echo "Staged $(ls -1 "${OUT_DIR}" | wc -l | tr -d ' ') artifacts in ${OUT_DIR}" diff --git a/.gitignore b/.gitignore index 9408389..73123e4 100644 --- a/.gitignore +++ b/.gitignore @@ -1,4 +1,5 @@ .env +.DS_Store # Binaries extracted from container images at build time (see # .build/fetch-binaries.sh). Not checked in. @@ -14,4 +15,4 @@ /debian/debhelper-build-stamp *.deb *.buildinfo -*.changes \ No newline at end of file +*.changes diff --git a/README.md b/README.md index 1e24fc1..3f4560c 100644 --- a/README.md +++ b/README.md @@ -35,7 +35,7 @@ build time (see `.build/fetch-binaries.sh`). | Path | Contents | |---|---| -| `/usr/lib/mlab/` | component binaries and helper scripts | +| `/usr/lib/mlab/` | component binaries, helper scripts, package-owned constants (`static.env`) | | `/etc/mlab/mlab-node.env` | configuration (non-secret) | | `/etc/mlab/api-key.env` | M-Lab API key (`0640 root:mlab-node`) | | `/etc/mlab/locate/verify.pub` | Locate token verification key | @@ -48,8 +48,9 @@ build time (see `.build/fetch-binaries.sh`). ## Building the package -Requires `dpkg-buildpackage`, debhelper, and (for binary extraction) -`skopeo`, `jq`, and `file`: +Requires `dpkg-buildpackage`, debhelper, `skopeo`, `jq`, and `file` for binary +extraction, plus `golang-go`, `git`, and a C toolchain to build +`generate-schemas-ndt7` from source: ```sh dpkg-buildpackage -us -uc -b diff --git a/config/mlab-node.env b/config/mlab-node.env index e0d08c7..f0d5590 100644 --- a/config/mlab-node.env +++ b/config/mlab-node.env @@ -37,8 +37,3 @@ IPV6= # TYPE is "physical" (bare-metal dedicated to M-Lab) or "virtual" (a VM). TYPE= - -# ### Static configuration. Do NOT change these. ### -PROJECT=mlab-autojoin -LOCATE_URL=locate.measurementlab.net -AUTH_URL=auth.mlab-oti.measurementlab.net diff --git a/config/static.env b/config/static.env new file mode 100644 index 0000000..60350ac --- /dev/null +++ b/config/static.env @@ -0,0 +1,8 @@ +# Package-owned constants for the mlab-node services. Installed under +# /usr/lib/mlab (not a conffile) so package upgrades can change them without +# conffile merge prompts. The units load this file before the operator-edited +# /etc/mlab/mlab-node.env, so values set there take precedence if you must +# override these. +PROJECT=mlab-autojoin +LOCATE_URL=locate.measurementlab.net +AUTH_URL=auth.mlab-oti.measurementlab.net diff --git a/debian/mlab-node.install b/debian/mlab-node.install index 3964942..49156c2 100644 --- a/debian/mlab-node.install +++ b/debian/mlab-node.install @@ -14,31 +14,21 @@ binaries/generate-schemas-traceroute usr/lib/mlab # Helper scripts. scripts/gen-uuid-prefix usr/lib/mlab scripts/write-metadata usr/lib/mlab -scripts/wait-ready usr/lib/mlab scripts/wait-socket usr/lib/mlab # systemd units. -systemd/mlab-node.target lib/systemd/system -systemd/mlab-node-register.service lib/systemd/system -systemd/mlab-node-register-ready.service lib/systemd/system -systemd/mlab-node-bbr.service lib/systemd/system -systemd/mlab-node-uuidprefix.service lib/systemd/system -systemd/mlab-node-metadata.service lib/systemd/system -systemd/mlab-node-schemas-ndt7.service lib/systemd/system -systemd/mlab-node-schemas-annotation2.service lib/systemd/system -systemd/mlab-node-schemas-traceroute.service lib/systemd/system -systemd/mlab-node-ndt-server.service lib/systemd/system -systemd/mlab-node-heartbeat.service lib/systemd/system -systemd/mlab-node-uuid-annotator.service lib/systemd/system -systemd/mlab-node-jostler.service lib/systemd/system -systemd/mlab-node-traceroute-caller.service lib/systemd/system -systemd/mlab-node-node-exporter.service lib/systemd/system +systemd/*.service lib/systemd/system +systemd/*.target lib/systemd/system # Configuration (files under /etc are registered as conffiles automatically). config/mlab-node.env etc/mlab config/api-key.env etc/mlab config/locate/verify.pub etc/mlab/locate +# Package-owned constants (not a conffile; loaded by the units before the +# operator-edited /etc/mlab/mlab-node.env). +config/static.env usr/lib/mlab + # Kernel module + sysctl drop-ins for BBR. config/modules-load.d/mlab-node.conf usr/lib/modules-load.d config/sysctl.d/90-mlab-node.conf usr/lib/sysctl.d diff --git a/debian/mlab-node.postinst b/debian/mlab-node.postinst deleted file mode 100755 index b06fbc1..0000000 --- a/debian/mlab-node.postinst +++ /dev/null @@ -1,15 +0,0 @@ -#!/bin/sh -set -e - -#DEBHELPER# - -# Restrict the API key file to the service group. Runs after the debhelper -# snippets above have created the mlab-node user/group (sysusers). -if [ "$1" = "configure" ]; then - if getent group mlab-node >/dev/null 2>&1; then - chown root:mlab-node /etc/mlab/api-key.env || true - chmod 0640 /etc/mlab/api-key.env || true - fi -fi - -exit 0 diff --git a/debian/mlab-node.tmpfiles b/debian/mlab-node.tmpfiles index fe654f0..834880f 100644 --- a/debian/mlab-node.tmpfiles +++ b/debian/mlab-node.tmpfiles @@ -13,3 +13,8 @@ d /var/lib/mlab/autocert 2750 mlab-node mlab-node - d /run/mlab 0755 mlab-node mlab-node - d /run/mlab/sockets 2770 mlab-node mlab-node - d /run/mlab/metadata 2755 mlab-node mlab-node - + +# Restrict the API key conffile to the service group (shipped 0644 root:root by +# dpkg). Applied on install by the debhelper tmpfiles snippet, which runs after +# the sysusers snippet has created the mlab-node group, and re-applied each boot. +z /etc/mlab/api-key.env 0640 root mlab-node - diff --git a/debian/rules b/debian/rules index d284aca..d891c69 100755 --- a/debian/rules +++ b/debian/rules @@ -12,18 +12,19 @@ execute_after_dh_install: dh_installsysusers # Extract the component binaries from the pinned container images (via skopeo, -# no Docker daemon) unless they are already staged. This runs as the package -# "build" step; the binaries are then installed via debian/mlab-node.install. +# no Docker daemon). The script owns its own caching: it exits immediately if +# binaries/ was staged by the current script version (stamp file), and +# re-fetches after any script change (e.g. a version bump) or interrupted run. override_dh_auto_build: - if [ ! -e binaries/ndt-server ]; then ./.build/fetch-binaries.sh; fi + ./.build/fetch-binaries.sh # Install the target so it is enabled (auto-start on boot) but not started at # install time - the operator must configure /etc/mlab/*.env first. The other -# units carry no [Install] section and are pulled in by the target, so they are -# only registered for daemon-reload, not enabled or started individually. +# units carry no [Install] section (dh_installsystemd only enables units that +# have one) and are pulled in by the target, so they are only registered for +# daemon-reload, not enabled or started individually. override_dh_installsystemd: - dh_installsystemd --no-start --no-enable - dh_installsystemd --no-start --name=mlab-node mlab-node.target + dh_installsystemd --no-start # The shipped binaries are M-Lab's released, tested artifacts; do not strip or # otherwise rewrite them. diff --git a/scripts/wait-ready b/scripts/wait-ready deleted file mode 100755 index f4e38d4..0000000 --- a/scripts/wait-ready +++ /dev/null @@ -1,15 +0,0 @@ -#!/bin/sh -# Poll the register-node readiness endpoint until it reports ready. This gates -# every dependent service on a successful node registration (replacing the -# docker-compose `depends_on: condition: service_healthy`). -set -eu - -url="${1:-http://localhost:8001/ready}" -for _ in $(seq 1 60); do - if curl -fsS -o /dev/null "${url}"; then - exit 0 - fi - sleep 2 -done -echo "mlab-node: ${url} did not become ready in time" >&2 -exit 1 diff --git a/scripts/wait-socket b/scripts/wait-socket index faf7068..8cdc217 100755 --- a/scripts/wait-socket +++ b/scripts/wait-socket @@ -1,13 +1,19 @@ #!/bin/sh -# Wait for a unix socket to appear before starting. `After=` only guarantees the -# producing service's process has started, not that it has created its socket; -# this closes that race for the tcpinfo/ipservice sockets. +# Wait for one or more unix sockets to appear before starting. `After=` only +# guarantees the producing service's process has started, not that it has +# created its socket; this closes that race for the tcpinfo/ipservice sockets. +# All sockets are watched in the same loop, so the timeout is a single 60s +# budget rather than 60s per socket. set -eu -sock="${1:?usage: wait-socket SOCKET_PATH}" +[ $# -ge 1 ] || { echo "usage: wait-socket SOCKET_PATH..." >&2; exit 2; } for _ in $(seq 1 60); do - [ -S "${sock}" ] && exit 0 + ok=1 + for sock in "$@"; do + [ -S "${sock}" ] || { ok=0; break; } + done + [ "${ok}" -eq 1 ] && exit 0 sleep 1 done -echo "mlab-node: socket ${sock} did not appear in time" >&2 +echo "mlab-node: socket(s) $* did not appear in time" >&2 exit 1 diff --git a/systemd/mlab-node-heartbeat.service b/systemd/mlab-node-heartbeat.service index 19038e5..7844855 100644 --- a/systemd/mlab-node-heartbeat.service +++ b/systemd/mlab-node-heartbeat.service @@ -13,6 +13,7 @@ BindsTo=mlab-node-register.service [Service] Type=exec +EnvironmentFile=/usr/lib/mlab/static.env EnvironmentFile=/etc/mlab/mlab-node.env EnvironmentFile=/etc/mlab/api-key.env ExecStart=/usr/lib/mlab/heartbeat \ diff --git a/systemd/mlab-node-jostler.service b/systemd/mlab-node-jostler.service index 4c01036..cf3c074 100644 --- a/systemd/mlab-node-jostler.service +++ b/systemd/mlab-node-jostler.service @@ -10,6 +10,7 @@ BindsTo=mlab-node-register.service [Service] Type=exec +EnvironmentFile=/usr/lib/mlab/static.env EnvironmentFile=/etc/mlab/mlab-node.env Environment=GOOGLE_APPLICATION_CREDENTIALS=/var/lib/mlab/node/service-account-autojoin.json ExecStart=/usr/lib/mlab/jostler \ diff --git a/systemd/mlab-node-metadata.service b/systemd/mlab-node-metadata.service index 8e387c4..2a81ac3 100644 --- a/systemd/mlab-node-metadata.service +++ b/systemd/mlab-node-metadata.service @@ -2,7 +2,6 @@ Description=Write external IP metadata files for traceroute-caller Documentation=https://github.com/m-lab/autonode PartOf=mlab-node.target -RequiresMountsFor=/run/mlab # Replaces the docker-compose "configs" that injected ${IPV4}/${IPV6} as the # /metadata/external-ip[v6] files. Regenerated each boot (the files live on the # /run tmpfs). diff --git a/systemd/mlab-node-node-exporter.service b/systemd/mlab-node-node-exporter.service index 79119d1..dba1253 100644 --- a/systemd/mlab-node-node-exporter.service +++ b/systemd/mlab-node-node-exporter.service @@ -28,13 +28,12 @@ RestartSec=5 # --- hardening --- # node-exporter shares nothing with the rest of the stack and needs no state, so # it runs under an ephemeral DynamicUser. It only reads world-readable /proc and -# /sys, so it needs no capabilities and no write access. +# /sys, so it needs no capabilities and no write access. ProtectProc= is left at +# its default: the collectors read system-wide /proc files. DynamicUser=yes NoNewPrivileges=yes ProtectSystem=strict ProtectHome=yes -ProtectProc=default -ProcSubset=all ProtectKernelTunables=yes ProtectKernelModules=yes ProtectKernelLogs=yes diff --git a/systemd/mlab-node-register-ready.service b/systemd/mlab-node-register-ready.service index 86fcb63..2719ee8 100644 --- a/systemd/mlab-node-register-ready.service +++ b/systemd/mlab-node-register-ready.service @@ -2,7 +2,6 @@ Description=Wait for M-Lab node registration to be ready Documentation=https://github.com/m-lab/autonode PartOf=mlab-node.target -Requires=mlab-node-register.service After=mlab-node-register.service BindsTo=mlab-node-register.service # This oneshot gates every dependent component on a successful registration @@ -12,7 +11,11 @@ BindsTo=mlab-node-register.service [Service] Type=oneshot RemainAfterExit=yes -ExecStart=/usr/lib/mlab/wait-ready http://localhost:8001/ready +# Poll until registration reports ready: retry connection-refused (register +# still starting) and HTTP errors (not yet registered) for up to ~120s. +ExecStart=/usr/bin/curl -fsS -o /dev/null \ + --retry 60 --retry-delay 2 --retry-connrefused --retry-all-errors \ + http://localhost:8001/ready # --- hardening --- User=mlab-node diff --git a/systemd/mlab-node-register.service b/systemd/mlab-node-register.service index 61bb819..dd11fd5 100644 --- a/systemd/mlab-node-register.service +++ b/systemd/mlab-node-register.service @@ -13,6 +13,7 @@ StartLimitBurst=5 [Service] Type=exec +EnvironmentFile=/usr/lib/mlab/static.env EnvironmentFile=/etc/mlab/mlab-node.env EnvironmentFile=/etc/mlab/api-key.env ExecStart=/usr/lib/mlab/autojoin-register \ diff --git a/systemd/mlab-node-traceroute-caller.service b/systemd/mlab-node-traceroute-caller.service index 8dff50b..05aa5ba 100644 --- a/systemd/mlab-node-traceroute-caller.service +++ b/systemd/mlab-node-traceroute-caller.service @@ -13,8 +13,7 @@ BindsTo=mlab-node-register.service [Service] Type=exec EnvironmentFile=/etc/mlab/mlab-node.env -ExecStartPre=/usr/lib/mlab/wait-socket /run/mlab/sockets/events.sock -ExecStartPre=/usr/lib/mlab/wait-socket /run/mlab/sockets/uuid-annotator.sock +ExecStartPre=/usr/lib/mlab/wait-socket /run/mlab/sockets/events.sock /run/mlab/sockets/uuid-annotator.sock ExecStart=/usr/lib/mlab/traceroute-caller \ -scamper.bin=/usr/lib/mlab/scamper \ -prometheusx.listen-address=:9994 \ diff --git a/systemd/mlab-node-uuid-annotator.service b/systemd/mlab-node-uuid-annotator.service index 1e827bc..90321fd 100644 --- a/systemd/mlab-node-uuid-annotator.service +++ b/systemd/mlab-node-uuid-annotator.service @@ -12,6 +12,7 @@ BindsTo=mlab-node-register.service [Service] Type=exec +EnvironmentFile=/usr/lib/mlab/static.env EnvironmentFile=/etc/mlab/mlab-node.env Environment=GOOGLE_APPLICATION_CREDENTIALS=/var/lib/mlab/node/service-account-autojoin.json # Wait for ndt-server's events socket, and clear any stale annotator socket. From 32f9e2a35e1fc3a8b3ff008bfda833dddb84415e Mon Sep 17 00:00:00 2001 From: Roberto D'Auria Date: Thu, 16 Jul 2026 14:08:30 +0200 Subject: [PATCH 12/25] build: build component binaries from pinned sources instead of extracting from images --- .build/build-binaries.sh | 222 +++++++++++++++++++++++++++++++++++++++ .build/fetch-binaries.sh | 201 ----------------------------------- .gitignore | 5 +- README.md | 17 +-- debian/changelog | 4 +- debian/control | 7 +- debian/mlab-node.install | 2 +- debian/rules | 15 +-- 8 files changed, 248 insertions(+), 225 deletions(-) create mode 100755 .build/build-binaries.sh delete mode 100755 .build/fetch-binaries.sh diff --git a/.build/build-binaries.sh b/.build/build-binaries.sh new file mode 100755 index 0000000..cbb6622 --- /dev/null +++ b/.build/build-binaries.sh @@ -0,0 +1,222 @@ +#!/bin/bash +# +# build-binaries.sh builds the component binaries from the pinned upstream +# sources and stages them under ./binaries/ so they can be shipped in the +# mlab-node Debian package. The version pins are the same tags the historical +# docker-compose deployment ran (via the corresponding container images), but +# everything is built from source here: no container images, no Docker, no +# skopeo — just git, go and a C toolchain (for scamper and the cgo builds). +# +# Building from source (rather than extracting binaries from the images, as an +# earlier iteration of this script did) is what makes non-amd64 packages +# possible at all: M-Lab publishes amd64-only images, but the sources build +# anywhere Go and gcc do. +# +# Each Go component replicates its upstream Dockerfile's build: same package +# path, same CGO setting, same -ldflags (version stamps). The one divergence is +# ndt-server: upstream builds it CGO+musl-static on Alpine; here it is built +# CGO+glibc-dynamic, which runs natively on Debian (dh_shlibdeps picks up the +# libc dependency). +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)" +OUT_DIR="${REPO_DIR}/binaries" + +# The staged tree is fresh only if it was produced by this exact script +# (the version pins live in it): the stamp records the script's hash and is +# written only after a fully successful run. Editing the script (e.g. bumping +# a pinned tag) or an interrupted build invalidates it. +STAMP_FILE="${OUT_DIR}/.build-stamp" +SCRIPT_HASH="$(sha256sum "${BASH_SOURCE[0]}" | awk '{print $1}')" +if [ "$(cat "${STAMP_FILE}" 2>/dev/null)" = "${SCRIPT_HASH}" ]; then + echo "binaries/ already staged by this script version; skipping build" + exit 0 +fi + +# Version pins. These mirror the image tags pinned in the original +# docker-compose.yml (each image was built from the repo tag of the same name). +# NOTE: the annotation2 schema generator intentionally comes from an older +# uuid-annotator tag (v0.5.8) than the running annotator (v0.5.10), matching +# the compose configuration. +# NOTE: heartbeat lives in the m-lab/locate repository (cmd/heartbeat); the +# measurementlab/heartbeat image was built from locate's tags. +NDT_SERVER_VERSION="v0.25.2" +LOCATE_VERSION="v0.19.1" +UUID_ANNOTATOR_VERSION="v0.5.10" +UUID_ANNOTATOR_SCHEMA_VERSION="v0.5.8" +JOSTLER_VERSION="v1.1.4" +TRACEROUTE_VERSION="v0.12.0" +AUTOJOIN_VERSION="v0.2.13" +NODE_EXPORTER_VERSION="v1.9.0" + +# scamper is a C tool; the traceroute-caller repo vendors the exact tarball its +# image builds (third_party/scamper), so we build that same tarball. +SCAMPER_DIST="scamper-cvs-20230302" + +# ndt-server's go.mod carries a bare "go 1.25" directive, which a system go +# older than 1.25 mis-resolves to a non-existent "go1.25" toolchain download. +# Pin a real release; go fetches it on demand if the system go is older. +NDT_GO_TOOLCHAIN="go1.25.11" + +for tool in go git gcc make file; do + command -v "${tool}" >/dev/null 2>&1 || { + echo "ERROR: required build tool '${tool}' not found in PATH" >&2 + exit 1 + } +done + +# The work tree (sources + Go module/build caches) runs to a couple of GB, so +# keep it inside the repo rather than under /tmp, which is often a small tmpfs. +# The Go module cache is written read-only; restore write permission before +# removing so cleanup never fails the script. +WORK_DIR="$(mktemp -d "${REPO_DIR}/.build-work.XXXXXX")" +trap 'chmod -R u+rwX "${WORK_DIR}" 2>/dev/null || true; rm -rf "${WORK_DIR}" 2>/dev/null || true' EXIT + +# Hermetic Go state: modules, build cache and on-demand toolchains all live +# under WORK_DIR, so a build never depends on (or pollutes) the host's Go dirs. +export GOPATH="${WORK_DIR}/go" +export GOCACHE="${WORK_DIR}/gocache" +export GOFLAGS="-trimpath -mod=readonly" + +# check_binary PATH — sanity-check the produced ELF. Everything is built +# against glibc (static or dynamic) by construction; this catches accidental +# linkage against anything that would not resolve on the target (e.g. a +# scamper private .so if --disable-shared regressed) and logs what was built. +check_binary() { + local bin="$1" info + info="$(file -L "${bin}")" + case "${info}" in + *"statically linked"* | *"static-pie linked"*) ;; + *"dynamically linked"*) + if echo "${info}" | grep -q "ld-musl"; then + echo "ERROR: ${bin##*/} is musl-linked and would not run on the target." >&2 + exit 1 + fi + ;; + *) + echo "ERROR: ${bin} does not look like an executable: ${info}" >&2 + exit 1 + ;; + esac + echo " ${bin##*/}: ${info#*: }" +} + +# clone_repo NAME URL TAG — shallow-clone URL at TAG (once per NAME+TAG) and +# print the checkout directory. The clone keeps enough git metadata for the +# upstream version -ldflags (git describe / git log on the tagged commit). +clone_repo() { + local name="$1" url="$2" tag="$3" + local src="${WORK_DIR}/src/${name}-${tag}" + if [ ! -d "${src}" ]; then + git clone --quiet --depth 1 --branch "${tag}" "${url}" "${src}" >&2 + fi + echo "${src}" +} + +# short_commit SRC_DIR — the abbreviated hash of the tagged commit, used by the +# m-lab repos to stamp prometheusx.GitShortCommit (exposed as a metric label). +short_commit() { git -C "$1" log -1 --format=%h; } + +# go_build SRC_DIR PKG_DIR DEST CGO [LDFLAGS [GOTOOLCHAIN]] +# Build the Go main package at SRC_DIR/PKG_DIR into OUT_DIR/DEST. Runs from +# within PKG_DIR so nested modules (cmd/generate-schemas in ndt-server) build +# against their own go.mod. +go_build() { + local src="$1" pkg="$2" dest="$3" cgo="$4" ldflags="${5:-}" toolchain="${6:-}" + echo ">> building ${dest} (${src##*/} ${pkg})" + ( + cd "${src}/${pkg}" + CGO_ENABLED="${cgo}" \ + GOTOOLCHAIN="${toolchain:-auto}" \ + go build ${ldflags:+-ldflags "${ldflags}"} -o "${OUT_DIR}/${dest}" . + ) + check_binary "${OUT_DIR}/${dest}" +} + +rm -rf "${OUT_DIR}" +mkdir -p "${OUT_DIR}" + +# --- ndt-server (+ ndt7 schema generator) ----------------------------------- +# Upstream build.sh: CGO on (the ndt5 BBR code is cgo), stamped with the tag +# and short commit. cmd/generate-schemas is a nested Go module, so it is built +# from within its own directory. +src="$(clone_repo ndt-server https://github.com/m-lab/ndt-server.git "${NDT_SERVER_VERSION}")" +go_build "${src}" . ndt-server 1 \ + "-X github.com/m-lab/ndt-server/version.Version=${NDT_SERVER_VERSION} \ + -X github.com/m-lab/go/prometheusx.GitShortCommit=$(short_commit "${src}")" \ + "${NDT_GO_TOOLCHAIN}" +go_build "${src}" cmd/generate-schemas generate-schemas-ndt7 1 "" "${NDT_GO_TOOLCHAIN}" + +# --- heartbeat (from m-lab/locate) ------------------------------------------- +src="$(clone_repo locate https://github.com/m-lab/locate.git "${LOCATE_VERSION}")" +go_build "${src}" cmd/heartbeat heartbeat 0 \ + "-X github.com/m-lab/go/prometheusx.GitShortCommit=$(short_commit "${src}")" + +# --- uuid-annotator (+ annotation2 schema generator, older pin) --------------- +src="$(clone_repo uuid-annotator https://github.com/m-lab/uuid-annotator.git "${UUID_ANNOTATOR_VERSION}")" +go_build "${src}" . uuid-annotator 0 \ + "-X github.com/m-lab/go/prometheusx.GitShortCommit=$(short_commit "${src}")" + +# The IPInfo AS-names CSV the annotator reads via -asname.url. The image +# shipped it at /data and pointed at it with a container ENV; systemd units +# have no image ENVs, so the package ships the file and passes the flag. +install -D -m 0644 "${src}/data/asnames.ipinfo.csv" "${OUT_DIR}/asnames.ipinfo.csv" +echo " asnames.ipinfo.csv: $(wc -c <"${OUT_DIR}/asnames.ipinfo.csv" | tr -d ' ') bytes" + +src="$(clone_repo uuid-annotator https://github.com/m-lab/uuid-annotator.git "${UUID_ANNOTATOR_SCHEMA_VERSION}")" +go_build "${src}" cmd/generate-schemas generate-schemas-annotation2 0 + +# --- jostler ------------------------------------------------------------------- +# Upstream also stamps main.Version (git describe, i.e. the tag) and +# main.GitCommit (the full hash). +src="$(clone_repo jostler https://github.com/m-lab/jostler.git "${JOSTLER_VERSION}")" +go_build "${src}" cmd/jostler jostler 0 \ + "-X github.com/m-lab/go/prometheusx.GitShortCommit=$(short_commit "${src}") \ + -X main.Version=${JOSTLER_VERSION} \ + -X main.GitCommit=$(git -C "${src}" log -1 --format=%H)" + +# --- traceroute-caller (+ traceroute schema generator + scamper) --------------- +src="$(clone_repo traceroute-caller https://github.com/m-lab/traceroute-caller.git "${TRACEROUTE_VERSION}")" +go_build "${src}" . traceroute-caller 0 \ + "-X github.com/m-lab/go/prometheusx.GitShortCommit=$(short_commit "${src}")" +go_build "${src}" cmd/generate-schemas generate-schemas-traceroute 0 + +# scamper: build the exact tarball the traceroute-caller image vendors. +# --disable-shared statically links scamper's internal libraries into the +# binary, so the package can ship the single file with no private .so deps. +# OpenSSL is deliberately not linked: configure would silently pick it up if +# the dev headers happen to be installed, making the build host-dependent. The +# image's scamper was built without it, and traceroute-caller does not use the +# TLS probes. There is no --without-openssl switch (AX_CHECK_OPENSSL), so point +# the search at an empty directory instead. +echo ">> building scamper (${SCAMPER_DIST}, vendored in traceroute-caller)" +scamper_build="${WORK_DIR}/scamper" +scamper_prefix="${WORK_DIR}/scamper-install" +mkdir -p "${scamper_build}" +tar -xzf "${src}/third_party/scamper/${SCAMPER_DIST}.tar.gz" -C "${scamper_build}" +( + cd "${scamper_build}/${SCAMPER_DIST}" + chmod +x ./configure + ./configure --quiet --prefix="${scamper_prefix}" --disable-shared --with-openssl=/nonexistent + make --quiet -j"$(nproc)" >/dev/null + make --quiet install >/dev/null +) +install -m 0755 "${scamper_prefix}/bin/scamper" "${OUT_DIR}/scamper" +check_binary "${OUT_DIR}/scamper" + +# --- autojoin-register ---------------------------------------------------------- +src="$(clone_repo autojoin https://github.com/m-lab/autojoin.git "${AUTOJOIN_VERSION}")" +go_build "${src}" cmd/register autojoin-register 0 \ + "-s -w -X main.Version=${AUTOJOIN_VERSION}" + +# --- node-exporter --------------------------------------------------------------- +src="$(clone_repo node_exporter https://github.com/prometheus/node_exporter.git "${NODE_EXPORTER_VERSION}")" +go_build "${src}" . node-exporter 0 \ + "-X github.com/prometheus/common/version.Version=${NODE_EXPORTER_VERSION#v} \ + -X github.com/prometheus/common/version.Revision=$(git -C "${src}" log -1 --format=%H) \ + -X github.com/prometheus/common/version.Branch=HEAD" + +printf '%s\n' "${SCRIPT_HASH}" > "${STAMP_FILE}" +echo +echo "Staged $(ls -1 "${OUT_DIR}" | wc -l | tr -d ' ') artifacts in ${OUT_DIR}" diff --git a/.build/fetch-binaries.sh b/.build/fetch-binaries.sh deleted file mode 100755 index 3f27e5e..0000000 --- a/.build/fetch-binaries.sh +++ /dev/null @@ -1,201 +0,0 @@ -#!/bin/bash -# -# fetch-binaries.sh extracts the component binaries from the pinned M-Lab -# container images and stages them under ./binaries/ so they can be shipped in -# the mlab-node Debian package. It uses skopeo to pull the image filesystems -# without a running Docker daemon: no Docker is required to build the package, -# and none is required at runtime. -# -# Required host tools (build-time only): skopeo, jq, tar, file, go, git. -# -# The binaries are version-pinned to the same image tags used by the historical -# docker-compose deployment, so the package ships the exact binaries M-Lab has -# tested. Each extracted ELF is checked: a statically linked or glibc-linked -# binary is fine on Debian; a musl-linked (Alpine) binary is NOT and aborts the -# build, because it would fail to run on the target. -set -euo pipefail - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -REPO_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)" -OUT_DIR="${REPO_DIR}/binaries" - -# The staged tree is fresh only if it was produced by this exact script -# (the version pins live in it): the stamp records the script's hash and is -# written only after a fully successful run. Editing the script (e.g. bumping -# a pinned tag) or an interrupted fetch invalidates it. -STAMP_FILE="${OUT_DIR}/.fetch-stamp" -SCRIPT_HASH="$(sha256sum "${BASH_SOURCE[0]}" | awk '{print $1}')" -if [ "$(cat "${STAMP_FILE}" 2>/dev/null)" = "${SCRIPT_HASH}" ]; then - echo "binaries/ already staged by this script version; skipping fetch" - exit 0 -fi - -# Image versions. These mirror the tags pinned in the original docker-compose.yml. -# NOTE: the annotation2 schema generator intentionally comes from an older -# uuid-annotator image (v0.5.8) than the running annotator (v0.5.10), matching -# the compose configuration. -IMG_NDT_SERVER="measurementlab/ndt-server:v0.25.2" -IMG_HEARTBEAT="measurementlab/heartbeat:v0.19.1" -IMG_UUID_ANNOTATOR="measurementlab/uuid-annotator:v0.5.10" -IMG_UUID_ANNOTATOR_SCHEMA="measurementlab/uuid-annotator:v0.5.8" -IMG_JOSTLER="measurementlab/jostler:v1.1.4" -IMG_TRACEROUTE="measurementlab/traceroute-caller:v0.12.0" -IMG_REGISTER="measurementlab/autojoin-register:v0.2.13" -IMG_NODE_EXPORTER="quay.io/prometheus/node-exporter:v1.9.0" - -# generate-schemas-ndt7 is built from source rather than extracted: the -# ndt-server image builds it on Alpine with CGO (musl-dev), so the shipped copy -# is musl-linked. We rebuild the same pinned source with CGO on the glibc build -# host, yielding a glibc-dynamic binary (NEEDED: libc.so.6) that runs natively. -# NDT_GO_TOOLCHAIN must satisfy the `go` directive in that version's go.mod; the -# system go fetches it on demand (the bare "go 1.25" directive otherwise mis- -# resolves to a non-existent "go1.25" download). -NDT_SERVER_SRC="https://github.com/m-lab/ndt-server.git" -NDT_SERVER_VERSION="${IMG_NDT_SERVER##*:}" -NDT_GO_TOOLCHAIN="go1.25.11" - -# extraction requests: "image|search-names|dest-name|type" -# search-names is a comma-separated list of candidate basenames to locate in the -# image rootfs (first match wins), to tolerate differing install paths. type is -# "bin" (installed 0755, ELF-checked) or "data" (installed 0644, unchecked). -REQUESTS=( - "${IMG_NDT_SERVER}|ndt-server|ndt-server|bin" - "${IMG_HEARTBEAT}|heartbeat|heartbeat|bin" - "${IMG_UUID_ANNOTATOR}|uuid-annotator|uuid-annotator|bin" - "${IMG_UUID_ANNOTATOR_SCHEMA}|generate-schemas|generate-schemas-annotation2|bin" - "${IMG_JOSTLER}|jostler|jostler|bin" - "${IMG_TRACEROUTE}|traceroute-caller|traceroute-caller|bin" - "${IMG_TRACEROUTE}|generate-schemas|generate-schemas-traceroute|bin" - "${IMG_TRACEROUTE}|scamper|scamper|bin" - "${IMG_REGISTER}|register,autojoin-register|autojoin-register|bin" - "${IMG_NODE_EXPORTER}|node_exporter,node-exporter|node-exporter|bin" - # The IPInfo AS-names CSV bundled in the uuid-annotator image. The image - # provides it via `ENV ASNAME_URL file:///data/asnames.ipinfo.csv`, which the - # binary reads through flagx.ArgsFromEnv (asname.url <- ASNAME_URL). systemd - # does not inherit image ENVs, so we ship the file and pass it via -asname.url. - "${IMG_UUID_ANNOTATOR}|asnames.ipinfo.csv|asnames.ipinfo.csv|data" -) - -for tool in skopeo jq tar file go git; do - command -v "${tool}" >/dev/null 2>&1 || { - echo "ERROR: required build tool '${tool}' not found in PATH" >&2 - exit 1 - } -done - -WORK_DIR="$(mktemp -d)" -# Image layers can contain root-owned, mode-000 entries (e.g. var/empty); make -# the tree writable before removing it so cleanup never fails the script. -trap 'chmod -R u+rwX "${WORK_DIR}" 2>/dev/null || true; rm -rf "${WORK_DIR}" 2>/dev/null || true' EXIT - -# mangle IMAGE — filesystem-safe directory name for an image reference. -mangle() { echo "$1" | tr '/:' '__'; } - -# unpack_image IMAGE DEST_ROOTFS -# Copies the image to a local OCI/dir layout with skopeo and extracts every -# layer (in manifest order) into DEST_ROOTFS to reconstruct the image rootfs. -declare -A UNPACKED=() -unpack_image() { - local image="$1" rootfs="$2" - local imgdir="${WORK_DIR}/img/$(mangle "${image}")" - - if [ -n "${UNPACKED[${image}]:-}" ]; then - return 0 - fi - - echo ">> pulling ${image}" - mkdir -p "${imgdir}" "${rootfs}" - # Force a single linux/amd64 instance so the dir: manifest is an image - # manifest (with .layers) even when the source is a multi-arch index. - skopeo --override-os linux --override-arch amd64 \ - copy --quiet "docker://${image}" "dir:${imgdir}" - - # Layer blobs are listed (in order) in the manifest; each is a gzipped tar. - local layer - while read -r layer; do - local blob="${imgdir}/${layer#sha256:}" - [ -f "${blob}" ] || { echo "ERROR: missing layer blob ${blob}" >&2; exit 1; } - # Extract over the rootfs; ignore whiteout/permission quirks (we only need - # to read out a handful of executables afterwards). - tar -xf "${blob}" -C "${rootfs}" 2>/dev/null || true - done < <(jq -r '.layers[].digest' "${imgdir}/manifest.json") - - UNPACKED[${image}]=1 -} - -# check_binary PATH — classify the ELF. glibc-dynamic and static binaries run on -# Debian as-is (dh_shlibdeps resolves shared-lib deps). A musl-linked (Alpine) -# binary would not run on the target, so it aborts the build; rebuild such -# outliers from source instead (see generate-schemas-ndt7 below). A non-ELF -# file is fatal too. -check_binary() { - local bin="$1" info - info="$(file -L "${bin}")" - case "${info}" in - *"statically linked"*) ;; # portable, OK - *"dynamically linked"*) - if echo "${info}" | grep -q "ld-musl"; then - echo "ERROR: ${bin##*/} is musl-linked and would not run on the target;" >&2 - echo " rebuild it from source instead (see generate-schemas-ndt7)." >&2 - exit 1 - fi - ;; # glibc dynamic, OK (dh_shlibdeps handles deps) - *"ELF"*) ;; - *) - echo "ERROR: ${bin} does not look like an executable: ${info}" >&2 - exit 1 - ;; - esac - echo " ${bin##*/}: ${info#*: }" -} - -rm -rf "${OUT_DIR}" -mkdir -p "${OUT_DIR}" - -for req in "${REQUESTS[@]}"; do - IFS='|' read -r image names dest type <<<"${req}" - rootfs="${WORK_DIR}/rootfs/$(mangle "${image}")" - unpack_image "${image}" "${rootfs}" - - found="" - IFS=',' read -ra candidates <<<"${names}" - for name in "${candidates[@]}"; do - # Prefer an executable regular file; search the whole rootfs. - found="$(find "${rootfs}" -type f -name "${name}" -perm -u+x 2>/dev/null | head -n1 || true)" - [ -z "${found}" ] && found="$(find "${rootfs}" -type f -name "${name}" 2>/dev/null | head -n1 || true)" - [ -n "${found}" ] && break - done - - if [ -z "${found}" ]; then - echo "ERROR: could not find any of [${names}] in ${image}" >&2 - exit 1 - fi - - if [ "${type}" = "bin" ]; then - install -D -m 0755 "${found}" "${OUT_DIR}/${dest}" - check_binary "${OUT_DIR}/${dest}" - else - install -D -m 0644 "${found}" "${OUT_DIR}/${dest}" - echo " ${dest}: $(wc -c <"${OUT_DIR}/${dest}" | tr -d ' ') bytes" - fi -done - -# Build generate-schemas-ndt7 from source (glibc-dynamic, CGO on), since the -# ndt-server image ships it musl-linked. cmd/generate-schemas is a nested Go -# module inside the ndt-server repo, so build from within that directory. -echo ">> building generate-schemas-ndt7 from ${NDT_SERVER_SRC}@${NDT_SERVER_VERSION}" -SRC_DIR="${WORK_DIR}/ndt-server" -git clone --quiet --depth 1 --branch "${NDT_SERVER_VERSION}" "${NDT_SERVER_SRC}" "${SRC_DIR}" -( - cd "${SRC_DIR}/cmd/generate-schemas" - CGO_ENABLED=1 \ - GOTOOLCHAIN="${NDT_GO_TOOLCHAIN}" \ - GOPATH="${WORK_DIR}/go" \ - GOCACHE="${WORK_DIR}/gocache" \ - go build -trimpath -o "${OUT_DIR}/generate-schemas-ndt7" . -) -check_binary "${OUT_DIR}/generate-schemas-ndt7" - -printf '%s\n' "${SCRIPT_HASH}" > "${STAMP_FILE}" -echo -echo "Staged $(ls -1 "${OUT_DIR}" | wc -l | tr -d ' ') artifacts in ${OUT_DIR}" diff --git a/.gitignore b/.gitignore index 73123e4..a06aa74 100644 --- a/.gitignore +++ b/.gitignore @@ -1,9 +1,10 @@ .env .DS_Store -# Binaries extracted from container images at build time (see -# .build/fetch-binaries.sh). Not checked in. +# Binaries built from the pinned upstream sources at build time (see +# .build/build-binaries.sh). Not checked in. /binaries/ +/.build-work.* # Debian build artifacts. /debian/mlab-node/ diff --git a/README.md b/README.md index 3f4560c..3af10af 100644 --- a/README.md +++ b/README.md @@ -28,8 +28,9 @@ metadata. The whole stack is controlled atomically through `mlab-node.target`; `register-node` is the keystone — if it goes down, the entire stack is brought down with it. -The component binaries are extracted from M-Lab's pinned container images at -build time (see `.build/fetch-binaries.sh`). +The component binaries are built from the pinned upstream sources at package +build time (see `.build/build-binaries.sh`) — the same version tags the +container images were built from. ## File layout @@ -48,17 +49,17 @@ build time (see `.build/fetch-binaries.sh`). ## Building the package -Requires `dpkg-buildpackage`, debhelper, `skopeo`, `jq`, and `file` for binary -extraction, plus `golang-go`, `git`, and a C toolchain to build -`generate-schemas-ndt7` from source: +Requires `dpkg-buildpackage`, debhelper, `golang-go`, `git`, `file`, and a C +toolchain (for scamper and the cgo builds), plus network access to clone the +pinned sources and fetch Go modules: ```sh dpkg-buildpackage -us -uc -b ``` -The build pulls the pinned component binaries from the M-Lab images via skopeo -(no Docker daemon) and stages them under `binaries/`. The build fails if any -extracted binary is musl-linked (it would not run on Debian). +The build clones each component's repository at its pinned tag, builds the +binaries (replicating the upstream image builds' flags), and stages them under +`binaries/`. No container images or Docker are involved. ## Installing and configuring diff --git a/debian/changelog b/debian/changelog index 5f356bd..710ad8e 100644 --- a/debian/changelog +++ b/debian/changelog @@ -2,7 +2,7 @@ mlab-node (1.0.0) unstable; urgency=medium * Initial release: native Debian packaging of the M-Lab autonode stack. Replaces the Docker Compose deployment with hardened systemd services. - Component binaries are extracted from the pinned M-Lab container images at - build time; no Docker is required at build or run time. + Component binaries are built from the pinned upstream sources at build + time; no Docker or container images are involved at build or run time. -- Measurement Lab Mon, 29 Jun 2026 00:00:00 +0000 diff --git a/debian/control b/debian/control index 6d9fb81..053b38a 100644 --- a/debian/control +++ b/debian/control @@ -4,8 +4,6 @@ Priority: optional Maintainer: Measurement Lab Build-Depends: debhelper-compat (= 13), - skopeo, - jq, file, golang-go, git, @@ -32,5 +30,6 @@ Description: M-Lab node (autonode) measurement stack node exporter - as hardened systemd services confined with systemd's own sandboxing directives. No Docker is required at build or run time. . - The component binaries are extracted from M-Lab's pinned container images at - build time. The whole stack is controlled atomically through mlab-node.target. + The component binaries are built at package build time from the same pinned + upstream sources M-Lab releases. The whole stack is controlled atomically + through mlab-node.target. diff --git a/debian/mlab-node.install b/debian/mlab-node.install index 49156c2..da873cb 100644 --- a/debian/mlab-node.install +++ b/debian/mlab-node.install @@ -1,4 +1,4 @@ -# Component binaries (extracted from M-Lab images by .build/fetch-binaries.sh). +# Component binaries (built from the pinned sources by .build/build-binaries.sh). binaries/ndt-server usr/lib/mlab binaries/heartbeat usr/lib/mlab binaries/uuid-annotator usr/lib/mlab diff --git a/debian/rules b/debian/rules index d891c69..3094d03 100755 --- a/debian/rules +++ b/debian/rules @@ -11,12 +11,12 @@ execute_after_dh_install: dh_installsysusers -# Extract the component binaries from the pinned container images (via skopeo, -# no Docker daemon). The script owns its own caching: it exits immediately if -# binaries/ was staged by the current script version (stamp file), and -# re-fetches after any script change (e.g. a version bump) or interrupted run. +# Build the component binaries from the pinned upstream sources. The script +# owns its own caching: it exits immediately if binaries/ was staged by the +# current script version (stamp file), and rebuilds after any script change +# (e.g. a version bump) or interrupted run. override_dh_auto_build: - ./.build/fetch-binaries.sh + ./.build/build-binaries.sh # Install the target so it is enabled (auto-start on boot) but not started at # install time - the operator must configure /etc/mlab/*.env first. The other @@ -26,7 +26,8 @@ override_dh_auto_build: override_dh_installsystemd: dh_installsystemd --no-start -# The shipped binaries are M-Lab's released, tested artifacts; do not strip or -# otherwise rewrite them. +# Ship the binaries exactly as built from the pinned upstream sources; do not +# strip or otherwise rewrite them (Go binaries gain little from stripping, and +# dwz does not apply). override_dh_strip: override_dh_dwz: From ff033af19bf6f2775815907f4a6b88bae0e1d24e Mon Sep 17 00:00:00 2001 From: Roberto D'Auria Date: Wed, 22 Jul 2026 23:11:47 +0200 Subject: [PATCH 13/25] docs(debian): shorten package description --- debian/control | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/debian/control b/debian/control index 053b38a..b68f46c 100644 --- a/debian/control +++ b/debian/control @@ -26,10 +26,4 @@ Description: M-Lab node (autonode) measurement stack (NDT) measurement platform, replacing the former Docker Compose deployment. . It runs ndt-server and its sidecar services - node registration, heartbeat, - uuid-annotator, jostler, traceroute-caller (with scamper) and a Prometheus - node exporter - as hardened systemd services confined with systemd's own - sandboxing directives. No Docker is required at build or run time. - . - The component binaries are built at package build time from the same pinned - upstream sources M-Lab releases. The whole stack is controlled atomically - through mlab-node.target. + uuid-annotator, jostler, traceroute-caller and a Prometheus node exporter. From ba838cf3e10a0d60f70d5a7ac25d131cd9b3996c Mon Sep 17 00:00:00 2001 From: Roberto D'Auria Date: Wed, 22 Jul 2026 23:22:36 +0200 Subject: [PATCH 14/25] fix(debian): restart mlab-node.target on upgrade instead of leaving stack stopped --- debian/mlab-node.postinst | 18 ++++++++++++++++++ debian/rules | 7 ++++++- 2 files changed, 24 insertions(+), 1 deletion(-) create mode 100644 debian/mlab-node.postinst diff --git a/debian/mlab-node.postinst b/debian/mlab-node.postinst new file mode 100644 index 0000000..c8034ce --- /dev/null +++ b/debian/mlab-node.postinst @@ -0,0 +1,18 @@ +#!/bin/sh +set -e + +#DEBHELPER# + +# On upgrade ($2 is the previously-installed version), restart the stack so +# it picks up the new binaries and unit files. The explicit daemon-reload +# mirrors dh_installsystemd's own restart-after-upgrade snippet: debhelper +# 13.31 defers reload to a dpkg file trigger that only fires after postinst, +# so without it the restart would use the stale in-memory units. Every +# component is PartOf=mlab-node.target and pulled in by it, so restarting the +# target cycles the whole stack. On first install ($2 is empty) do nothing: +# the target is enabled but must not start before the operator configures +# /etc/mlab/*.env. +if [ "$1" = "configure" ] && [ -n "$2" ] && [ -d /run/systemd/system ]; then + systemctl --system daemon-reload >/dev/null || true + deb-systemd-invoke restart mlab-node.target >/dev/null || true +fi diff --git a/debian/rules b/debian/rules index 3094d03..c8794ef 100755 --- a/debian/rules +++ b/debian/rules @@ -23,8 +23,13 @@ override_dh_auto_build: # units carry no [Install] section (dh_installsystemd only enables units that # have one) and are pulled in by the target, so they are only registered for # daemon-reload, not enabled or started individually. +# +# --no-stop-on-upgrade keeps the old services running while an upgrade +# unpacks; the postinst then restarts mlab-node.target (upgrades only) so the +# stack picks up the new binaries. Without it, prerm stops every unit and +# --no-start would leave the node down after the upgrade. override_dh_installsystemd: - dh_installsystemd --no-start + dh_installsystemd --no-start --no-stop-on-upgrade # Ship the binaries exactly as built from the pinned upstream sources; do not # strip or otherwise rewrite them (Go binaries gain little from stripping, and From 860d27b03766a9f844b618e8da6b73623ced9a32 Mon Sep 17 00:00:00 2001 From: Roberto D'Auria Date: Wed, 22 Jul 2026 23:43:12 +0200 Subject: [PATCH 15/25] fix(debian): resolve lintian findings (unit path, PIE, copyright, overrides) --- .build/build-binaries.sh | 7 ++++++- debian/copyright | 13 +++++++------ debian/mlab-node.install | 8 +++++--- debian/mlab-node.lintian-overrides | 7 +++++++ 4 files changed, 25 insertions(+), 10 deletions(-) create mode 100644 debian/mlab-node.lintian-overrides diff --git a/.build/build-binaries.sh b/.build/build-binaries.sh index cbb6622..22bbe2d 100755 --- a/.build/build-binaries.sh +++ b/.build/build-binaries.sh @@ -124,12 +124,17 @@ short_commit() { git -C "$1" log -1 --format=%h; } # against their own go.mod. go_build() { local src="$1" pkg="$2" dest="$3" cgo="$4" ldflags="${5:-}" toolchain="${6:-}" + # cgo builds link dynamically anyway, so build them PIE for ASLR (lintian: + # hardening-no-pie); pure-Go builds stay internally-linked static, where + # -buildmode=pie does not apply. + local pie="" + [ "${cgo}" = "1" ] && pie="-buildmode=pie" echo ">> building ${dest} (${src##*/} ${pkg})" ( cd "${src}/${pkg}" CGO_ENABLED="${cgo}" \ GOTOOLCHAIN="${toolchain:-auto}" \ - go build ${ldflags:+-ldflags "${ldflags}"} -o "${OUT_DIR}/${dest}" . + go build ${pie} ${ldflags:+-ldflags "${ldflags}"} -o "${OUT_DIR}/${dest}" . ) check_binary "${OUT_DIR}/${dest}" } diff --git a/debian/copyright b/debian/copyright index e89a326..909856d 100644 --- a/debian/copyright +++ b/debian/copyright @@ -3,16 +3,17 @@ Upstream-Name: autonode Source: https://github.com/m-lab/autonode Files: * -Copyright: Measurement Lab +Copyright: 2025-2026 Measurement Lab License: Apache-2.0 Files: binaries/* -Copyright: Measurement Lab and the Prometheus Authors +Copyright: 2025-2026 Measurement Lab and the Prometheus Authors Comment: - These binaries are extracted at build time from M-Lab's released container - images (and the Prometheus node_exporter image). They are not part of the - source in this repository. Each component is licensed under its own upstream - license; see the respective M-Lab and Prometheus projects. + These binaries are built at package build time from the pinned upstream + sources by .build/build-binaries.sh. They are not part of the source in this + repository. Each component is licensed under its own upstream license; see + the respective M-Lab, Prometheus and scamper projects (scamper, vendored in + m-lab/traceroute-caller, is GPL-2 licensed). License: Apache-2.0 License: Apache-2.0 diff --git a/debian/mlab-node.install b/debian/mlab-node.install index da873cb..ca1081d 100644 --- a/debian/mlab-node.install +++ b/debian/mlab-node.install @@ -16,9 +16,11 @@ scripts/gen-uuid-prefix usr/lib/mlab scripts/write-metadata usr/lib/mlab scripts/wait-socket usr/lib/mlab -# systemd units. -systemd/*.service lib/systemd/system -systemd/*.target lib/systemd/system +# systemd units. Installed under the canonical /usr path: on merged-/usr +# systems, shipping files under the aliased /lib is a policy violation +# (lintian: aliased-location). +systemd/*.service usr/lib/systemd/system +systemd/*.target usr/lib/systemd/system # Configuration (files under /etc are registered as conffiles automatically). config/mlab-node.env etc/mlab diff --git a/debian/mlab-node.lintian-overrides b/debian/mlab-node.lintian-overrides new file mode 100644 index 0000000..58b7c94 --- /dev/null +++ b/debian/mlab-node.lintian-overrides @@ -0,0 +1,7 @@ +# The pure-Go binaries are statically linked by nature (internal linking, no +# cgo); this is how upstream builds and ships them. +statically-linked-binary [usr/lib/mlab/*] + +# Shipped unstripped on purpose (debian/rules overrides dh_strip): Go binaries +# gain little from stripping and the symbols aid debugging in the field. +unstripped-binary-or-object [usr/lib/mlab/*] From 1885735c932a3cee762b8009baa68ae500d3d866 Mon Sep 17 00:00:00 2001 From: Roberto D'Auria Date: Thu, 23 Jul 2026 00:14:22 +0200 Subject: [PATCH 16/25] build: strip binaries (-s -w) and cache components individually --- .build/build-binaries.sh | 216 ++++++++++++++++++++--------- .gitignore | 3 + debian/mlab-node.lintian-overrides | 4 - debian/rules | 14 +- 4 files changed, 163 insertions(+), 74 deletions(-) diff --git a/.build/build-binaries.sh b/.build/build-binaries.sh index 22bbe2d..7204fe1 100755 --- a/.build/build-binaries.sh +++ b/.build/build-binaries.sh @@ -13,26 +13,30 @@ # anywhere Go and gcc do. # # Each Go component replicates its upstream Dockerfile's build: same package -# path, same CGO setting, same -ldflags (version stamps). The one divergence is -# ndt-server: upstream builds it CGO+musl-static on Alpine; here it is built -# CGO+glibc-dynamic, which runs natively on Debian (dh_shlibdeps picks up the -# libc dependency). +# path, same CGO setting, same -ldflags (version stamps). Divergences from +# upstream: +# - ndt-server is built CGO+glibc-dynamic instead of CGO+musl-static on +# Alpine; it runs natively on Debian (dh_shlibdeps picks up the libc dep). +# - all binaries are linked with -s -w (no symbol table, no DWARF), roughly +# halving them. Panic backtraces stay fully symbolized (Go's pclntab is +# unaffected); only delve/pprof against the shipped binary lose fidelity. +# +# Caching: every component carries its own stamp under binaries/.stamps/, +# keyed on its recipe (pinned tag, package path, build flags, toolchain +# versions). Re-running the script rebuilds only components whose recipe +# changed — bumping one version pin rebuilds that one component. Delete +# binaries/ (or bump STAMP_SCHEMA below) to force a full rebuild. set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" REPO_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)" OUT_DIR="${REPO_DIR}/binaries" +STAMP_DIR="${OUT_DIR}/.stamps" -# The staged tree is fresh only if it was produced by this exact script -# (the version pins live in it): the stamp records the script's hash and is -# written only after a fully successful run. Editing the script (e.g. bumping -# a pinned tag) or an interrupted build invalidates it. -STAMP_FILE="${OUT_DIR}/.build-stamp" -SCRIPT_HASH="$(sha256sum "${BASH_SOURCE[0]}" | awk '{print $1}')" -if [ "$(cat "${STAMP_FILE}" 2>/dev/null)" = "${SCRIPT_HASH}" ]; then - echo "binaries/ already staged by this script version; skipping build" - exit 0 -fi +# Salt mixed into every recipe key. Bump it to invalidate all cached +# components at once, e.g. when the build logic itself changes in a way the +# per-component keys cannot see. +STAMP_SCHEMA=1 # Version pins. These mirror the image tags pinned in the original # docker-compose.yml (each image was built from the repo tag of the same name). @@ -66,6 +70,12 @@ for tool in go git gcc make file; do } done +# Toolchain fingerprints for the recipe keys: a Go upgrade invalidates every +# component; a gcc/glibc change only the native (cgo/C) ones, whose output — +# including the computed libc6 dependency — depends on the host toolchain. +GO_VERSION="$(go version | awk '{print $3}')" +NATIVE_TOOLCHAIN="gcc=$(gcc -dumpfullversion) libc=$(getconf GNU_LIBC_VERSION)" + # The work tree (sources + Go module/build caches) runs to a couple of GB, so # keep it inside the repo rather than under /tmp, which is often a small tmpfs. # The Go module cache is written read-only; restore write permission before @@ -73,12 +83,51 @@ done WORK_DIR="$(mktemp -d "${REPO_DIR}/.build-work.XXXXXX")" trap 'chmod -R u+rwX "${WORK_DIR}" 2>/dev/null || true; rm -rf "${WORK_DIR}" 2>/dev/null || true' EXIT -# Hermetic Go state: modules, build cache and on-demand toolchains all live -# under WORK_DIR, so a build never depends on (or pollutes) the host's Go dirs. -export GOPATH="${WORK_DIR}/go" -export GOCACHE="${WORK_DIR}/gocache" +# Hermetic Go state by default: modules, build cache and on-demand toolchains +# all live under WORK_DIR, so a build never depends on (or pollutes) the +# host's Go dirs. CI overrides AUTONODE_GOPATH/AUTONODE_GOCACHE to persistent +# directories so modules and compiled packages survive across runs (only +# WORK_DIR is removed by the EXIT trap). +export GOPATH="${AUTONODE_GOPATH:-${WORK_DIR}/go}" +export GOCACHE="${AUTONODE_GOCACHE:-${WORK_DIR}/gocache}" export GOFLAGS="-trimpath -mod=readonly" +BUILT=0 +SKIPPED=0 + +# recipe_key ARGS... — cache key for one component: a hash of everything that +# determines its output except the sources themselves, which are pinned by +# tag and therefore immutable. +recipe_key() { + printf '%s\n' "schema=${STAMP_SCHEMA}" "go=${GO_VERSION}" \ + "goflags=${GOFLAGS}" "$@" | sha256sum | awk '{print $1}' +} + +# needs_build NAME KEY OUTPUTS... — decide whether NAME must be (re)built. +# Only a matching stamp with every staged output present counts as cached. +needs_build() { + local name="$1" key="$2" out ok=1 + shift 2 + [ "$(cat "${STAMP_DIR}/${name}" 2>/dev/null)" = "${key}" ] || ok=0 + for out in "$@"; do + [ -e "${OUT_DIR}/${out}" ] || ok=0 + done + if [ "${ok}" = 1 ]; then + echo " ${name}: cached, skipping" + SKIPPED=$((SKIPPED + 1)) + return 1 + fi + rm -f "${STAMP_DIR}/${name}" + return 0 +} + +# stamp NAME KEY — record a successful component build. +stamp() { + mkdir -p "${STAMP_DIR}" + printf '%s\n' "$2" > "${STAMP_DIR}/$1" + BUILT=$((BUILT + 1)) +} + # check_binary PATH — sanity-check the produced ELF. Everything is built # against glibc (static or dynamic) by construction; this catches accidental # linkage against anything that would not resolve on the target (e.g. a @@ -124,6 +173,8 @@ short_commit() { git -C "$1" log -1 --format=%h; } # against their own go.mod. go_build() { local src="$1" pkg="$2" dest="$3" cgo="$4" ldflags="${5:-}" toolchain="${6:-}" + # Ship stripped: -s (symbol table) -w (DWARF); see the header comment. + ldflags="-s -w${ldflags:+ ${ldflags}}" # cgo builds link dynamically anyway, so build them PIE for ASLR (lintian: # hardening-no-pie); pure-Go builds stay internally-linked static, where # -buildmode=pie does not apply. @@ -134,58 +185,87 @@ go_build() { cd "${src}/${pkg}" CGO_ENABLED="${cgo}" \ GOTOOLCHAIN="${toolchain:-auto}" \ - go build ${pie} ${ldflags:+-ldflags "${ldflags}"} -o "${OUT_DIR}/${dest}" . + go build ${pie} -ldflags "${ldflags}" -o "${OUT_DIR}/${dest}" . ) check_binary "${OUT_DIR}/${dest}" } -rm -rf "${OUT_DIR}" +# build_go DEST URL TAG PKG_DIR CGO [LDFLAGS_TEMPLATE [GOTOOLCHAIN]] +# Cached build of one Go component: skipped entirely (including the clone) +# when its recipe key matches the stamp of a previous run. LDFLAGS_TEMPLATE +# may reference @COMMIT@ / @COMMIT_FULL@, replaced with the short/full hash of +# the tagged commit — a pure function of TAG, hence not part of the key. +build_go() { + local dest="$1" url="$2" tag="$3" pkg="$4" cgo="$5" tmpl="${6:-}" toolchain="${7:-}" + local native="" + [ "${cgo}" = "1" ] && native="${NATIVE_TOOLCHAIN}" + local key + key="$(recipe_key "url=${url}" "tag=${tag}" "pkg=${pkg}" "cgo=${cgo}" \ + "ldflags=${tmpl}" "toolchain=${toolchain}" "native=${native}")" + needs_build "${dest}" "${key}" "${dest}" || return 0 + local src + src="$(clone_repo "$(basename "${url}" .git)" "${url}" "${tag}")" + local ldflags="${tmpl//@COMMIT@/$(short_commit "${src}")}" + ldflags="${ldflags//@COMMIT_FULL@/$(git -C "${src}" log -1 --format=%H)}" + go_build "${src}" "${pkg}" "${dest}" "${cgo}" "${ldflags}" "${toolchain}" + stamp "${dest}" "${key}" +} + mkdir -p "${OUT_DIR}" +rm -f "${OUT_DIR}/.build-stamp" # pre-.stamps/ whole-script stamp, now unused # --- ndt-server (+ ndt7 schema generator) ----------------------------------- # Upstream build.sh: CGO on (the ndt5 BBR code is cgo), stamped with the tag # and short commit. cmd/generate-schemas is a nested Go module, so it is built # from within its own directory. -src="$(clone_repo ndt-server https://github.com/m-lab/ndt-server.git "${NDT_SERVER_VERSION}")" -go_build "${src}" . ndt-server 1 \ +NDT_SERVER_URL="https://github.com/m-lab/ndt-server.git" +build_go ndt-server "${NDT_SERVER_URL}" "${NDT_SERVER_VERSION}" . 1 \ "-X github.com/m-lab/ndt-server/version.Version=${NDT_SERVER_VERSION} \ - -X github.com/m-lab/go/prometheusx.GitShortCommit=$(short_commit "${src}")" \ + -X github.com/m-lab/go/prometheusx.GitShortCommit=@COMMIT@" \ "${NDT_GO_TOOLCHAIN}" -go_build "${src}" cmd/generate-schemas generate-schemas-ndt7 1 "" "${NDT_GO_TOOLCHAIN}" +build_go generate-schemas-ndt7 "${NDT_SERVER_URL}" "${NDT_SERVER_VERSION}" \ + cmd/generate-schemas 1 "" "${NDT_GO_TOOLCHAIN}" # --- heartbeat (from m-lab/locate) ------------------------------------------- -src="$(clone_repo locate https://github.com/m-lab/locate.git "${LOCATE_VERSION}")" -go_build "${src}" cmd/heartbeat heartbeat 0 \ - "-X github.com/m-lab/go/prometheusx.GitShortCommit=$(short_commit "${src}")" +build_go heartbeat https://github.com/m-lab/locate.git "${LOCATE_VERSION}" \ + cmd/heartbeat 0 \ + "-X github.com/m-lab/go/prometheusx.GitShortCommit=@COMMIT@" # --- uuid-annotator (+ annotation2 schema generator, older pin) --------------- -src="$(clone_repo uuid-annotator https://github.com/m-lab/uuid-annotator.git "${UUID_ANNOTATOR_VERSION}")" -go_build "${src}" . uuid-annotator 0 \ - "-X github.com/m-lab/go/prometheusx.GitShortCommit=$(short_commit "${src}")" +UUID_ANNOTATOR_URL="https://github.com/m-lab/uuid-annotator.git" +build_go uuid-annotator "${UUID_ANNOTATOR_URL}" "${UUID_ANNOTATOR_VERSION}" . 0 \ + "-X github.com/m-lab/go/prometheusx.GitShortCommit=@COMMIT@" # The IPInfo AS-names CSV the annotator reads via -asname.url. The image # shipped it at /data and pointed at it with a container ENV; systemd units # have no image ENVs, so the package ships the file and passes the flag. -install -D -m 0644 "${src}/data/asnames.ipinfo.csv" "${OUT_DIR}/asnames.ipinfo.csv" -echo " asnames.ipinfo.csv: $(wc -c <"${OUT_DIR}/asnames.ipinfo.csv" | tr -d ' ') bytes" +key="$(recipe_key "component=asnames" "url=${UUID_ANNOTATOR_URL}" \ + "tag=${UUID_ANNOTATOR_VERSION}")" +if needs_build asnames.ipinfo.csv "${key}" asnames.ipinfo.csv; then + src="$(clone_repo uuid-annotator "${UUID_ANNOTATOR_URL}" "${UUID_ANNOTATOR_VERSION}")" + install -D -m 0644 "${src}/data/asnames.ipinfo.csv" "${OUT_DIR}/asnames.ipinfo.csv" + echo " asnames.ipinfo.csv: $(wc -c <"${OUT_DIR}/asnames.ipinfo.csv" | tr -d ' ') bytes" + stamp asnames.ipinfo.csv "${key}" +fi -src="$(clone_repo uuid-annotator https://github.com/m-lab/uuid-annotator.git "${UUID_ANNOTATOR_SCHEMA_VERSION}")" -go_build "${src}" cmd/generate-schemas generate-schemas-annotation2 0 +build_go generate-schemas-annotation2 "${UUID_ANNOTATOR_URL}" \ + "${UUID_ANNOTATOR_SCHEMA_VERSION}" cmd/generate-schemas 0 # --- jostler ------------------------------------------------------------------- # Upstream also stamps main.Version (git describe, i.e. the tag) and # main.GitCommit (the full hash). -src="$(clone_repo jostler https://github.com/m-lab/jostler.git "${JOSTLER_VERSION}")" -go_build "${src}" cmd/jostler jostler 0 \ - "-X github.com/m-lab/go/prometheusx.GitShortCommit=$(short_commit "${src}") \ +build_go jostler https://github.com/m-lab/jostler.git "${JOSTLER_VERSION}" \ + cmd/jostler 0 \ + "-X github.com/m-lab/go/prometheusx.GitShortCommit=@COMMIT@ \ -X main.Version=${JOSTLER_VERSION} \ - -X main.GitCommit=$(git -C "${src}" log -1 --format=%H)" + -X main.GitCommit=@COMMIT_FULL@" # --- traceroute-caller (+ traceroute schema generator + scamper) --------------- -src="$(clone_repo traceroute-caller https://github.com/m-lab/traceroute-caller.git "${TRACEROUTE_VERSION}")" -go_build "${src}" . traceroute-caller 0 \ - "-X github.com/m-lab/go/prometheusx.GitShortCommit=$(short_commit "${src}")" -go_build "${src}" cmd/generate-schemas generate-schemas-traceroute 0 +TRACEROUTE_URL="https://github.com/m-lab/traceroute-caller.git" +build_go traceroute-caller "${TRACEROUTE_URL}" "${TRACEROUTE_VERSION}" . 0 \ + "-X github.com/m-lab/go/prometheusx.GitShortCommit=@COMMIT@" +build_go generate-schemas-traceroute "${TRACEROUTE_URL}" "${TRACEROUTE_VERSION}" \ + cmd/generate-schemas 0 # scamper: build the exact tarball the traceroute-caller image vendors. # --disable-shared statically links scamper's internal libraries into the @@ -195,33 +275,41 @@ go_build "${src}" cmd/generate-schemas generate-schemas-traceroute 0 # image's scamper was built without it, and traceroute-caller does not use the # TLS probes. There is no --without-openssl switch (AX_CHECK_OPENSSL), so point # the search at an empty directory instead. -echo ">> building scamper (${SCAMPER_DIST}, vendored in traceroute-caller)" -scamper_build="${WORK_DIR}/scamper" -scamper_prefix="${WORK_DIR}/scamper-install" -mkdir -p "${scamper_build}" -tar -xzf "${src}/third_party/scamper/${SCAMPER_DIST}.tar.gz" -C "${scamper_build}" -( - cd "${scamper_build}/${SCAMPER_DIST}" - chmod +x ./configure - ./configure --quiet --prefix="${scamper_prefix}" --disable-shared --with-openssl=/nonexistent - make --quiet -j"$(nproc)" >/dev/null - make --quiet install >/dev/null -) -install -m 0755 "${scamper_prefix}/bin/scamper" "${OUT_DIR}/scamper" -check_binary "${OUT_DIR}/scamper" +SCAMPER_CONFIGURE_FLAGS="--disable-shared --with-openssl=/nonexistent" +key="$(recipe_key "component=scamper" "dist=${SCAMPER_DIST}" \ + "vendored-in=${TRACEROUTE_VERSION}" "configure=${SCAMPER_CONFIGURE_FLAGS}" \ + "native=${NATIVE_TOOLCHAIN}")" +if needs_build scamper "${key}" scamper; then + src="$(clone_repo traceroute-caller "${TRACEROUTE_URL}" "${TRACEROUTE_VERSION}")" + echo ">> building scamper (${SCAMPER_DIST}, vendored in traceroute-caller)" + scamper_build="${WORK_DIR}/scamper" + scamper_prefix="${WORK_DIR}/scamper-install" + mkdir -p "${scamper_build}" + tar -xzf "${src}/third_party/scamper/${SCAMPER_DIST}.tar.gz" -C "${scamper_build}" + ( + cd "${scamper_build}/${SCAMPER_DIST}" + chmod +x ./configure + # shellcheck disable=SC2086 # deliberate word splitting of the flags + ./configure --quiet --prefix="${scamper_prefix}" ${SCAMPER_CONFIGURE_FLAGS} + make --quiet -j"$(nproc)" >/dev/null + make --quiet install >/dev/null + ) + install -m 0755 "${scamper_prefix}/bin/scamper" "${OUT_DIR}/scamper" + check_binary "${OUT_DIR}/scamper" + stamp scamper "${key}" +fi # --- autojoin-register ---------------------------------------------------------- -src="$(clone_repo autojoin https://github.com/m-lab/autojoin.git "${AUTOJOIN_VERSION}")" -go_build "${src}" cmd/register autojoin-register 0 \ - "-s -w -X main.Version=${AUTOJOIN_VERSION}" +build_go autojoin-register https://github.com/m-lab/autojoin.git \ + "${AUTOJOIN_VERSION}" cmd/register 0 \ + "-X main.Version=${AUTOJOIN_VERSION}" # --- node-exporter --------------------------------------------------------------- -src="$(clone_repo node_exporter https://github.com/prometheus/node_exporter.git "${NODE_EXPORTER_VERSION}")" -go_build "${src}" . node-exporter 0 \ +build_go node-exporter https://github.com/prometheus/node_exporter.git \ + "${NODE_EXPORTER_VERSION}" . 0 \ "-X github.com/prometheus/common/version.Version=${NODE_EXPORTER_VERSION#v} \ - -X github.com/prometheus/common/version.Revision=$(git -C "${src}" log -1 --format=%H) \ + -X github.com/prometheus/common/version.Revision=@COMMIT_FULL@ \ -X github.com/prometheus/common/version.Branch=HEAD" -printf '%s\n' "${SCRIPT_HASH}" > "${STAMP_FILE}" echo -echo "Staged $(ls -1 "${OUT_DIR}" | wc -l | tr -d ' ') artifacts in ${OUT_DIR}" +echo "Components: ${BUILT} built, ${SKIPPED} cached; staged in ${OUT_DIR}" diff --git a/.gitignore b/.gitignore index a06aa74..83e0c6e 100644 --- a/.gitignore +++ b/.gitignore @@ -6,6 +6,9 @@ /binaries/ /.build-work.* +# Persistent Go caches used by CI (AUTONODE_GOPATH/AUTONODE_GOCACHE). +/.go-cache/ + # Debian build artifacts. /debian/mlab-node/ /debian/.debhelper/ diff --git a/debian/mlab-node.lintian-overrides b/debian/mlab-node.lintian-overrides index 58b7c94..aa11da0 100644 --- a/debian/mlab-node.lintian-overrides +++ b/debian/mlab-node.lintian-overrides @@ -1,7 +1,3 @@ # The pure-Go binaries are statically linked by nature (internal linking, no # cgo); this is how upstream builds and ships them. statically-linked-binary [usr/lib/mlab/*] - -# Shipped unstripped on purpose (debian/rules overrides dh_strip): Go binaries -# gain little from stripping and the symbols aid debugging in the field. -unstripped-binary-or-object [usr/lib/mlab/*] diff --git a/debian/rules b/debian/rules index c8794ef..66fdf05 100755 --- a/debian/rules +++ b/debian/rules @@ -12,9 +12,9 @@ execute_after_dh_install: dh_installsysusers # Build the component binaries from the pinned upstream sources. The script -# owns its own caching: it exits immediately if binaries/ was staged by the -# current script version (stamp file), and rebuilds after any script change -# (e.g. a version bump) or interrupted run. +# owns its own caching: each component carries a recipe-keyed stamp under +# binaries/.stamps/, so only components whose pin or build flags changed are +# rebuilt. override_dh_auto_build: ./.build/build-binaries.sh @@ -31,8 +31,10 @@ override_dh_auto_build: override_dh_installsystemd: dh_installsystemd --no-start --no-stop-on-upgrade -# Ship the binaries exactly as built from the pinned upstream sources; do not -# strip or otherwise rewrite them (Go binaries gain little from stripping, and -# dwz does not apply). +# The Go binaries are linked with -s -w (see .build/build-binaries.sh), which +# leaves dh_strip nothing to do except scamper (C). Strip it, but skip the +# dbgsym package - nothing would consume it. dwz does not apply to Go. override_dh_strip: + dh_strip --no-automatic-dbgsym + override_dh_dwz: From 2abda0682e40db71e5e54d96c3a49f61674b8d54 Mon Sep 17 00:00:00 2001 From: Roberto D'Auria Date: Thu, 23 Jul 2026 00:14:22 +0200 Subject: [PATCH 17/25] ci: build the package on GitHub Actions with per-component caching --- .github/workflows/build-deb.yml | 138 ++++++++++++++++++++++++++++++++ README.md | 28 +++++-- 2 files changed, 161 insertions(+), 5 deletions(-) create mode 100644 .github/workflows/build-deb.yml diff --git a/.github/workflows/build-deb.yml b/.github/workflows/build-deb.yml new file mode 100644 index 0000000..3a30e26 --- /dev/null +++ b/.github/workflows/build-deb.yml @@ -0,0 +1,138 @@ +# Build the mlab-node Debian package. +# +# Every push/PR builds the .deb in a debian:bookworm container (the oldest +# supported target: its glibc sets the package's libc6 floor, so the .deb +# installs on bookworm and anything newer) and uploads it as a CI artifact. +# Pushing a v* tag additionally creates a GitHub Release with the .deb +# attached, versioned from the tag. +# +# Caching: binaries/ carries per-component stamps (see +# .build/build-binaries.sh), so a restored cache rebuilds only the components +# whose pins or build flags changed; the Go module/build caches make even a +# full rebuild incremental. +name: build-deb + +on: + pull_request: + push: + branches: [main] + tags: ['v*'] + workflow_dispatch: + +concurrency: + group: build-deb-${{ github.ref }} + cancel-in-progress: ${{ !startsWith(github.ref, 'refs/tags/') }} + +jobs: + build: + runs-on: ubuntu-latest + container: debian:bookworm + env: + AUTONODE_GOPATH: ${{ github.workspace }}/.go-cache/gopath + AUTONODE_GOCACHE: ${{ github.workspace }}/.go-cache/gocache + DEBEMAIL: support@measurementlab.net + DEBFULLNAME: Measurement Lab + steps: + - name: Install build dependencies + run: | + apt-get update + apt-get install -y --no-install-recommends \ + build-essential debhelper devscripts lintian git file \ + ca-certificates curl + + - uses: actions/checkout@v4 + + # Go comes from setup-go, not apt: bookworm's golang-go is far too old + # for the pinned components. Its built-in cache is disabled because it + # keys on go.sum files, which this repo does not have. + - uses: actions/setup-go@v5 + with: + go-version: stable + cache: false + + - name: Compute cache fingerprint + id: fp + run: echo "go=$(go version | awk '{print $3}')" >> "$GITHUB_OUTPUT" + + # Stale restores are useful (restore-keys): the per-component stamps let + # the build script skip everything whose recipe did not change. + - name: Cache built component binaries + uses: actions/cache@v4 + with: + path: binaries + key: binaries-${{ steps.fp.outputs.go }}-${{ hashFiles('.build/build-binaries.sh') }} + restore-keys: | + binaries-${{ steps.fp.outputs.go }}- + binaries- + + - name: Cache Go modules and build cache + uses: actions/cache@v4 + with: + path: .go-cache + key: go-${{ steps.fp.outputs.go }}-${{ hashFiles('.build/build-binaries.sh') }} + restore-keys: | + go-${{ steps.fp.outputs.go }}- + go- + + # Tag builds get the tag as their version; anything else gets a + # ~dev..g suffix, which sorts before the plain version so + # snapshots upgrade cleanly to the eventual release. + - name: Set package version + run: | + if [ "${GITHUB_REF_TYPE}" = "tag" ]; then + dch -b --newversion "${GITHUB_REF_NAME#v}" \ + --distribution unstable "Release ${GITHUB_REF_NAME}." + else + base="$(dpkg-parsechangelog -SVersion)" + dch -b --newversion "${base}~dev.${GITHUB_RUN_NUMBER}.g${GITHUB_SHA::7}" \ + --distribution UNRELEASED "CI snapshot build." + fi + dpkg-parsechangelog -SVersion + + # -d: the golang-go build-dep is satisfied by setup-go, invisible to dpkg. + - name: Build package + run: dpkg-buildpackage -us -uc -b -d + + - name: Lintian + run: lintian --fail-on error --tag-display-limit 0 ../mlab-node_*.changes + + # The postinst guards every systemd interaction on /run/systemd/system, + # so installing in the build container exercises the maintainer scripts + # (sysusers, tmpfiles, enable) without an init system. Installing on + # bookworm itself proves the computed libc6 floor is satisfiable on the + # oldest supported target. + - name: Install smoke test + run: | + apt-get install -y ../mlab-node_*.deb + dpkg -V mlab-node + echo "Depends: $(dpkg-deb -f ../mlab-node_*.deb Depends)" + + - name: Collect artifacts + run: mkdir -p dist && mv ../mlab-node_* dist/ + + - uses: actions/upload-artifact@v4 + with: + name: mlab-node-deb + path: dist/ + if-no-files-found: error + + release: + if: github.ref_type == 'tag' + needs: build + runs-on: ubuntu-latest + permissions: + contents: write + steps: + - uses: actions/download-artifact@v4 + with: + name: mlab-node-deb + path: dist + + - name: Create GitHub Release + env: + GH_TOKEN: ${{ github.token }} + run: | + gh release create "${GITHUB_REF_NAME}" dist/*.deb \ + --repo "${GITHUB_REPOSITORY}" \ + --title "${GITHUB_REF_NAME}" \ + --generate-notes diff --git a/README.md b/README.md index 3af10af..1b6ef9c 100644 --- a/README.md +++ b/README.md @@ -49,17 +49,35 @@ container images were built from. ## Building the package -Requires `dpkg-buildpackage`, debhelper, `golang-go`, `git`, `file`, and a C -toolchain (for scamper and the cgo builds), plus network access to clone the -pinned sources and fetch Go modules: +**CI is the canonical build**: every push runs the +[`build-deb` workflow](.github/workflows/build-deb.yml), which builds the +package in a `debian:bookworm` container — bookworm's glibc sets the +package's `libc6` floor, so the .deb installs on Debian 12 and anything +newer — and uploads it as a workflow artifact. Pushing a `v*` tag creates a +GitHub Release with the .deb attached; Releases are what production machines +should install. + +Local builds still work and use the same script. Requires +`dpkg-buildpackage`, debhelper, a recent Go, `git`, `file`, and a C toolchain +(for scamper and the cgo builds), plus network access to clone the pinned +sources and fetch Go modules: ```sh dpkg-buildpackage -us -uc -b ``` +(Add `-d` if Go is installed from upstream tarballs rather than the +`golang-go` package.) Note that a locally built .deb inherits the host's +glibc as its `libc6` dependency floor — a package built on a newer distro may +not install on bookworm; use the CI artifacts for anything that leaves your +machine. + The build clones each component's repository at its pinned tag, builds the -binaries (replicating the upstream image builds' flags), and stages them under -`binaries/`. No container images or Docker are involved. +binaries (replicating the upstream image builds' flags, plus `-s -w` +stripping), and stages them under `binaries/`. No container images or Docker +are involved. Each component is cached with a recipe-keyed stamp under +`binaries/.stamps/`, so re-builds only recompile components whose version pin +or build flags changed; delete `binaries/` to force a full rebuild. ## Installing and configuring From bb5703879d1da5373e842fd13bce9284eba49463 Mon Sep 17 00:00:00 2001 From: Roberto D'Auria Date: Thu, 23 Jul 2026 00:24:16 +0200 Subject: [PATCH 18/25] ci: fix container-job shell and workspace paths --- .github/workflows/build-deb.yml | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/.github/workflows/build-deb.yml b/.github/workflows/build-deb.yml index ad92e75..4334ae3 100644 --- a/.github/workflows/build-deb.yml +++ b/.github/workflows/build-deb.yml @@ -37,9 +37,11 @@ jobs: arch: arm64 runs-on: ${{ matrix.runner }} container: debian:bookworm + # Container jobs default to sh; the version-stamping step uses bashisms. + defaults: + run: + shell: bash env: - AUTONODE_GOPATH: ${{ github.workspace }}/.go-cache/gopath - AUTONODE_GOCACHE: ${{ github.workspace }}/.go-cache/gocache DEBEMAIL: support@measurementlab.net DEBFULLNAME: Measurement Lab steps: @@ -52,6 +54,14 @@ jobs: - uses: actions/checkout@v4 + # ${{ github.workspace }} resolves to the HOST path in container jobs; + # $GITHUB_WORKSPACE holds the container-side mount, which is what the + # build script (and the actions/cache 'path: .go-cache' below) need. + - name: Point Go caches into the workspace + run: | + echo "AUTONODE_GOPATH=${GITHUB_WORKSPACE}/.go-cache/gopath" >> "$GITHUB_ENV" + echo "AUTONODE_GOCACHE=${GITHUB_WORKSPACE}/.go-cache/gocache" >> "$GITHUB_ENV" + # Go comes from setup-go, not apt: bookworm's golang-go is far too old # for the pinned components. Its built-in cache is disabled because it # keys on go.sum files, which this repo does not have. From 92f73d9213dada9938663cb33752413522bdedb4 Mon Sep 17 00:00:00 2001 From: Roberto D'Auria Date: Thu, 23 Jul 2026 11:58:19 +0200 Subject: [PATCH 19/25] fix(ndt-server): pass -autocert.dir and -htmldir, defaults are CWD-relative --- systemd/mlab-node-ndt-server.service | 2 ++ 1 file changed, 2 insertions(+) diff --git a/systemd/mlab-node-ndt-server.service b/systemd/mlab-node-ndt-server.service index 3910ec3..d9053df 100644 --- a/systemd/mlab-node-ndt-server.service +++ b/systemd/mlab-node-ndt-server.service @@ -19,6 +19,7 @@ ExecStartPre=/bin/rm -f /run/mlab/sockets/events.sock ExecStart=/usr/lib/mlab/ndt-server \ -uuid-prefix-file=/var/lib/mlab/schemas/uuid.prefix \ -datadir=/var/lib/mlab/resultsdir/ndt \ + -htmldir=/usr/share/mlab/html \ -ndt7_addr=:443 \ -ndt7_addr_cleartext=:80 \ -compress-results=false \ @@ -36,6 +37,7 @@ ExecStart=/usr/lib/mlab/ndt-server \ -ndt5_ws_addr=127.0.0.1:3001 \ -tcpinfo.eventsocket=/run/mlab/sockets/events.sock \ -autocert.enabled=true \ + -autocert.dir=/var/lib/mlab/autocert \ -autocert.hostname=@/var/lib/mlab/node/hostname \ -token.machine=@/var/lib/mlab/node/hostname \ -token.verify-key=/etc/mlab/locate/verify.pub \ From 388f8e077737d4fc1953f7eefb64dbd65acc4a16 Mon Sep 17 00:00:00 2001 From: Roberto D'Auria Date: Thu, 23 Jul 2026 12:01:26 +0200 Subject: [PATCH 20/25] build: keep Go's scratch dir (GOTMPDIR) out of /tmp --- .build/build-binaries.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.build/build-binaries.sh b/.build/build-binaries.sh index f78a3e9..d6c20dd 100755 --- a/.build/build-binaries.sh +++ b/.build/build-binaries.sh @@ -90,6 +90,10 @@ trap 'chmod -R u+rwX "${WORK_DIR}" 2>/dev/null || true; rm -rf "${WORK_DIR}" 2>/ # WORK_DIR is removed by the EXIT trap). export GOPATH="${AUTONODE_GOPATH:-${WORK_DIR}/go}" export GOCACHE="${AUTONODE_GOCACHE:-${WORK_DIR}/gocache}" +# Go's per-build scratch dir ($WORK) follows TMPDIR, not GOCACHE - point it +# at WORK_DIR too, or large builds can fill a small /tmp tmpfs. +export GOTMPDIR="${WORK_DIR}/gotmp" +mkdir -p "${GOTMPDIR}" # -modcacherw keeps the module cache writable so cleanup and CI cache # handling never trip over go's default read-only files (belt and braces # with the chmod in the EXIT trap). From 08ac35d7f280f53b60f35ea49b066a83434c15bd Mon Sep 17 00:00:00 2001 From: Roberto D'Auria Date: Thu, 23 Jul 2026 12:09:27 +0200 Subject: [PATCH 21/25] fix(debian): restart the stack with --no-block so installs do not hang --- debian/mlab-node.postinst | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/debian/mlab-node.postinst b/debian/mlab-node.postinst index c8034ce..7bb6a4b 100644 --- a/debian/mlab-node.postinst +++ b/debian/mlab-node.postinst @@ -14,5 +14,12 @@ set -e # /etc/mlab/*.env. if [ "$1" = "configure" ] && [ -n "$2" ] && [ -d /run/systemd/system ]; then systemctl --system daemon-reload >/dev/null || true - deb-systemd-invoke restart mlab-node.target >/dev/null || true + # --no-block: queue the restart and return, instead of holding dpkg for + # the minutes the stack can take to come up (register-ready gates every + # service on a registration retry loop). Startup failures land in the + # journal, not the apt transcript. systemctl is called directly because + # deb-systemd-invoke cannot forward --no-block; the policy-rc.d layer it + # provides matters in chroots, which the /run/systemd/system guard + # already excludes. + systemctl --no-block restart mlab-node.target >/dev/null || true fi From 4dd8a12a4dd28d9c821386c8185f8574f5f1ff81 Mon Sep 17 00:00:00 2001 From: Roberto D'Auria Date: Thu, 23 Jul 2026 12:12:59 +0200 Subject: [PATCH 22/25] build(scamper): disable privsep, redundant and unworkable under the systemd sandbox --- .build/build-binaries.sh | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/.build/build-binaries.sh b/.build/build-binaries.sh index d6c20dd..8e7f420 100755 --- a/.build/build-binaries.sh +++ b/.build/build-binaries.sh @@ -277,12 +277,19 @@ build_go generate-schemas-traceroute "${TRACEROUTE_URL}" "${TRACEROUTE_VERSION}" # scamper: build the exact tarball the traceroute-caller image vendors. # --disable-shared statically links scamper's internal libraries into the # binary, so the package can ship the single file with no private .so deps. +# --disable-privsep drops scamper's built-in privilege separation (chroot to +# /var/empty + setuid), which cannot work - by design - inside the +# traceroute-caller unit's systemd sandbox (read-only /var, no CAP_SYS_CHROOT +# or CAP_SETUID, chroot filtered) and is redundant there: the unit itself +# confines scamper, and raw sockets come from the inherited CAP_NET_RAW +# ambient capability. (The container image instead granted scamper +# cap_sys_chroot,cap_setuid,cap_setgid file capabilities to let privsep run.) # OpenSSL is deliberately not linked: configure would silently pick it up if # the dev headers happen to be installed, making the build host-dependent. The # image's scamper was built without it, and traceroute-caller does not use the # TLS probes. There is no --without-openssl switch (AX_CHECK_OPENSSL), so point # the search at an empty directory instead. -SCAMPER_CONFIGURE_FLAGS="--disable-shared --with-openssl=/nonexistent" +SCAMPER_CONFIGURE_FLAGS="--disable-shared --disable-privsep --with-openssl=/nonexistent" key="$(recipe_key "component=scamper" "dist=${SCAMPER_DIST}" \ "vendored-in=${TRACEROUTE_VERSION}" "configure=${SCAMPER_CONFIGURE_FLAGS}" \ "native=${NATIVE_TOOLCHAIN}")" From c5d83d1f7a437a456fd7192d9af9236ffd6d5577 Mon Sep 17 00:00:00 2001 From: Roberto D'Auria Date: Thu, 23 Jul 2026 21:08:59 +0200 Subject: [PATCH 23/25] fix(ndt-server): drain via lame-duck ExecStop instead of SIGKILL on stop --- systemd/mlab-node-ndt-server.service | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/systemd/mlab-node-ndt-server.service b/systemd/mlab-node-ndt-server.service index d9053df..8ce5be6 100644 --- a/systemd/mlab-node-ndt-server.service +++ b/systemd/mlab-node-ndt-server.service @@ -45,6 +45,21 @@ ExecStart=/usr/lib/mlab/ndt-server \ Restart=always RestartSec=5 +# Graceful (lame-duck) stop. ndt-server's signal protocol is two SIGTERMs: +# the 1st flips /health (127.0.0.1:8000) to 500 — heartbeat notices within +# 10s and reports score 0, so Locate stops directing new clients here — and +# the 2nd exits the server. ExecStop sends the 1st and waits out the drain +# window; when it returns, systemd delivers its regular KillSignal (SIGTERM) +# as the 2nd, and the server exits cleanly with in-flight tests finished. +# The window mirrors the k8s platform's 100s termination grace period (10s +# heartbeat notice + 30s Locate propagation + 60s max test length). Do not +# send the 2nd SIGTERM from the script: ndt-server closes its signal channel +# after the 2nd, so a 3rd could panic the exiting process. +ExecStop=/bin/sh -c 'kill -s TERM "$MAINPID"; sleep 100' +# Bounds each stop phase (ExecStop above, then the SIGTERM kill phase) +# separately; must exceed the 100s drain or systemd would SIGKILL mid-drain. +TimeoutStopSec=130 + # --- hardening --- User=mlab-node Group=mlab-node From bfbce2df2c7a9d3c4a48dfa183699d5c65043252 Mon Sep 17 00:00:00 2001 From: Roberto D'Auria Date: Fri, 24 Jul 2026 00:36:25 +0200 Subject: [PATCH 24/25] docs: rewrite README prose, drop Docker references --- README.md | 60 ++++++++++++++++++++++++++----------------------------- 1 file changed, 28 insertions(+), 32 deletions(-) diff --git a/README.md b/README.md index 5309d5a..6cd61b9 100644 --- a/README.md +++ b/README.md @@ -1,17 +1,15 @@ # autonode (mlab-node) Native Debian packaging of the M-Lab "autonode" Network Diagnostic Tool (NDT) -measurement stack. The components run as hardened **systemd** services, confined -with systemd's own sandboxing directives — there is no Docker at build or run -time. +measurement stack. Each component runs as a hardened systemd service, confined +with systemd's own sandboxing directives. -This replaces the previous Docker Compose deployment. Documentation on -host-managed M-Lab servers is in the +Documentation on host-managed M-Lab servers is in the [Wiki](https://github.com/m-lab/autonode/wiki/Host%E2%80%90managed-Deployments). ## What's in the package -The `mlab-node` package installs and runs, as systemd services: +The `mlab-node` package installs and runs the following systemd services: | Service | Unit | |---|---| @@ -23,14 +21,13 @@ The `mlab-node` package installs and runs, as systemd services: | Traceroute caller (scamper) | `mlab-node-traceroute-caller.service` | | Prometheus node exporter | `mlab-node-node-exporter.service` | -plus one-shot units for schema generation, the UUID prefix, BBR, and external-IP -metadata. The whole stack is controlled atomically through `mlab-node.target`; -`register-node` is the keystone — if it goes down, the entire stack is brought -down with it. +There are also a few one-shot units for schema generation, the UUID prefix, +BBR, and external-IP metadata. The whole stack is started and stopped as a +unit through `mlab-node.target`; if `register-node` goes down, the entire +stack is brought down with it. -The component binaries are built from the pinned upstream sources at package -build time (see `.build/build-binaries.sh`) — the same version tags the -container images were built from. +The component binaries are built at package build time from the pinned +upstream sources (see `.build/build-binaries.sh`). ## File layout @@ -49,16 +46,16 @@ container images were built from. ## Building the package -**CI is the canonical build**: every push runs the -[`build-deb` workflow](.github/workflows/build-deb.yml), which builds the -package for **amd64 and arm64** (natively, on `ubuntu-24.04-arm` runners for -the latter) in a `debian:bookworm` container — bookworm's glibc sets the -package's `libc6` floor, so the .deb installs on Debian 12 and anything -newer — and uploads them as workflow artifacts. Pushing a `v*` tag creates a -GitHub Release with the .debs attached; Releases are what production machines -should install. +CI is the canonical way to build the package: every push runs the +[`build-deb` workflow](.github/workflows/build-deb.yml), which builds it for +amd64 and arm64 (natively, on `ubuntu-24.04-arm` runners for the latter) in a +`debian:bookworm` container and uploads the resulting .debs as workflow +artifacts. Building on bookworm sets the package's `libc6` floor to bookworm's +glibc, so the .deb installs on Debian 12 and anything newer. Pushing a `v*` +tag creates a GitHub Release with the .debs attached; production machines +should install from Releases. -Local builds still work and use the same script. Requires +Local builds work too and use the same script. You'll need `dpkg-buildpackage`, debhelper, a recent Go, `git`, `file`, and a C toolchain (for scamper and the cgo builds), plus network access to clone the pinned sources and fetch Go modules: @@ -69,14 +66,13 @@ dpkg-buildpackage -us -uc -b (Add `-d` if Go is installed from upstream tarballs rather than the `golang-go` package.) Note that a locally built .deb inherits the host's -glibc as its `libc6` dependency floor — a package built on a newer distro may -not install on bookworm; use the CI artifacts for anything that leaves your -machine. +glibc as its `libc6` dependency floor, so a package built on a newer distro +may not install on bookworm; use the CI artifacts for anything that leaves +your machine. The build clones each component's repository at its pinned tag, builds the -binaries (replicating the upstream image builds' flags, plus `-s -w` -stripping), and stages them under `binaries/`. No container images or Docker -are involved. Each component is cached with a recipe-keyed stamp under +binaries (using the upstream builds' flags, plus `-s -w` stripping), and +stages them under `binaries/`. Each component is cached with a recipe-keyed stamp under `binaries/.stamps/`, so re-builds only recompile components whose version pin or build flags changed; delete `binaries/` to force a full rebuild. @@ -111,7 +107,7 @@ sudo systemctl restart mlab-node.target ## Prometheus metrics -Exposed on the host (all `network_mode: host` equivalents are now native): -9990 ndt-server, 9991 jostler, 9992 uuid-annotator, 9993 heartbeat, -9994 traceroute-caller, 9995 node-exporter. ndt-server serves measurements on -80 (cleartext) and 443 (TLS). +All metrics are exposed directly on the host: 9990 ndt-server, 9991 jostler, +9992 uuid-annotator, 9993 heartbeat, 9994 traceroute-caller, +9995 node-exporter. ndt-server serves measurements on 80 (cleartext) and +443 (TLS). From 80f20895bd50fa3afda0948214308c2145761443 Mon Sep 17 00:00:00 2001 From: Roberto D'Auria Date: Fri, 24 Jul 2026 00:41:07 +0200 Subject: [PATCH 25/25] Revert "docs: rewrite README prose, drop Docker references" This reverts commit bfbce2df2c7a9d3c4a48dfa183699d5c65043252. --- README.md | 60 +++++++++++++++++++++++++++++-------------------------- 1 file changed, 32 insertions(+), 28 deletions(-) diff --git a/README.md b/README.md index 6cd61b9..5309d5a 100644 --- a/README.md +++ b/README.md @@ -1,15 +1,17 @@ # autonode (mlab-node) Native Debian packaging of the M-Lab "autonode" Network Diagnostic Tool (NDT) -measurement stack. Each component runs as a hardened systemd service, confined -with systemd's own sandboxing directives. +measurement stack. The components run as hardened **systemd** services, confined +with systemd's own sandboxing directives — there is no Docker at build or run +time. -Documentation on host-managed M-Lab servers is in the +This replaces the previous Docker Compose deployment. Documentation on +host-managed M-Lab servers is in the [Wiki](https://github.com/m-lab/autonode/wiki/Host%E2%80%90managed-Deployments). ## What's in the package -The `mlab-node` package installs and runs the following systemd services: +The `mlab-node` package installs and runs, as systemd services: | Service | Unit | |---|---| @@ -21,13 +23,14 @@ The `mlab-node` package installs and runs the following systemd services: | Traceroute caller (scamper) | `mlab-node-traceroute-caller.service` | | Prometheus node exporter | `mlab-node-node-exporter.service` | -There are also a few one-shot units for schema generation, the UUID prefix, -BBR, and external-IP metadata. The whole stack is started and stopped as a -unit through `mlab-node.target`; if `register-node` goes down, the entire -stack is brought down with it. +plus one-shot units for schema generation, the UUID prefix, BBR, and external-IP +metadata. The whole stack is controlled atomically through `mlab-node.target`; +`register-node` is the keystone — if it goes down, the entire stack is brought +down with it. -The component binaries are built at package build time from the pinned -upstream sources (see `.build/build-binaries.sh`). +The component binaries are built from the pinned upstream sources at package +build time (see `.build/build-binaries.sh`) — the same version tags the +container images were built from. ## File layout @@ -46,16 +49,16 @@ upstream sources (see `.build/build-binaries.sh`). ## Building the package -CI is the canonical way to build the package: every push runs the -[`build-deb` workflow](.github/workflows/build-deb.yml), which builds it for -amd64 and arm64 (natively, on `ubuntu-24.04-arm` runners for the latter) in a -`debian:bookworm` container and uploads the resulting .debs as workflow -artifacts. Building on bookworm sets the package's `libc6` floor to bookworm's -glibc, so the .deb installs on Debian 12 and anything newer. Pushing a `v*` -tag creates a GitHub Release with the .debs attached; production machines -should install from Releases. +**CI is the canonical build**: every push runs the +[`build-deb` workflow](.github/workflows/build-deb.yml), which builds the +package for **amd64 and arm64** (natively, on `ubuntu-24.04-arm` runners for +the latter) in a `debian:bookworm` container — bookworm's glibc sets the +package's `libc6` floor, so the .deb installs on Debian 12 and anything +newer — and uploads them as workflow artifacts. Pushing a `v*` tag creates a +GitHub Release with the .debs attached; Releases are what production machines +should install. -Local builds work too and use the same script. You'll need +Local builds still work and use the same script. Requires `dpkg-buildpackage`, debhelper, a recent Go, `git`, `file`, and a C toolchain (for scamper and the cgo builds), plus network access to clone the pinned sources and fetch Go modules: @@ -66,13 +69,14 @@ dpkg-buildpackage -us -uc -b (Add `-d` if Go is installed from upstream tarballs rather than the `golang-go` package.) Note that a locally built .deb inherits the host's -glibc as its `libc6` dependency floor, so a package built on a newer distro -may not install on bookworm; use the CI artifacts for anything that leaves -your machine. +glibc as its `libc6` dependency floor — a package built on a newer distro may +not install on bookworm; use the CI artifacts for anything that leaves your +machine. The build clones each component's repository at its pinned tag, builds the -binaries (using the upstream builds' flags, plus `-s -w` stripping), and -stages them under `binaries/`. Each component is cached with a recipe-keyed stamp under +binaries (replicating the upstream image builds' flags, plus `-s -w` +stripping), and stages them under `binaries/`. No container images or Docker +are involved. Each component is cached with a recipe-keyed stamp under `binaries/.stamps/`, so re-builds only recompile components whose version pin or build flags changed; delete `binaries/` to force a full rebuild. @@ -107,7 +111,7 @@ sudo systemctl restart mlab-node.target ## Prometheus metrics -All metrics are exposed directly on the host: 9990 ndt-server, 9991 jostler, -9992 uuid-annotator, 9993 heartbeat, 9994 traceroute-caller, -9995 node-exporter. ndt-server serves measurements on 80 (cleartext) and -443 (TLS). +Exposed on the host (all `network_mode: host` equivalents are now native): +9990 ndt-server, 9991 jostler, 9992 uuid-annotator, 9993 heartbeat, +9994 traceroute-caller, 9995 node-exporter. ndt-server serves measurements on +80 (cleartext) and 443 (TLS).