diff --git a/README.md b/README.md index d3a5c8f..5507cf5 100644 --- a/README.md +++ b/README.md @@ -2,15 +2,16 @@ **Energy Optimization Governor** for HPC systems. -ThrottlePoint dynamically caps CPU frequency and NVIDIA GPU power based on -electricity price, grid carbon intensity, and temperature. Which signals drive -the decision is fully configurable, and you can plug in your own temperature -source. +ThrottlePoint caps CPU frequency (and NVIDIA GPU power) based on grid carbon +intensity, electricity price, or temperature. The shipped configuration +throttles the CPU only while the Wattnet CO₂ intensity is above a threshold +(430 gCO₂/kWh by default). Which signals drive the decision is configurable, +and you can plug in your own steps. It runs from cron as root, with no daemon and no batch-system integration. Each -run reads the signal its mode is configured for (a 1–10 rating from -price and carbon intensity, or a temperature band) and sets the CPU's maximum -scaling frequency accordingly. Running jobs are never signalled, paused, or +run reads its signal (Wattnet CO₂ against the threshold by default, or a +price/CO₂ rating or a temperature band) and sets the CPU's maximum scaling +frequency accordingly. Running jobs are never signalled, paused, or killed; they run under a lower clock ceiling while conditions are bad. --- @@ -97,14 +98,15 @@ sudo dpkg -i ../hpc-eff_*.deb # does not enable cron; see below Then configure and start: ```bash -sudo vi /etc/hpc_eff/config.ini # pick control_mode, then its section (see below) +sudo vi /etc/hpc_eff/config.ini # set WATTNET_EMAIL and WATTNET_PASSWORD sudo hpc-eff # one real run: applies a cap immediately sudo hpc-eff --enable # install /etc/cron.d/hpc-eff (every 10 min) ``` -What to set depends on the mode: `co2` needs `[CO2_API]` (an API key) and -`[frequency_tables]`; `temperature` needs `[TEMPERATURE_SOURCE]` and -`[CPU_THERMO]`, and calls no external API at all. +With the shipped config, the only required edit is the Wattnet login in +`[CO2_API]`; check `[frequency_tables]` for your CPU. The `temperature` mode +needs `[TEMPERATURE_SOURCE]` and `[CPU_THERMO]` instead and calls no external +API. Full prerequisites and build detail: **[docs/install.md](docs/install.md)**. Configuration and verification: **[docs/deployment.md](docs/deployment.md)**. @@ -115,12 +117,15 @@ Fleet rollout and uninstall: ## Regulation modes -`[MODE] control_mode` is the single switch per node. It is required. +The shipped config uses a `[PIPELINE]` section: CO₂ from Wattnet against +`[CO2_API] THRESHOLD`. Without `[PIPELINE]`, `[MODE] control_mode` selects a +preset and is then required. -| Mode | CPU | GPU | Inputs | +| Setup | CPU | GPU | Inputs | |---|---|---|---| -| `co2` | rating 1–10 → max-frequency cap | untouched | spot price + grid carbon intensity | -| `temperature` | hysteresis thermal bands | NVIDIA power limiting | one temperature sensor | +| CO₂ threshold (shipped) | full speed, or the lowest table frequency above the threshold | untouched | Wattnet carbon intensity | +| `control_mode=co2` | rating 1–10 → max-frequency cap | untouched | spot price + grid carbon intensity | +| `control_mode=temperature` | hysteresis thermal bands | NVIDIA power limiting | one temperature sensor | CPU regulation is mutually exclusive: a node is driven either by carbon/price or by temperature, never both. Details: @@ -149,7 +154,7 @@ JSON on stdout. Contract, built-in plugins, security rules and starter files: | Signal | Source | Notes | |---|---|---| | Electricity price | [spotovaelektrina.cz](https://spotovaelektrina.cz) (Czech OTE spot market) | Current price via API; the 12-month baseline is scraped from HTML | -| Carbon intensity | [Wattnet](https://api.wattnet.eu) (default) or [Nowtricity](https://www.nowtricity.com/) | Selectable via `[CO2_API] TYPE`; Nowtricity needs an API key, Wattnet a login (email + password, register at [api.wattnet.eu/token-request](https://api.wattnet.eu/token-request/)) | +| Carbon intensity | [Wattnet](https://api.wattnet.eu) | Needs a login (email + password, register at [api.wattnet.eu/token-request](https://api.wattnet.eu/token-request/)) | | Temperature | IPMI sensor, HTTP API, or your own Python module | "Bring your own reader": see [docs/configuration.md](docs/configuration.md#temperature_source-pluggable-temperature-reading) | --- diff --git a/docs/cluster-rollout.md b/docs/cluster-rollout.md index 6ac6ca8..6d905aa 100644 --- a/docs/cluster-rollout.md +++ b/docs/cluster-rollout.md @@ -14,12 +14,12 @@ coordination, no shared state, and no head node. Rollout is therefore only 1. **Build once per architecture.** The RPM is `noarch`; the DEB builds as `amd64` and needs a matching build host for other architectures. -2. **Ship one config per hardware group.** In `co2` mode `[frequency_tables]` - is the only hardware-specific part; the API keys and aggregation weights - are identical everywhere. In `temperature` mode it is `[CPU_THERMO]` and +2. **Ship one config per hardware group.** With the shipped CO₂ config + `[frequency_tables]` is the only hardware-specific part; the Wattnet login + and `THRESHOLD` are identical everywhere. In `temperature` mode it is `[CPU_THERMO]` and `[GPU_POWER]`, and the grouping is by *sensor*, not just by CPU model. 3. **Enable cron last**, after a manual `sudo hpc-eff` on one node of each - group has been verified. On Debian, remember the package already enabled it. + group has been verified. Neither package enables cron on install. 4. **Stagger the interval** across large groups in `co2` mode if you are worried about hammering the upstream APIs: `--cron-interval 10` everywhere means hundreds of nodes fetching the same price in the same second. There diff --git a/docs/configuration.md b/docs/configuration.md index f8be489..ecbd093 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -11,31 +11,38 @@ If that file does not exist, `hpc-eff` falls back to directory**, which is how you can run it straight from a git checkout, and also why a run from an unexpected directory can appear to ignore your settings. -The whole file is parsed upfront regardless of mode, so a syntax error -anywhere (even in a section your mode does not use) crashes the run. What -depends on `[MODE] control_mode` is which sections' *values* the active -regulator actually reads: - -| Section | `co2` mode | `temperature` mode | -|---|:--:|:--:| -| `[SYSTEM]` | yes | yes | -| `[MODE]` | yes | yes | -| `[logging]` | yes | yes | -| `[CO2_API]` | yes | n/a | -| `[aggregation]` | yes | n/a | -| `[frequency_tables]` | yes | n/a | -| `[TEMPERATURE_SOURCE]` | n/a | yes | -| `[CPU_THERMO]` | n/a | yes | -| `[GPU_POWER]` | n/a | yes | -| `[PIPELINE]`, `[plugin:NAME]` | optional | optional | +The whole file is parsed upfront, so a syntax error anywhere (even in a +section nothing reads) crashes the run. + +Comments must be on their own line. `KEY=430 # note` makes the value +`430 # note`; for a number that is a config error. + +The shipped `config.ini.example` regulates on Wattnet CO₂ against a +threshold, through `[PIPELINE]`. The `[MODE]` presets are the older way and +apply only when `[PIPELINE]` is absent. Which sections are read: + +| Section | Shipped default (`[PIPELINE]`, CO₂ threshold) | `control_mode=co2` (price + CO₂) | `control_mode=temperature` | +|---|:--:|:--:|:--:| +| `[CO2_API]` | yes | yes | n/a | +| `[frequency_tables]` | yes | yes | n/a | +| `[PIPELINE]`, `[plugin:NAME]` | yes | n/a | n/a | +| `[logging]` | yes | yes | yes | +| `[SYSTEM]` | power reading only | yes | yes | +| `[MODE]` | n/a | yes | yes | +| `[aggregation]` | n/a | yes | n/a | +| `[TEMPERATURE_SOURCE]` | n/a | n/a | yes | +| `[CPU_THERMO]` | n/a | n/a | yes | +| `[GPU_POWER]` | n/a | n/a | yes | --- ## `[MODE]` -The single user-facing switch. **Required**: an unset or unrecognised value -makes the run exit with `Config error: [MODE] control_mode must be one of: -temperature|co2` rather than silently doing nothing. +Selects a preset when there is no `[PIPELINE]` section (the shipped config has +one, so `[MODE]` is not needed there). Without `[PIPELINE]` it is +**required**: an unset or unrecognised value makes the run exit with +`Config error: [MODE] control_mode must be one of: temperature|co2` rather +than silently doing nothing. With `[PIPELINE]` present, `[MODE]` is ignored. ```ini [MODE] @@ -76,32 +83,13 @@ only, never acted on. --- -## `[CO2_API]`: carbon intensity backend +## `[CO2_API]`: carbon intensity (Wattnet) -```ini -[CO2_API] -TYPE=wattnet -``` - -`TYPE` selects the backend; the other backend's keys are ignored. - -### `TYPE=nowtricity` - -| Key | Default | -|---|---| -| `NOWTRICITY_USER_AGENT` | `HPC-Eff-Agent` | -| `NOWTRICITY_BASE_URL` | `https://www.nowtricity.com/api` | -| `NOWTRICITY_ZONE` | `czech-republic` | - -Two endpoints are called per run: `emissions-previous-24h//` and -`current-emissions//`. So a 10-minute interval means ~288 API calls per -node per day: check that against your plan's quota before a cluster-wide -rollout. +Wattnet is the only CO₂ backend. (Nowtricity was removed: a config that still +sets `TYPE=nowtricity` fails with a clear error in the `state.json` summary; +delete `TYPE` and the `NOWTRICITY_*` keys.) -An API key for the `nowtricity` backend is sent as `X-Api-Key`; get one -at nowtricity.com. - -### `TYPE=wattnet` (default) +### Wattnet **Wattnet requires a login (email and password). There are no API keys.** @@ -118,7 +106,6 @@ at nowtricity.com. ```ini [CO2_API] -TYPE=wattnet WATTNET_EMAIL=throttlepoint@your-site.example WATTNET_PASSWORD= ``` @@ -159,19 +146,71 @@ The 24-hour window is computed automatically. Wattnet splits the response into several series by data state: final data ends about 4 hours ago, newer data is pending or a preview and may still be revised slightly. All series are merged (final data wins on overlap) so the newest point is about now. The 15-minute -samples are averaged per clock hour; the newest, possibly partial, hour is the -"current" value. **If fewer than 24 hourly values come back the call fails**, -and the CO₂ fallback below applies. +samples are averaged per clock hour for the 24 h history. The "current" value +is the trailing 1 h average: the newest four 15-minute points. **If fewer than +24 hourly values come back the call fails**, and the CO₂ fallback below applies. + +### Threshold: when to throttle + +```ini +[CO2_API] +# gCO2/kWh, 1 h average +THRESHOLD=430 +# gCO2/kWh +HYSTERESIS=5 +``` + +Two levels only, the model of the GreenDIGIT threshold simulator: + +| Current CO₂ (1 h average) | Rating | CPU max frequency | +|---|---|---| +| above `THRESHOLD` | 10 | **last** value of the CPU's row in `[frequency_tables]` | +| at or below `THRESHOLD` | 1 | **first** value (full speed) | + +Once throttled, full speed returns only when the value drops below +`THRESHOLD - HYSTERESIS` (425 with the defaults), so a value hovering around +the threshold does not switch the cap on and off every run. + +The value can be changed while running: cron re-reads `config.ini` on every +run, so a new `THRESHOLD` applies within 10 minutes. A value that is not a +finite number (`430g`, `nan`), a negative `HYSTERESIS` or `MAX_AGE_MIN`, or an +inline comment after the value is treated like missing data: rating 1 and +`CO2 config error ([CO2_API] KEY=...)` in the `state.json` summary. + +The throttle carries over between runs only from a recent run (at most +`MAX_AGE_MIN` minutes old). After cron was off for longer, the next run starts +fresh. + +`state.json` records `co2_threshold`, `co2_throttled` and a summary such as +`CO2 437.2 g/kWh, threshold 430 (release below 425): throttled (rating 10)`. + +**How often it throttles** depends on the threshold. On the CZ winter data of +the GreenDIGIT simulator (Nov 2025 – Feb 2026, 1 h average, hysteresis 5): + +| `THRESHOLD` | Throttled time | Days with throttling (of 120) | +|---|---|---| +| 430 | 13.5 % | 31 (all in Jan–Feb) | +| 440 | 4.9 % | 15 | +| 450 | 2.3 % | 8 | +| 460 | 0.9 % | 3 | + +Without `THRESHOLD` the rating is the old 24 h percentile grade (1–10): the +current hour ranked against the last 24 hours. That throttles every day, even +on a clean day, because every day has a dirtiest hour. ### When the CO₂ fetch fails -The last real CO₂ grade is reused if it is at most `MAX_AGE_MIN` minutes old -(default 120, i.e. one missed hourly value plus slack). Older or none: rating 1, -no slowdown without evidence. Each failure logs a WARNING and a -`CO2 unavailable (...)` summary in `state.json`. +The last real result is reused if it is at most `MAX_AGE_MIN` minutes old +(default 120, i.e. one missed hourly value plus slack): the throttle decision +with `THRESHOLD`, the grade without it. Older or none: rating 1, no slowdown +without evidence. A reused result is never reused again, so an outage cannot +extend it. Each failure logs a WARNING and a `CO2 unavailable (...)` summary in +`state.json`. | Key | Default | |---|---| +| `THRESHOLD` | unset (24 h ranking); shipped config: `430` | +| `HYSTERESIS` | `5` | | `MAX_AGE_MIN` | `120` | --- @@ -198,7 +237,9 @@ Values are **MHz, highest first**. The key is chosen by matching Selection is `index = round((rating - 1) / 9 × (len - 1))` over the descending-sorted list: rating 1 takes the first entry, rating 10 the last, -the rest are spread evenly. The list length is your granularity; see +the rest are spread evenly. With `THRESHOLD` (the shipped default) only +ratings 1 and 10 occur, so only the first and the last entry matter: full +speed and throttled. The list length is your granularity; see [deployment.md](deployment.md#co2-mode-the-frequency-table). If the section or the key is missing entirely the built-in fallback @@ -209,6 +250,9 @@ Set `default` explicitly. ## `[aggregation]`: combining price and CO₂ +**Not used by the shipped config.** Only for the price + CO₂ blend: +`control_mode=co2` without `[PIPELINE]`, or `score = price, co2, combine`. + ```ini [aggregation] rating_type=average @@ -380,8 +424,9 @@ plugins per stage. score = co2 act = cpu_freq -[plugin:threshold_score] # only for your own executables -exec = /etc/hpc_eff/plugins/threshold_score.py +# only for your own executables +[plugin:my_score] +exec = /etc/hpc_eff/plugins/my_score.py timeout = 30 ``` diff --git a/docs/deployment.md b/docs/deployment.md index 5126d08..38b09ad 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -4,18 +4,17 @@ Configuring, running, verifying, and tuning one node. Build and install the package first with [install.md](install.md); when this node is working, [cluster-rollout.md](cluster-rollout.md) covers the rest of the fleet. -A compute node caps its CPU maximum frequency according to whichever signal -`[MODE] control_mode` selects: +A compute node caps its CPU maximum frequency from one signal: -| Mode | Description | Use case | -|------|-------------|----------| -| `co2` | Price/CO₂ regulator only | Cost and carbon optimisation | -| `temperature` | Temperature regulator + NVIDIA GPU power | Thermal management, datacenter continuity | +| Setup | Description | Use case | +|-------|-------------|----------| +| **CO₂ threshold** (shipped config, `[PIPELINE]`) | Throttle while Wattnet CO₂ is above `THRESHOLD` | Carbon: slow down only on dirty electricity | +| `control_mode=co2` (no `[PIPELINE]`) | Price + CO₂ rating 1–10 | Cost and carbon, throttles daily | +| `control_mode=temperature` (no `[PIPELINE]`) | Temperature regulator + NVIDIA GPU power | Thermal management, datacenter continuity | -`control_mode` is the single switch for the whole node; there are no separate -feature flags to set. The CPU is driven by either temperature or price/CO₂, never both. GPU power -regulation rides along with `temperature` mode only (NVIDIA GPUs only; safely -no-ops on nodes without them). +The CPU is driven by one of them, never two. GPU power regulation rides along +with `temperature` mode only (NVIDIA GPUs only; safely no-ops on nodes +without them). Neither mode is a fallback for the other. Pick per node according to what actually constrains it. What each mode does with its input is explained in @@ -31,74 +30,63 @@ Edit `/etc/hpc_eff/config.ini`. If it is missing for any reason: sudo install -D -m 640 /etc/hpc_eff/config.ini.example /etc/hpc_eff/config.ini ``` -It is required; an unset or unknown value aborts the run. Everything below -depends on which mode you picked. +The shipped file is the CO₂ threshold setup below. `[MODE] control_mode` is +needed only if you remove `[PIPELINE]` to use one of the older presets; then +an unset or unknown value aborts the run. -`[SYSTEM] POWERREADINGCMD` (`ipmitool dcmi power reading`) is logging-only in -both modes. Leave it as is if you have no BMC; the failure is caught. +`[SYSTEM] POWERREADINGCMD` (`ipmitool dcmi power reading`) is logging-only. +Leave it as is if you have no BMC; the failure is caught. -### Mode: `co2` +### CO₂ threshold (shipped default) -The four things to set beyond the mode itself: +What to set: | Section / key | Set it to | Notes | |---|---|---| -| `[CO2_API] TYPE` | `wattnet` (default) or `nowtricity` | Picks the backend. | -| `[CO2_API] NOWTRICITY_API_KEY` | your key | Only for `TYPE=nowtricity`. Also set `NOWTRICITY_USER_AGENT` to something identifying your site. | -| `[CO2_API] WATTNET_EMAIL`, `WATTNET_PASSWORD` | your Wattnet login | Only for `TYPE=wattnet`. Wattnet has no API keys; register at [api.wattnet.eu/token-request](https://api.wattnet.eu/token-request/), see [configuration.md](configuration.md#typewattnet-default). | +| `[CO2_API] WATTNET_EMAIL`, `WATTNET_PASSWORD` | your Wattnet login | The only required edit. Wattnet has no API keys; register at [api.wattnet.eu/token-request](https://api.wattnet.eu/token-request/), see [configuration.md](configuration.md#wattnet). | | `[frequency_tables]` | frequencies for your CPU, MHz, highest first | See [the frequency table](#co2-mode-the-frequency-table). | -| `[aggregation]` | `rating_type` and weights | See below. | +| `[CO2_API] THRESHOLD`, `HYSTERESIS` | gCO₂/kWh | Shipped `430` and `5`. See below. | -Minimum working `co2` config: +Minimum working config (CO₂ only, the shipped default): ```ini -[SYSTEM] -SCORENAME=SPEC2017 -SCORE=3.8 -POWERREADINGCMD=ipmitool dcmi power reading -DEBUG=yes - -[MODE] -control_mode=co2 - [CO2_API] -TYPE=wattnet WATTNET_EMAIL= WATTNET_PASSWORD= -WATTNET_ZONE=CZ - -[aggregation] -rating_type=average -weight_price=0.6 -weight_co2=0.4 +THRESHOLD=430 +HYSTERESIS=5 [frequency_tables] amd_epyc=2400,2000,1500 default=3000,2800,2700,2600,2400,2200 -[logging] -db_path=/var/lib/hpc_eff/history.db -state_json_path=/var/lib/hpc_eff/state.json -history_length=1000 -log_level=INFO +[PIPELINE] +read = power, cpufreq +score = co2 +act = cpu_freq ``` -> **Caution:** the config file holds an API key and the package installs it +> **Caution:** the config file holds the Wattnet password and the package installs it > mode 644. Tighten it with `sudo chmod 640 /etc/hpc_eff/config.ini`. Nothing > needs it world-readable; it only ever runs as root. -#### Choosing how conservative to be +#### Choosing the threshold -`[aggregation] rating_type` decides how the two signals combine: +A higher `THRESHOLD` throttles less often. On the CZ data of the GreenDIGIT +simulator (Nov 2025 – Feb 2026): -| Value | Effect | When to pick it | +| `THRESHOLD` | Throttled time | Days with throttling (of 120) | |---|---|---| -| `average` | Weighted mean (default 0.6 price / 0.4 CO₂) | Balanced; the recommended default to start from | -| `max` | The worse of the two: throttle if *either* is bad | Maximum carbon and cost avoidance, most throttling | -| `price` | Price only, CO₂ ignored | Pure cost optimisation, or no CO₂ API key yet | +| 430 | 13.5 % | 31 (all in Jan–Feb) | +| 440 | 4.9 % | 15 | +| 450 | 2.3 % | 8 | + +The value can be changed on a running node: the next cron run (within +10 minutes) uses it. Details: [configuration.md](configuration.md#threshold-when-to-throttle). -The reasoning behind the 0.6/0.4 split is in -[regulation-modes.md](regulation-modes.md#why-price-is-weighted-higher-than-carbon). +`[aggregation]` (price + CO₂ weights) is not used by this setup; see +[configuration.md](configuration.md) (`[aggregation]`) if you +blend price and CO₂. ### Mode: `temperature` @@ -181,21 +169,23 @@ sudo hpc-eff > **Warning:** there is no `--dry-run` flag; this applies the frequency cap > immediately. Do it on one node first, ideally a drained one. -With `DEBUG=yes` you get the whole pipeline on stdout. In `co2` mode: +With `DEBUG=yes` you get the whole pipeline on stdout. With the shipped +config, at 437 g/kWh (above 430) on a Xeon Gold: ``` [DEBUG] Starting HPC efficiency evaluator... -[DEBUG] control_mode='co2' applied -> FEATURES {...} -[DEBUG] Current electricity price (CZK/MWh): 2431 -[DEBUG] Average monthly prices last year (CZK/MWh): [...] -[DEBUG] The current price of 2431 is average (6). +[DEBUG] [PIPELINE] present and no control_mode: using the pipeline as configured +[DEBUG] Pipeline: {'read': ['power', 'cpufreq'], 'score': ['co2'], 'act': ['cpu_freq'], 'record': ['db_log', 'state_json']} [DEBUG] Last 24 hours CO2 values (g CO2eq/kWh): [...] -[DEBUG] Current CO2 value grade from 1 (low) to 10 (high): 4 -[DEBUG] Current rating: 5 -2026-08-21 10:00:03 INFO | Set max frequency: cpupower frequency-set -u 2600000 -2026-08-21 10:00:03 INFO | DB log entry inserted +[DEBUG] Current CO2 value (g CO2eq/kWh): 437.2 +[DEBUG] Current rating: 10 +2026-10-26 10:43:15 INFO | Set max frequency: cpupower frequency-set -u 2200000 +2026-10-26 10:43:15 INFO | DB log entry inserted ``` +The `state.json` summary says why: +`CO2 437.2 g/kWh, threshold 430 (release below 425): throttled (rating 10)`. + In `temperature` mode there are no price or CO₂ lines. You get the reading and the band decision instead: @@ -210,8 +200,8 @@ Only an actual band change logs `cpu_thermo applied target`. A steady node logs `Temperature 27.0°C → target 3.10GHz already set.` instead, which is the normal case, not a fault. -`Config error: [MODE] control_mode must be one of: …` means you skipped -[step 1](#1-configure). +`Config error: [MODE] control_mode must be one of: …` means `[PIPELINE]` was +removed and no `control_mode` is set; see [step 1](#1-configure). --- @@ -229,17 +219,18 @@ This must match the frequency the run applied. If the log says success but sysfs is unchanged, the frequency tool is a no-op on this node; see [troubleshooting.md](troubleshooting.md). -**The database got a row.** In `co2` mode: +**The database got a row.** With the shipped config: ```bash sudo sqlite3 /var/lib/hpc_eff/history.db \ - "SELECT timestamp, rating, rating_price, rating_co2, price, co2_current, freq_max + "SELECT timestamp, rating, co2_current, freq_max FROM hpc_eff_log ORDER BY id DESC LIMIT 5;" ``` -`rating_co2` being NULL means the CO₂ API call failed and the run silently -fell back to the price rating. That is by design, but if it is NULL every -time, your API key or egress is wrong. +`rating` is 10 (throttled) or 1 (full speed). `co2_current` being NULL means +the Wattnet call failed; the reason is in the `state.json` summary +(`CO2 unavailable (...)`). If it is NULL every time, the login or egress is +wrong. In `temperature` mode the rating and price columns stay NULL (nothing computes them), so look at the temperature columns instead: @@ -335,7 +326,9 @@ amd_epyc=2400,2000,1500 ``` The rating picks an index across the list: rating 1 → first entry, rating 10 → -last entry, everything else scaled evenly between. The number of entries is +last entry, everything else scaled evenly between. With the shipped +`THRESHOLD` only ratings 1 and 10 occur, so only the first entry (full speed) +and the last entry (throttled) matter. The number of entries is your granularity: three entries means the node only ever has three clock ceilings. Six to eight gives a smooth response; two gives an on/off switch. @@ -409,9 +402,9 @@ This writes `/etc/cron.d/hpc-eff`: */10 * * * * root /usr/bin/hpc-eff >/dev/null 2>&1 ``` -Interval guidance differs by mode. In `co2` mode both the spot price and the -CO₂ figures have hourly resolution, so anything below about 5 minutes adds API -traffic without changing the decision. In `temperature` mode the interval is +Interval guidance differs by mode. With the CO₂ setup the Wattnet data has +15-minute resolution and the signal is a 1 h average, so anything below about +5 minutes adds API traffic without changing the decision. In `temperature` mode the interval is also your reaction time to a hot room, and it sets how long recovery takes: upshifts step one band per run, so at 10 minutes a node needs 20 minutes of sustained cool to get from `LOW_FREQUENCY` back to `HIGH_FREQUENCY`. Shorten diff --git a/docs/install.md b/docs/install.md index 6fc13f3..04e458e 100644 --- a/docs/install.md +++ b/docs/install.md @@ -28,8 +28,8 @@ Additionally, for CO₂ regulation (the default): | Requirement | Why | Check | |---|---|---| -| Outbound HTTPS | Price and CO₂ data | see below | -| A Wattnet login (default) **or** a Nowtricity API key | CO₂ rating | Wattnet: email + password, registered at https://api.wattnet.eu/token-request/ (see [configuration.md](configuration.md#typewattnet-default)) · Nowtricity: https://www.nowtricity.com/ | +| Outbound HTTPS | CO₂ data (and price, only if you enable it) | see below | +| A Wattnet login | CO₂ rating | email + password, registered at https://api.wattnet.eu/token-request/ (see [configuration.md](configuration.md#wattnet)) | Additionally, for `control_mode = temperature`: @@ -46,9 +46,8 @@ config; set `https_proxy` in the cron environment if you need one. | Host | Purpose | Required? | |---|---|---| -| `spotovaelektrina.cz` | Current price and historical monthly averages | Yes: price rating | -| `www.nowtricity.com` | CO₂ intensity (alternative backend) | Only if `TYPE=nowtricity` | -| `api.wattnet.eu` | CO₂ intensity and the daily token login (default backend) | Only if `TYPE=wattnet` | +| `api.wattnet.eu` | CO₂ intensity and the daily token login | Yes | +| `spotovaelektrina.cz` | Current price and historical monthly averages | Only if you add the `price` plugin (not the default) | Pre-flight check: @@ -140,7 +139,7 @@ gh run download -n hpc-eff-deb # or hpc-eff-rpm | `/etc/cron.d/hpc-eff` | Created by `hpc-eff --enable` | > **Note:** neither the DEB nor the RPM enables cron on install. Nothing -> regulates until you run `sudo hpc-eff --enable`, after setting the API key +> regulates until you run `sudo hpc-eff --enable`, after setting the Wattnet login > and checking one manual `sudo hpc-eff` run. Upgrading keeps an existing > cron job. diff --git a/docs/monitoring.md b/docs/monitoring.md index e41f090..448c2ac 100644 --- a/docs/monitoring.md +++ b/docs/monitoring.md @@ -37,12 +37,12 @@ sudo sqlite3 /var/lib/hpc_eff/history.db ".schema hpc_eff_log" | `price` | CZK/MWh | `co2` | current spot price | | `co2_current` | gCO₂eq/kWh | `co2` | current grid carbon intensity | | `co2_median` | gCO₂eq/kWh | `co2` | median of the last 24 h | -| `co2_grade` | 1–10 | `co2` | CO₂ percentile grade, **TEXT column**, holds `unknown` when the API failed | +| `co2_grade` | 1–10 | `co2` | CO₂ percentile grade against the last 24 h (informational with `THRESHOLD`), **TEXT column**, holds `unknown` when the API failed | | `power_w` | W | `co2` | instantaneous node power from `POWERREADINGCMD` | | `cpu_freq_current` | MHz | `co2` | frequency observed *before* this run's change | -| `rating` | 1–10 | always | `co2`: the combined rating that chose `freq_max`. `temperature`: always `5`, an unused leftover; the thermal bands choose `freq_max` instead | +| `rating` | 1–10 | always | the rating that chose `freq_max`. With `THRESHOLD` (shipped): `10` throttled, `1` full speed. Price + CO₂ mode: the combined rating. `temperature`: always `5`, an unused leftover; the thermal bands choose `freq_max` instead | | `rating_price` | 1–10 | `co2` | price component; NULL if the price fetch failed | -| `rating_co2` | 1–10 | `co2` | CO₂ component; NULL if the CO₂ fetch failed | +| `rating_co2` | 1–10 | `co2` | CO₂ component. With `THRESHOLD`: 10 or 1, also when a recent decision is reused during an outage; NULL when there is no usable CO₂ result | | `temperature` | °C | `temperature` | the sensor reading | | `gpu_power_limit` | W | `temperature` | limit in effect this run (== `gpu_target_power`) | | `gpu_target_power` | W | `temperature` | limit this run aimed for | @@ -86,17 +86,17 @@ A JSON file, rewritten in full on every run: "score_name": "SPEC2017", "score_value": 3.8, "power_cmd": "ipmitool dcmi power reading", - "plugins": { "read": ["power", "cpufreq"], "score": ["price", "co2", "combine"], + "plugins": { "read": ["power", "cpufreq"], "score": ["co2"], "act": ["cpu_freq"], "record": ["db_log", "state_json"] } }, "current": { - "timestamp": "2026-08-21T10:00:03Z", - "rating": 5, "rating_price": 6, "rating_co2": 4, - "selected_freq_khz": 2600000, - "summary": "Price 2431 is average; Combined rating 5 (price 6, CO2 4)", - "action_price": "Price 2431 is average; Combined rating 5 (price 6, CO2 4)", - "price": 2431, "co2_current": 412, "co2_median": 448, "co2_grade": 4, - "power_w": 287, "cpu_freq_current": 2712 + "timestamp": "2026-10-26T09:44:32Z", + "rating": 10, "rating_co2": 10, + "co2_current": 437.2, "co2_median": 393.0, "co2_grade": 10, + "co2_threshold": 430.0, "co2_throttled": true, "co2_fallback": false, + "selected_freq_khz": 2200000, + "summary": "CO2 437.2 g/kWh, threshold 430 (release below 425): throttled (rating 10); Rating 10 set 2200000kHz", + "power_w": 287, "cpu_freq_current": 2998 }, "history": [ "…same shape, newest first, up to history_length entries…" ] } @@ -105,7 +105,13 @@ A JSON file, rewritten in full on every run: `current` is a copy of `history[0]`. `static` is refreshed every run, so it also serves as a liveness marker for the config the node is running with. -The example above is a `co2`-mode node. A `temperature`-mode node carries +The example above is a node with the shipped config, throttled at 437 g/kWh. +`co2_threshold` and `co2_throttled` exist only in `state.json`, not in the +database; `co2_throttled` is what the next run reads for the hysteresis. +`co2_fallback: true` means this run had no fresh CO₂ data and either reused a +recent decision or ran at full speed; `summary` says which. A price + CO₂ node +carries `price`, `rating_price` and the deprecated `action_price` as well. A +`temperature`-mode node carries `temperature`, `thermo_freq_limit`, and `action_temp` instead, plus the `gpu_*` fields on GPU nodes; the price and rating keys are absent entirely. diff --git a/docs/plugins.md b/docs/plugins.md index 1d53f0e..dcc20c0 100644 --- a/docs/plugins.md +++ b/docs/plugins.md @@ -30,15 +30,15 @@ omitted. A stage you leave out is empty, except `record`, which defaults to `db_log, state_json`. ```ini -# Wattnet (or Nowtricity) CO2 alone drives the CPU cap. No price fetch. +# Wattnet CO2 alone drives the CPU cap. No price fetch. [PIPELINE] score = co2 act = cpu_freq ``` On its own, `co2` makes its grade the `rating`. Add `combine` only when `price` -is listed too, to choose how the two mix (`[aggregation]`). To pick the data -source, set `[CO2_API] TYPE = wattnet`. +is listed too, to choose how the two mix (`[aggregation]`). The CO₂ data comes +from Wattnet (`[CO2_API]`). An invalid pipeline (unknown key, unknown plugin, a plugin listed under the wrong stage, a bad executable) aborts the run before anything starts, with @@ -54,10 +54,12 @@ A plugin is any executable. Declare it in a `[plugin:NAME]` section and list score = co2, threshold_score act = cpu_freq, echo_action +# timeout: seconds, default 30. Any other key is passed to the plugin as-is. +# Comments must be on their own line: "30 # note" would be the value. [plugin:threshold_score] exec = /etc/hpc_eff/plugins/threshold_score.py -timeout = 30 # seconds, default 30 -low = 150 # any other key is passed to the plugin as-is +timeout = 30 +low = 150 high = 450 [plugin:echo_action] @@ -70,8 +72,10 @@ the pipeline. `exec = builtin:NAME` selects a built-in explicitly. Working examples are in [`examples/plugins/`](../examples/plugins), installed with the package under `/usr/share/doc/hpc-eff/examples/` (RPM: `/usr/share/doc/hpc_eff/examples/`): -`template.py` (a blank starter), `threshold_score.py` (score from absolute -gCO₂/kWh thresholds) and `echo_action.py` (an action that changes nothing). +`template.py` (a blank starter), `threshold_score.py` (a linear rating between +two gCO₂/kWh values; an example of an executable score plugin; for a plain +threshold use the built-in `[CO2_API] THRESHOLD` instead) and `echo_action.py` +(an action that changes nothing). ### Installing a plugin, step by step @@ -220,7 +224,7 @@ is how a plugin listed after `combine` can override the built-in rating. | `power` | read | `[SYSTEM] POWERREADINGCMD` | `power_w` | | `cpufreq` | read | the running CPU frequency | `cpu_freq_current` | | `price` | score | electricity price vs. a year of monthly averages | `price`, `rating_price` | -| `co2` | score | `[CO2_API]` (Nowtricity or Wattnet), graded against the last 24 h | `co2_current`, `co2_median`, `co2_grade`, `rating_co2` | +| `co2` | score | `[CO2_API]` (Wattnet): with `THRESHOLD`, rating 10 above it and 1 below (hysteresis `HYSTERESIS`); without, graded against the last 24 h | `co2_current`, `co2_median`, `co2_grade`, `rating_co2`, `co2_threshold`, `co2_throttled` | | `combine` | score | `rating_price`, `rating_co2`, `[aggregation]` | `rating` | | `cpu_freq` | act | `rating`, `[frequency_tables]` | `selected_freq_khz`, `freq_max` | | `cpu_thermo` | act | `[TEMPERATURE_SOURCE]`, `[CPU_THERMO]` | `temperature`, `freq_max` | diff --git a/docs/regulation-modes.md b/docs/regulation-modes.md index a0d163d..c85df50 100644 --- a/docs/regulation-modes.md +++ b/docs/regulation-modes.md @@ -1,5 +1,11 @@ # Regulation modes +> **The shipped default does not use `[MODE]`.** `config.ini.example` contains a +> `[PIPELINE]` (`score = co2`, `act = cpu_freq`) with `[CO2_API] THRESHOLD`: the +> CPU is throttled only while Wattnet CO₂ is above the threshold, see +> [configuration.md](configuration.md#threshold-when-to-throttle). This page +> describes the `[MODE] control_mode` presets used when `[PIPELINE]` is removed. + `[MODE] control_mode` is the one switch that decides how a node is regulated. There are two values, and they are mutually exclusive by construction: a node is driven either by carbon/price or by temperature, never by both. @@ -16,7 +22,7 @@ than running a no-op cycle. ## `co2` mode -Deployment walkthrough: [deployment.md](deployment.md#mode-co2). +Deployment walkthrough: [deployment.md](deployment.md#1-configure). This section explains what the numbers mean. ### The pipeline diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index d8fb530..44887a3 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -18,8 +18,10 @@ Remember this is a real run: it applies a frequency change. Config error: [MODE] control_mode must be one of: temperature|co2 (got None). ``` -`[MODE] control_mode` is unset or misspelled. It is required and there is no -default. Check the section header spelling too: configparser will not warn you +There is no `[PIPELINE]` section and `[MODE] control_mode` is unset or +misspelled. Without `[PIPELINE]` it is required and there is no default. The +shipped config has a `[PIPELINE]`; if yours lacks it, copy that section from +`/etc/hpc_eff/config.ini.example`. Check the section header spelling too: configparser will not warn you about a `[MODEL]` typo; it finds no key and the value reads as unset. --- @@ -97,8 +99,6 @@ Test each source directly: ```bash curl -v https://spotovaelektrina.cz/api/v1/price/get-actual-price-czk -curl -v -H "X-Api-Key: $KEY" -H "User-Agent: test" \ - https://www.nowtricity.com/api/current-emissions/czech-republic/ # Wattnet: log in, then query with the returned token curl -X POST https://api.wattnet.eu/token-request/get_token \ -H "Content-Type: application/json" -d '{"email": "", "password": ""}' @@ -124,13 +124,12 @@ is why the node keeps working and nothing looks broken. | Backend | Likely cause | |---|---| -| `nowtricity` | Missing/expired `NOWTRICITY_API_KEY`, or the quota is exhausted. Each run makes **two** calls: 288/day/node at a 10-minute interval. | -| `nowtricity` | Wrong `NOWTRICITY_ZONE` slug. | +| old config | `TYPE=nowtricity is no longer supported`: an upgraded node kept its old config. Delete `TYPE` and the `NOWTRICITY_*` keys, add the Wattnet login. | | `wattnet` | `Wattnet needs WATTNET_EMAIL and WATTNET_PASSWORD`: the login is not set. | | `wattnet` | `Wattnet login rejected`: wrong email/password, or the account is not registered at [api.wattnet.eu/token-request](https://api.wattnet.eu/token-request/). | | `wattnet` | `Wattnet rejected a freshly issued token`: login works but the account has no API access. Contact Wattnet. (A rejected *cached* token is handled automatically: Wattnet answers it with a redirect to its login page, and ThrottlePoint logs in again once.) | | `wattnet` | Fewer than 24 hourly values returned; the call raises rather than working with a partial window. | -| both | Egress blocked to that host specifically. | +| `wattnet` | Egress blocked to `api.wattnet.eu` specifically. | On a large cluster, check the API quota before blaming the code: hundreds of nodes each polling every 10 minutes adds up fast. @@ -240,8 +239,8 @@ sudo nvidia-smi -i 0 -pl 200 # does a manual set work? `hpc-eff` reads `/etc/hpc_eff/config.ini`, and **falls back to `src/hpc_eff/config.ini.example` relative to the current working directory** if that file is absent. Running from a git checkout with no installed config picks -up the example, with a placeholder API key and whatever mode the example -ships. +up the example, with an empty Wattnet login, so every run reports +`Wattnet needs WATTNET_EMAIL and WATTNET_PASSWORD` and runs at full speed. ```bash ls -l /etc/hpc_eff/config.ini diff --git a/src/hpc_eff/config.ini.example b/src/hpc_eff/config.ini.example index 64750df..32bd2dc 100644 --- a/src/hpc_eff/config.ini.example +++ b/src/hpc_eff/config.ini.example @@ -1,189 +1,51 @@ -[SYSTEM] -SCORENAME=SPEC2017 -SCORE=3.8 -POWERREADINGCMD=ipmitool dcmi power reading -DEBUG=yes +# ThrottlePoint configuration. Reference: docs/configuration.md +# +# Setup: set WATTNET_EMAIL and WATTNET_PASSWORD, then +# chmod 600 /etc/hpc_eff/config.ini +# hpc-eff # test run +# hpc-eff --enable # install the cron job [CO2_API] -# Which CO2 intensity API to use: wattnet (default) | nowtricity -TYPE=wattnet - -# Nowtricity API credentials (https://www.nowtricity.com/) -NOWTRICITY_USER_AGENT=UserAgentString -NOWTRICITY_API_KEY=*** -NOWTRICITY_BASE_URL=https://www.nowtricity.com/api -NOWTRICITY_ZONE=czech-republic - -# Wattnet API settings (used only when TYPE=wattnet) -# Start/end timestamps are calculated automatically for the trailing 24h -WATTNET_URL=https://api.wattnet.eu/v1/footprints -# Login. Wattnet has no API keys: tokens last 24 h and cannot be refreshed, -# so ThrottlePoint requests one with these credentials and caches it in -# WATTNET_TOKEN_CACHE until shortly before it expires. Wattnet suggests a -# dedicated account (the email need not be a real mailbox), not a personal one. +# Wattnet account (https://api.wattnet.eu/token-request/) WATTNET_EMAIL= WATTNET_PASSWORD= -#WATTNET_TOKEN_CACHE=/var/lib/hpc_eff/wattnet_token.json + +# Throttle above this CO2 intensity (gCO2/kWh, 1 h average) +THRESHOLD=430 +# Release the throttle only below THRESHOLD - HYSTERESIS +HYSTERESIS=5 + WATTNET_ZONE=CZ WATTNET_FOOTPRINT_TYPE=carbon WATTNET_SCOPE=operational +WATTNET_URL=https://api.wattnet.eu/v1/footprints +#WATTNET_TOKEN_CACHE=/var/lib/hpc_eff/wattnet_token.json -# When the CO2 API fails, reuse the last real grade if it is at most this many -# minutes old (data is hourly, so 120 = one missed hour plus slack). Older or -# none: rating 1, i.e. no slowdown without evidence. +# Minutes to keep the last decision if Wattnet is unreachable MAX_AGE_MIN=120 -[logging] -db_path=/var/lib/hpc_eff/history.db -log_level=INFO -state_json_path=/var/lib/hpc_eff/state.json -history_length=1000 - -[MODE] -# The SINGLE switch that selects how the node is regulated. Required. -# -# temperature -> CPU thermal control (see [CPU_THERMO]) + NVIDIA GPU power -# regulation (see [GPU_POWER]) -# co2 -> CPU price/CO2 frequency control only (rating 1-10 -> max freq) -# -# CPU regulation is mutually exclusive (temperature OR price), so the two modes -# never overlap. An unset or unknown value aborts the run. -control_mode=co2 - [frequency_tables] -# Per-CPU-model frequency tables (MHz), highest first. Used by the price/CO2 -# regulator to map a rating (1-10) onto a CPU max frequency. -# AMD EPYC +# CPU max frequency in MHz, highest first. +# First value = full speed, last value = throttled. amd_epyc=2400,2000,1500 - -# Intel Xeon Gold 6130 intel_xeon_gold=3000,2900,2800,2700,2600,2200 - -# Intel E5-2650v3 intel_e5=3000,2800,2700,2600,2400,2300 - -# Fallback for unknown model default=3000,2800,2700,2600,2400,2200 -[TEMPERATURE_SOURCE] -# Where to read temperature from. TYPE can be: ipmi | http_api | custom - -# Example 1: IPMI sensor via ipmitool -TYPE=ipmi -IPMI_SENSOR_NAME=INLET_AIR_TEMP - -# Example 2: HTTP API with optional JSON extraction (GreenDIGIT style) -# TYPE=http_api -# HTTP_URL=http://192.168.1.100/api/greendigit/data -# HTTP_JSON_PATH=temperature.value -# If the response is {"temperature": {"value": 25.5}, ...} then -# HTTP_JSON_PATH=temperature.value extracts 25.5. Omit it if the endpoint -# returns a plain number. - -# Example 3: Custom reader ("bring your own reader") -# TYPE=custom -# MODULE=/opt/mysensors/my_reader.py # path to a .py file OR an importable dotted module name -# FUNCTION=read # callable returning Celsius as a number; defaults to "read" - -[CPU_THERMO] -# Temperature bands (degrees Celsius) and the target frequency for each band. -MID_LIMIT=29 -HIGH_LIMIT=32 - -# Target frequencies (string format, e.g. "3.10GHz") -HIGH_FREQUENCY=3.10GHz -MID_FREQUENCY=2.30GHz -LOW_FREQUENCY=1.50GHz - -# Optional Slack webhook for thermal change notifications. -#SLACK_URL= - -# ============================================================================ -# GPU POWER REGULATION (NVIDIA GPUs only) -# ============================================================================ - -[GPU_POWER] -# Uses GLOBAL ambient temperature from [TEMPERATURE_SOURCE] (same as CPU_THERMO) -# All GPUs get the SAME power limit based on this temperature. - -# Temperature thresholds (degrees Celsius) -# MID_LIMIT: Start reducing power when temp exceeds this (default: 70°C) -# HIGH_LIMIT: Aggressive power reduction when temp exceeds this (default: 80°C) -MID_LIMIT=70 -HIGH_LIMIT=80 - -# Power limits - can be absolute (Watts) or percentage of max (%) -# HIGH_POWER: Normal operation (below MID_LIMIT) - default: 100% -# MID_POWER: Reduced power (between MID_LIMIT and HIGH_LIMIT) - default: 70% -# LOW_POWER: Minimum power (above HIGH_LIMIT) - default: 40% -# -# Examples for different GPU types: -# A100/H100 (absolute): HIGH_POWER=300, MID_POWER=200, LOW_POWER=100 -# A40/RTX (percentage): HIGH_POWER=100%, MID_POWER=70%, LOW_POWER=40% -HIGH_POWER=100% -MID_POWER=70% -LOW_POWER=40% - -# GPU regulation is logged together with CPU data in the unified hpc_eff_log -# table of [logging] db_path -- there is no separate GPU database. - -# Optional Slack webhook for power change notifications -# SLACK_URL=https://hooks.slack.com/services/XXX/YYY/ZZZ - -# ============================================================================ -# RATING AGGREGATION (price + CO2) -# ============================================================================ - -[aggregation] -# How the final rating (1-10) that drives the CPU max frequency is computed: -# price - price rating only (legacy behaviour; used when section is absent) -# average - weighted average of the price and CO2 ratings -# max - the worse (higher) of the two ratings ("throttle if either is bad") -# -# If the CO2 API is unavailable, "average" and "max" fall back to the price -# rating so the node keeps regulating. -rating_type=average - -# Weights for rating_type=average (normalized by their sum, so they need not -# add up to 1). Price is weighted higher because the two ratings are computed -# against different baselines: the price rating compares against a full year -# of monthly averages and is a stable, absolute signal, while the CO2 grade is -# a percentile within only the last 24 hours and therefore sweeps the whole -# 1-10 range every day - even on a uniformly clean day. Weighting CO2 equal or -# higher would let that day-relative volatility dominate the frequency cap. -weight_price=0.6 -weight_co2=0.4 - -# ============================================================================ -# PLUGIN PIPELINE (optional; replaces the [MODE] control_mode presets) -# ============================================================================ -# Each evaluation runs four stages: read -> score -> act -> record. Every step -# is a plugin: a built-in, or your own executable (see docs/plugins.md). -# Without a [PIPELINE] section the plan follows [MODE] control_mode and behaves -# exactly as before. -# -# Built-ins: read: power, cpufreq | score: price, co2, combine | -# act: cpu_freq, cpu_thermo, gpu_power | record: db_log, state_json -# -# DEFAULT: regulate by CO2 alone (Nowtricity or Wattnet per [CO2_API]), no price. -# While this section is present, [MODE] control_mode is ignored. To use the -# price+CO2 blend or temperature mode instead, comment this section out. - [PIPELINE] read = power, cpufreq score = co2 act = cpu_freq -# Example: swap in your own executable. A [plugin:NAME] section with `exec` -# (absolute path, root-owned, not group/world-writable) overrides a built-in -# of the same name or defines a new plugin. Other keys are passed to the plugin. -# -#[PIPELINE] -#score = co2, threshold_score -#act = cpu_freq -# -#[plugin:threshold_score] -#exec = /etc/hpc_eff/plugins/threshold_score.py -#timeout = 30 -#low = 150 -#high = 450 +[logging] +db_path=/var/lib/hpc_eff/history.db +state_json_path=/var/lib/hpc_eff/state.json +history_length=1000 +log_level=INFO + +[SYSTEM] +# Power reading for the log only +POWERREADINGCMD=ipmitool dcmi power reading +SCORENAME=SPEC2017 +SCORE=3.8 +DEBUG=yes diff --git a/src/hpc_eff/pipeline/builtins.py b/src/hpc_eff/pipeline/builtins.py index bba0756..8e1b27b 100644 --- a/src/hpc_eff/pipeline/builtins.py +++ b/src/hpc_eff/pipeline/builtins.py @@ -148,37 +148,92 @@ def price(req, config, rt): return {"set": values, "state": state} -def _last_fresh_co2_grade(config): - """(grade, age in minutes) of the newest real CO2 grade in state.json, or None. - - Grid CO2 is published hourly, so a grade up to [CO2_API] MAX_AGE_MIN old - (default 120) still describes the grid; older ones are a guess. - """ - max_age = config.getint("CO2_API", "MAX_AGE_MIN", fallback=120) +def _state_history(config): + """state.json history (newest first), or [] when missing or unreadable.""" path = config.get("logging", "state_json_path", fallback=DEFAULT_STATE_PATH) try: with open(path, "r") as f: - history = json.load(f).get("history", []) + return json.load(f).get("history", []) except Exception: - return None - now = datetime.now(timezone.utc) - for entry in history: + return [] + + +def _last_fresh_co2_grade(config): + """(entry, age in minutes) of the newest real CO2 evaluation in state.json, or None. + + Grid CO2 is published hourly, so a result up to [CO2_API] MAX_AGE_MIN old + (default 120) still describes the grid; older ones are a guess. + """ + max_age = _number(config, "MAX_AGE_MIN", 120, minimum=0, integer=True) + for entry in _state_history(config): if entry.get("co2_fallback") or not isinstance(entry.get("co2_grade"), int): continue - try: - ts = datetime.strptime(entry["timestamp"], "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=timezone.utc) - except (KeyError, ValueError): + age_min = _entry_age_min(entry) + if age_min is None: continue - age_min = int((now - ts).total_seconds() // 60) if age_min <= max_age: - return entry["co2_grade"], age_min + return entry, age_min return None return None +def _number(config, key, default, minimum=None, integer=False): + """[CO2_API] as a finite number >= minimum; a clear ValueError naming the key otherwise.""" + raw = config.get("CO2_API", key, fallback="").strip() + if not raw: + return default + try: + value = int(raw) if integer else float(raw) + except ValueError: + raise ValueError(f"[CO2_API] {key}={raw!r} is not a number") + if value != value or value in (float("inf"), float("-inf")): + raise ValueError(f"[CO2_API] {key}={raw!r} is not a finite number") + if minimum is not None and value < minimum: + raise ValueError(f"[CO2_API] {key}={raw!r} must be at least {minimum}") + return value + + +def _threshold(config): + """[CO2_API] THRESHOLD in gCO2/kWh, or None when unset (24 h ranking mode).""" + return _number(config, "THRESHOLD", None) + + +def _entry_age_min(entry): + """Minutes since a state.json entry was written, or None when its timestamp is unusable.""" + try: + ts = datetime.strptime(entry["timestamp"], "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=timezone.utc) + except (KeyError, ValueError, TypeError): + return None + return int((datetime.now(timezone.utc) - ts).total_seconds() // 60) + + +def _throttle_decision(current, threshold, hysteresis, was_throttled): + """Throttle above `threshold`; once throttled, release only below threshold - hysteresis.""" + if was_throttled: + return current > threshold - hysteresis + return current > threshold + + def co2(req, config, rt): - """CO2 intensity (Nowtricity or Wattnet per [CO2_API]), graded 1-10 as a 24 h percentile.""" + """CO2 intensity from Wattnet ([CO2_API]) -> rating. + + With [CO2_API] THRESHOLD set: two levels. Above the threshold the rating is + 10 (lowest frequency in the table), otherwise 1 (full table maximum), with + HYSTERESIS g/kWh before releasing. Without it: the 24 h percentile grade. + """ co2_error = None + try: + threshold = _threshold(config) + # HYSTERESIS only matters with a threshold; a typo must not break 24 h mode + hysteresis = _number(config, "HYSTERESIS", 5.0, minimum=0) if threshold is not None else 0.0 + max_age = _number(config, "MAX_AGE_MIN", 120, minimum=0, integer=True) + except ValueError as e: + # A typo must not leave cpu_freq on its neutral default (a mid-table cap) + logging.getLogger("hpc_eff").warning("CO2 config error: %s", e) + summary = f"CO2 config error ({' '.join(str(e).split())[:160]}); rating 1 (no slowdown)" + return {"set": {"rating": 1}, + "state": {"rating": 1, "co2_fallback": True, "summary": summary}} + throttled = None try: history, current, median, grade = co2_value(config) rt.debug_log(f"Last 24 hours CO2 values (g CO2eq/kWh): {history}") @@ -193,12 +248,26 @@ def co2(req, config, rt): reason = " ".join(str(e).split())[:160] co2_error = f"CO2 unavailable ({type(e).__name__}: {reason})" last = _last_fresh_co2_grade(config) - if last is not None: - grade, age_min = last + if (last is not None and threshold is not None and isinstance(last[0].get("co2_throttled"), bool) + and last[0].get("co2_threshold") == threshold): + entry, age_min = last + throttled = entry["co2_throttled"] + co2_error += (f"; reusing the {'throttled' if throttled else 'full speed'} decision " + f"from {age_min} min ago") + elif last is not None and threshold is None: + entry, age_min = last + grade = entry["co2_grade"] co2_error += f"; reusing grade {grade} from {age_min} min ago" else: co2_error += "; no CO2 data within max age, rating 1 (no slowdown)" + if threshold is not None and current is not None: + previous = (_state_history(config) or [{}])[0] + age = _entry_age_min(previous) + # Carry the throttle over only from a recent run (cron off for days = fresh start) + was = bool(previous.get("co2_throttled")) and age is not None and age <= max_age + throttled = _throttle_decision(current, threshold, hysteresis, was) + values = { "co2_current": current, "co2_median": median, @@ -206,11 +275,21 @@ def co2(req, config, rt): # CO2 grade (1 low - 10 high) doubles as the CO2 rating "rating_co2": grade if isinstance(grade, int) else None, } + if threshold is not None: + # Threshold mode: the rating is the throttle decision, not the 24 h grade + values["rating_co2"] = None if throttled is None else (10 if throttled else 1) state = dict(values) + if threshold is not None: + state["co2_threshold"] = threshold + state["co2_throttled"] = throttled # A fallback grade is never reused itself, so a long outage cannot chain it forward state["co2_fallback"] = co2_error is not None if co2_error: state["summary"] = co2_error + elif threshold is not None: + verdict = "throttled (rating 10)" if throttled else "full speed (rating 1)" + state["summary"] = (f"CO2 {current:g} g/kWh, threshold {threshold:g}" + f" (release below {threshold - hysteresis:g}): {verdict}") if values["rating_co2"] is not None: # On its own, the CO2 grade is the rating. `combine`, when listed, overrides it. values["rating"] = state["rating"] = values["rating_co2"] diff --git a/src/hpc_eff/utils/co2_value.py b/src/hpc_eff/utils/co2_value.py index efdfb74..b85b974 100644 --- a/src/hpc_eff/utils/co2_value.py +++ b/src/hpc_eff/utils/co2_value.py @@ -12,47 +12,6 @@ TOKEN_REFRESH_MARGIN = timedelta(minutes=10) -def _nowtricity_url(config, endpoint): - """Build Nowtricity API URL from config: BASE_URL/endpoint/ZONE/""" - base = config.get('CO2_API', 'NOWTRICITY_BASE_URL', - fallback='https://www.nowtricity.com/api').rstrip('/') - zone = config.get('CO2_API', 'NOWTRICITY_ZONE', fallback='czech-republic') - return f'{base}/{endpoint}/{zone}/' - - -def fetch_emissions_data(api_headers, config): - """ - Fetch emissions data for the past 24 hours from the Nowtricity API. - - Returns: - list of int: A list of emission values (in g CO2eq/kWh). - """ - url = _nowtricity_url(config, 'emissions-previous-24h') - - response = requests.get(url, headers=api_headers, timeout=10) - response.raise_for_status() - data = response.json() - - # Extract emission values - return [entry['value'] for entry in data['emissions']] - - -def fetch_current_emission(api_headers, config): - """ - Fetch the current emission value from the Nowtricity API. - - Returns: - int: The current emission value (in g CO2eq/kWh). - """ - url = _nowtricity_url(config, 'current-emissions') - - response = requests.get(url, headers=api_headers, timeout=10) - response.raise_for_status() - data = response.json() - - return data['emissions']['value'] - - def _jwt_expiry(token): """The `exp` claim of a JWT as an aware datetime, or None. Not verified: only used for caching.""" try: @@ -98,8 +57,9 @@ def _wattnet_token(config, force_refresh=False): cached in WATTNET_TOKEN_CACHE (mode 600) until shortly before it expires, as Wattnet asks. """ - email = config.get('CO2_API', 'WATTNET_EMAIL', fallback='').strip() - password = config.get('CO2_API', 'WATTNET_PASSWORD', fallback='').strip() + # raw=True: ConfigParser would otherwise treat '%' in a password as interpolation + email = config.get('CO2_API', 'WATTNET_EMAIL', fallback='', raw=True).strip() + password = config.get('CO2_API', 'WATTNET_PASSWORD', fallback='', raw=True).strip() if not (email and password): raise ValueError("Wattnet needs WATTNET_EMAIL and WATTNET_PASSWORD in [CO2_API]") @@ -220,10 +180,14 @@ def rejected(response): for ts, value in merged.items(): by_hour.setdefault(ts[:13], []).append(value) - # Newest hour first (to match Nowtricity order), last 24 hours. The newest - # hour may be partial (e.g. 12:00-12:30); its average is the "current" value. + # Newest clock hour first, last 24 hours (the 24 h history for grading) result = [round(sum(v) / len(v)) for _, v in sorted(by_hour.items(), reverse=True)][:24] - current_hourly = result[0] if result else None + + # "Current" is the trailing 1 h average: the newest four 15-minute points, + # the signal the GreenDIGIT threshold simulator uses. Unlike the newest + # clock hour it never shrinks to a single point just after the hour. + newest = [merged[ts] for ts in sorted(merged)[-4:]] + current_hourly = round(sum(newest) / len(newest), 1) if len(result) < 24: raise ValueError(f"Wattnet API returned only {len(result)}/24 hourly values") @@ -256,7 +220,7 @@ def assign_grade(current_value, historical_values): def co2_value(config): """ - Main execution function - unified for both Nowtricity and Wattnet APIs. + CO2 intensity from Wattnet, graded against the last 24 hours. Args: config: configparser.ConfigParser instance with [CO2_API] section @@ -268,22 +232,17 @@ def co2_value(config): - median_value: median of historical_values - grade: 1-10 rating based on percentile """ + # Wattnet is the only backend. An old config still naming another one + # (e.g. TYPE=nowtricity, kept as a conffile across upgrades) fails loudly + # instead of silently switching source. api_type = config.get('CO2_API', 'TYPE', fallback='wattnet').strip().lower() + if api_type != 'wattnet': + raise ValueError(f"[CO2_API] TYPE={api_type} is no longer supported: Nowtricity was " + "removed, set WATTNET_EMAIL and WATTNET_PASSWORD and drop TYPE") - if api_type == "wattnet": - historical_values, current_hourly = fetch_wattnet_24h(config) - # For current value, use the most recent hourly average - # (consistent with nowtricity where current ≈ history[0]) - current_value = current_hourly if current_hourly is not None else historical_values[0] - else: - # Nowtricity: read credentials from [CO2_API] - headers = { - 'User-Agent': config.get('CO2_API', 'NOWTRICITY_USER_AGENT', fallback='HPC-Eff-Agent'), - 'X-Api-Key': config.get('CO2_API', 'NOWTRICITY_API_KEY') - } - - historical_values = fetch_emissions_data(headers, config) - current_value = fetch_current_emission(headers, config) + historical_values, current_hourly = fetch_wattnet_24h(config) + # The most recent hourly average is the "current" value + current_value = current_hourly if current_hourly is not None else historical_values[0] median_value = statistics.median(historical_values) grade = assign_grade(current_value, historical_values) diff --git a/tests/test_pipeline.py b/tests/test_pipeline.py index b8902c4..2a3fe01 100644 --- a/tests/test_pipeline.py +++ b/tests/test_pipeline.py @@ -76,8 +76,8 @@ def test_current_is_newest_hour(self): history, current, _median, grade = co2_mod.co2_value(cfg) self.assertEqual(len(history), 24) - self.assertEqual(current, history[0]) - self.assertGreater(current, 190) # newest hour (was ~102 before the fix) + self.assertEqual(current, 193.5) # trailing 1 h: mean of the newest four points 192..195 + self.assertGreater(current, 190) # newest data (was ~102 before the fix) self.assertEqual(grade, 10) @@ -100,7 +100,7 @@ def test_merges_all_series_and_groups_by_clock_hour(self): mock.patch.object(co2_mod.requests, "get", return_value=response): history, current = co2_mod.fetch_wattnet_24h(cfg) self.assertEqual(len(history), 24) - self.assertEqual(current, 300) # 12:00-12:30 preview hour, not 08:30 + self.assertEqual(current, (320 + 300 * 3) / 4) # trailing 1 h: 11:45 pending + 12:00-12:30 preview, not 08:30 self.assertEqual(history[1], 320) # 11:00 hour, pending self.assertEqual(history[4], round((400 * 3 + 320) / 4)) # 08:00 hour: final 08:00-08:30 + pending 08:45 @@ -117,6 +117,14 @@ def test_final_value_wins_on_overlap(self): self.assertEqual(current, 100) +class RemovedBackendTest(unittest.TestCase): + def test_old_nowtricity_config_fails_loudly(self): + cfg = make_config("[CO2_API]\nTYPE=nowtricity\nNOWTRICITY_API_KEY=x\n") + with mock.patch.object(co2_mod.requests, "get", side_effect=AssertionError("network used")): + with self.assertRaisesRegex(ValueError, "Nowtricity was removed"): + co2_mod.co2_value(cfg) + + class WattnetTokenTest(TempDirCase): """Login, cache and refresh of the 24 h Wattnet token.""" @@ -130,6 +138,14 @@ def login_response(self, token="T1", hours=24): "expires_at": (datetime.now(timezone.utc) + timedelta(hours=hours)).isoformat()} return r + def test_percent_in_password_is_sent_verbatim(self): + cfg = configparser.ConfigParser() # the interpolating parser main.py uses + cfg.read_string(f"[CO2_API]\nWATTNET_EMAIL=bot@x\nWATTNET_PASSWORD=Ab%9xyz%%\n" + f"WATTNET_TOKEN_CACHE={self.tmp / 'tok.json'}\n") + with mock.patch.object(co2_mod.requests, "post", return_value=self.login_response()) as post: + co2_mod._wattnet_token(cfg) + self.assertEqual(post.call_args.kwargs["json"], {"email": "bot@x", "password": "Ab%9xyz%%"}) + def test_login_then_cache_hit(self): with mock.patch.object(co2_mod.requests, "post", return_value=self.login_response()) as post: self.assertEqual(co2_mod._wattnet_token(self.cfg()), "T1") @@ -395,6 +411,120 @@ def test_co2_failure_with_price_still_falls_back_to_price(self): "score = price, co2, combine\nact = cpu_freq\n", co2=RuntimeError("401")) self.assertEqual(self.set_freq.call_args[0][0], 8) # price only, no fake CO2 = 1 + # ---- threshold mode ([CO2_API] THRESHOLD) + TH = "[CO2_API]\nTHRESHOLD=430\nHYSTERESIS=5\n[PIPELINE]\nscore = co2\nact = cpu_freq\n" + + def seed_entries(self, *entries): + """entries: (minutes_ago, extra_fields), newest first.""" + now = datetime.now(timezone.utc) + history = [dict({"timestamp": (now - timedelta(minutes=m)).strftime("%Y-%m-%dT%H:%M:%SZ")}, **f) + for m, f in entries] + self.state.write_text(json.dumps({"static": {}, "history": history, "current": history[0]})) + + def co2_now(self, value): + return (list(range(300, 324)), value, 310, 7) # percentile grade 7 must be ignored + + def test_threshold_below_runs_full_speed(self): + self.run_cfg(self.TH, co2=self.co2_now(429.9)) + self.assertEqual(self.set_freq.call_args[0][0], 1) + e = self.entry() + self.assertIs(e["co2_throttled"], False) + self.assertEqual(e["co2_threshold"], 430) + self.assertIn("full speed", e["summary"]) + + def test_threshold_above_throttles_to_table_minimum(self): + self.run_cfg(self.TH, co2=self.co2_now(430.1)) + self.assertEqual(self.set_freq.call_args[0][0], 10) # rating 10 = lowest table value + self.assertIs(self.entry()["co2_throttled"], True) + + def test_threshold_exactly_equal_is_not_above(self): + self.run_cfg(self.TH, co2=self.co2_now(430)) + self.assertEqual(self.set_freq.call_args[0][0], 1) + + def test_hysteresis_holds_throttle_until_below_release(self): + self.seed_entries((10, {"co2_throttled": True, "co2_grade": 9})) + self.run_cfg(self.TH, co2=self.co2_now(425.5)) # above 430 - 5 + self.assertEqual(self.set_freq.call_args[0][0], 10) + + def test_hysteresis_releases_below_threshold_minus_hysteresis(self): + self.seed_entries((10, {"co2_throttled": True, "co2_grade": 9})) + self.run_cfg(self.TH, co2=self.co2_now(424.9)) + self.assertEqual(self.set_freq.call_args[0][0], 1) + + def test_hysteresis_does_not_apply_on_the_way_up(self): + self.seed_entries((10, {"co2_throttled": False, "co2_grade": 3})) + self.run_cfg(self.TH, co2=self.co2_now(428)) + self.assertEqual(self.set_freq.call_args[0][0], 1) + + def test_threshold_outage_reuses_recent_decision(self): + self.seed_entries((30, {"co2_throttled": True, "co2_threshold": 430.0, "co2_grade": 9, "co2_fallback": False})) + self.run_cfg(self.TH, co2=RuntimeError("401")) + self.assertEqual(self.set_freq.call_args[0][0], 10) + self.assertIn("reusing the throttled decision from 30 min ago", self.entry()["summary"]) + + def test_threshold_outage_stale_decision_runs_full_speed(self): + self.seed_entries((121, {"co2_throttled": True, "co2_grade": 9, "co2_fallback": False})) + self.run_cfg(self.TH, co2=RuntimeError("401")) + self.assertEqual(self.set_freq.call_args[0][0], 1) + + def test_threshold_outage_ignores_old_percentile_grade(self): + # an entry from before THRESHOLD was configured has a grade but no decision + self.seed_entries((10, {"co2_grade": 9, "co2_fallback": False})) + self.run_cfg(self.TH, co2=RuntimeError("401")) + self.assertEqual(self.set_freq.call_args[0][0], 1) + + def test_threshold_fallback_decision_is_not_chained(self): + self.seed_entries((10, {"co2_throttled": True, "co2_fallback": True, "co2_grade": "unknown"}), + (130, {"co2_throttled": True, "co2_grade": 9, "co2_fallback": False})) + self.run_cfg(self.TH, co2=RuntimeError("401")) + self.assertEqual(self.set_freq.call_args[0][0], 1) + + def test_bad_threshold_value_runs_full_speed_with_reason(self): + self.run_cfg("[CO2_API]\nTHRESHOLD=430g\n[PIPELINE]\nscore = co2\nact = cpu_freq\n", + co2=self.co2_now(500)) + self.assertEqual(self.set_freq.call_args[0][0], 1) # not cpu_freq's neutral 5 + self.assertIn("THRESHOLD='430g' is not a number", self.entry()["summary"]) + + def test_outage_ignores_decision_made_under_another_threshold(self): + self.seed_entries((30, {"co2_throttled": True, "co2_threshold": 400.0, "co2_grade": 9, "co2_fallback": False})) + self.run_cfg(self.TH, co2=RuntimeError("401")) + self.assertEqual(self.set_freq.call_args[0][0], 1) + + def test_stale_throttle_does_not_hold_hysteresis(self): + # cron was off for 3 days: 428 g is inside (425, 430] but this is a fresh start + self.seed_entries((3 * 24 * 60, {"co2_throttled": True, "co2_threshold": 430.0, "co2_grade": 9})) + self.run_cfg(self.TH, co2=self.co2_now(428)) + self.assertEqual(self.set_freq.call_args[0][0], 1) + + def test_bad_settings_run_full_speed_and_name_the_key(self): + for line, key in (("THRESHOLD=nan", "THRESHOLD"), ("THRESHOLD=inf", "THRESHOLD"), + ("HYSTERESIS=-5", "HYSTERESIS"), ("HYSTERESIS=5g", "HYSTERESIS"), + ("MAX_AGE_MIN=120min", "MAX_AGE_MIN"), ("MAX_AGE_MIN=-1", "MAX_AGE_MIN")): + with self.subTest(line=line): + self.logs.clear() + self.run_cfg(f"[CO2_API]\nTHRESHOLD=430\n{line}\n[PIPELINE]\nscore = co2\nact = cpu_freq\n" + if not line.startswith("THRESHOLD") else + f"[CO2_API]\n{line}\n[PIPELINE]\nscore = co2\nact = cpu_freq\n", + co2=self.co2_now(500)) + self.assertEqual(self.set_freq.call_args[0][0], 1) + self.assertIn(f"[CO2_API] {key}=", self.entry()["summary"]) + + def test_hysteresis_typo_does_not_break_24h_mode(self): + self.run_cfg("[CO2_API]\nHYSTERESIS=5g\n[PIPELINE]\nscore = co2\nact = cpu_freq\n", co2=self.co2_now(500)) + self.assertEqual(self.set_freq.call_args[0][0], 7) # the percentile grade, unaffected + + def test_max_age_typo_is_caught_without_an_outage(self): + self.run_cfg(self.TH + "", co2=self.co2_now(500)) # sanity: valid config works + self.assertEqual(self.set_freq.call_args[0][0], 10) + self.run_cfg("[CO2_API]\nTHRESHOLD=430\nMAX_AGE_MIN=120min\n[PIPELINE]\nscore = co2\nact = cpu_freq\n", + co2=self.co2_now(500)) + self.assertEqual(self.set_freq.call_args[0][0], 1) + + def test_no_threshold_keeps_24h_ranking(self): + self.run_cfg("[PIPELINE]\nscore = co2\nact = cpu_freq\n", co2=self.co2_now(500)) + self.assertEqual(self.set_freq.call_args[0][0], 7) # the percentile grade + self.assertNotIn("co2_throttled", self.entry()) + def test_executable_score_plugin_sets_rating(self): path = self.example("threshold_score.py") self.run_cfg(f"[PIPELINE]\nscore = co2, threshold_score\nact = cpu_freq\n"