Measure and chart semantic strings and paths per release

.github/workflows/benchmark-history.yml

@@ -16,6 +16,17 @@ name: Benchmark History
 # The backfill running as a single job still matters: separate runs land on different CI hosts,
 # and that difference is larger than most releases are. Within one job the points are comparable
 # as they stand; across jobs, BaselineBenchmarks is what ties them together.
+#
+# Three subjects share one job: quantities, strings, and paths. One job rather than a matrix so
+# that the reference workload is measured once and stamped on every entry the run produces -- which
+# is what lets a strings point and a quantities point from the same run be compared at all. It also
+# keeps the results to a single push.
+#
+# That applies from the first run of this workflow onward. The histories committed before it carry
+# baselines from two different machines, so the three charts' time rows are not yet on one scale.
+#
+# The cost is wall clock, and `subjects` on the dispatch is the answer to that: a long backfill is
+# split by subject across runs rather than by raising the timeout.
 
 on:
   release:
@@ -27,6 +38,11 @@ on:
         required: false
         default: "3.3.1 4.0.0 4.1.0 4.2.0 4.3.2 5.0.0 5.1.0 5.2.0 5.2.4 5.3.2"
         type: string
+      subjects:
+        description: "Space-separated subjects to measure: quantities strings paths"
+        required: false
+        default: "quantities strings paths"
+        type: string
 
 permissions:
   contents: write
@@ -37,29 +53,33 @@ concurrency:
 
 env:
   DOTNET_VERSION: "10.0"
-  HISTORY: docs/benchmarks/history.json
-  CHART: docs/benchmarks/performance.svg
   # Already ignored, and ktsu.Sdk regenerates .gitignore on build so a new entry would not last.
   RUNS: BenchmarkDotNet.Artifacts
-  # The set drawn in the README. One operation per storage type rather than every operation at one
-  # storage type: a quantity is a value type over T and does almost nothing of its own, so what a
-  # release changes it changes per storage type.
-  HEADLINE_FILTER: >-
-    *ConstructionBenchmarks*FromNauticalMile
-    *UnitConversionBenchmarks*InNauticalMile
-    *OperatorBenchmarks*LengthTimesLength
-    *VectorBenchmarks*.Length
-    *ComparisonBenchmarks*CompareToInterface
-    *AbstractionCostBenchmarks*
   # Short runs: three iterations is enough for a trend line, and a release should not tie up a
   # runner for half an hour.
   BENCHMARK_JOB: short
+  # One line per chart: name|history|chart|filter. An environment variable cannot hold an array,
+  # and three steps need the same three triples, so this is the one place they are written.
+  #
+  # The quantities filter names one operation per panel. The strings and paths filters name whole
+  # classes instead, so they measure more than they draw: ingest stores every row and only render
+  # selects, which lets a panel be promoted later without re-running any history. Do not prune these
+  # to match the panel count.
+  # What varies between subjects is the axis: a quantity is a value type over T and does almost
+  # nothing of its own, so its release changes land per storage type; a semantic string spends its
+  # cost at creation, so its axis is how much validation the type declares.
+  SUBJECTS: |
+    quantities|docs/benchmarks/history.json|docs/benchmarks/performance.svg|*ConstructionBenchmarks*FromNauticalMile *UnitConversionBenchmarks*InNauticalMile *OperatorBenchmarks*LengthTimesLength *VectorBenchmarks*.Length *ComparisonBenchmarks*CompareToInterface *AbstractionCostBenchmarks*
+    strings|docs/benchmarks/strings-history.json|docs/benchmarks/strings-performance.svg|*StringCreationBenchmarks* *StringOperationBenchmarks*
+    paths|docs/benchmarks/paths-history.json|docs/benchmarks/paths-performance.svg|*PathCreationBenchmarks* *PathOperationBenchmarks*
 
 jobs:
   measure:
     name: Measure and chart
     runs-on: ubuntu-latest
-    timeout-minutes: 240
+    # Three subjects rather than one. The dispatch's `subjects` input is the intended way to split
+    # a long backfill across runs; raising this number further is not.
+    timeout-minutes: 360
 
     steps:
       - name: Checkout Repository
@@ -102,19 +122,28 @@ jobs:
           work="${RUNNER_TEMP}/bench-$version"
           git worktree add --detach "$work" "$TAG"
 
-          (cd "$work" && dotnet run -c Release --project Semantics.Benchmarks -- \
-            --filter $HEADLINE_FILTER \
-            --job "$BENCHMARK_JOB" \
-            --artifacts "$GITHUB_WORKSPACE/$RUNS/$version")
-
-          dotnet run scripts/benchmark-history.cs -- ingest \
-            --history "$HISTORY" \
-            --results "$RUNS/$version" \
-            --version "$version" \
-            --commit "$(git rev-parse --short "$TAG^{commit}")" \
-            --date "$(git log -1 --format=%cs "$TAG")" \
-            --run-id "${{ github.run_id }}" \
-            --baseline-ns "${{ steps.baseline.outputs.ns }}"
+          # Every command in these loops inherits the here-string as stdin. A process that drains it
+          # would truncate the loop to one iteration and still exit zero, silently dropping subjects,
+          # so each invocation reads from /dev/null instead. BenchmarkDotNet does prompt when a
+          # filter selects nothing, which the backfill treats as an ordinary skip.
+          while IFS='|' read -r subject history chart filter; do
+            [ -n "$subject" ] || continue
+            echo "::group::$subject $version"
+            (cd "$work" && dotnet run -c Release --project Semantics.Benchmarks -- \
+              --filter $filter \
+              --job "$BENCHMARK_JOB" \
+              --artifacts "$GITHUB_WORKSPACE/$RUNS/$subject/$version" </dev/null)
+
+            dotnet run scripts/benchmark-history.cs -- ingest \
+              --history "$history" \
+              --results "$RUNS/$subject/$version" \
+              --version "$version" \
+              --commit "$(git rev-parse --short "$TAG^{commit}")" \
+              --date "$(git log -1 --format=%cs "$TAG")" \
+              --run-id "${{ github.run_id }}" \
+              --baseline-ns "${{ steps.baseline.outputs.ns }}" </dev/null
+            echo "::endgroup::"
+          done <<< "$SUBJECTS"
 
           git worktree remove --force "$work"
 
@@ -123,56 +152,70 @@ jobs:
         shell: bash
         env:
           VERSIONS: ${{ inputs.versions }}
+          SUBJECTS_WANTED: ${{ inputs.subjects }}
           BASELINE_NS: ${{ steps.baseline.outputs.ns }}
         run: |
           set -euo pipefail
           read -ra versions <<< "$VERSIONS"
+          read -ra wanted <<< "$SUBJECTS_WANTED"
 
           for version in "${versions[@]}"; do
-            echo "::group::$version"
-            # Through the environment rather than a -p: switch, because BenchmarkDotNet generates
-            # and builds a project of its own per run, which a property passed on the command line
-            # does not reach. MSBuild reads environment variables as properties in every project.
-            #
-            # A version whose API the current benchmarks cannot express is reported and skipped,
-            # rather than failing the whole backfill after the ones before it have been measured.
-            if ! BenchmarkAgainstVersion="$version" dotnet run -c Release --project Semantics.Benchmarks -- \
-                 --filter $HEADLINE_FILTER \
-                 --job "$BENCHMARK_JOB" \
-                 --artifacts "$GITHUB_WORKSPACE/$RUNS/$version"; then
-              echo "::warning::$version could not be benchmarked by the current suite; skipping"
+            while IFS='|' read -r subject history chart filter; do
+              [ -n "$subject" ] || continue
+              # Skip a subject the dispatch did not ask for.
+              case " ${wanted[*]} " in *" $subject "*) ;; *) continue ;; esac
+
+              echo "::group::$subject $version"
+              # Through the environment rather than a -p: switch, because BenchmarkDotNet generates
+              # and builds a project of its own per run, which a property passed on the command line
+              # does not reach. MSBuild reads environment variables as properties in every project.
+              if ! BenchmarkAgainstVersion="$version" dotnet run -c Release --project Semantics.Benchmarks -- \
+                   --filter $filter \
+                   --job "$BENCHMARK_JOB" \
+                   --artifacts "$GITHUB_WORKSPACE/$RUNS/$subject/$version" </dev/null; then
+                echo "::warning::$subject $version could not be benchmarked by the current suite; skipping"
+                echo "::endgroup::"
+                continue
+              fi
+
+              tag="v$version"
+              commit=""
+              date=""
+              if git rev-parse -q --verify "$tag^{commit}" >/dev/null; then
+                commit="$(git rev-parse --short "$tag^{commit}")"
+                date="$(git log -1 --format=%cs "$tag")"
+              fi
+
+              # Skipped here too: a package can build against these benchmarks and still throw from
+              # every one of them at run time, which BenchmarkDotNet reports as a table of NA rather
+              # than as a failure. Ingest refuses such a run, and the backfill carries on.
+              if ! dotnet run scripts/benchmark-history.cs -- ingest \
+                   --history "$history" \
+                   --results "$RUNS/$subject/$version" \
+                   --version "$version" \
+                   --commit "$commit" \
+                   --date "$date" \
+                   --run-id "${{ github.run_id }}" \
+                   --baseline-ns "$BASELINE_NS" </dev/null; then
+                echo "::warning::$subject $version produced no usable measurement; skipping"
+              fi
               echo "::endgroup::"
-              continue
-            fi
-
-            tag="v$version"
-            commit=""
-            date=""
-            if git rev-parse -q --verify "$tag^{commit}" >/dev/null; then
-              commit="$(git rev-parse --short "$tag^{commit}")"
-              date="$(git log -1 --format=%cs "$tag")"
-            fi
-
-            # Skipped here too, and for the same reason: a package can build against these
-            # benchmarks and still throw from every one of them at run time, which BenchmarkDotNet
-            # reports as a table of NA rather than as a failure. Ingest refuses such a run, and
-            # the backfill carries on to the next version.
-            if ! dotnet run scripts/benchmark-history.cs -- ingest \
-                 --history "$HISTORY" \
-                 --results "$RUNS/$version" \
-                 --version "$version" \
-                 --commit "$commit" \
-                 --date "$date" \
-                 --run-id "${{ github.run_id }}" \
-                 --baseline-ns "$BASELINE_NS"; then
-              echo "::warning::$version produced no usable measurement; skipping"
-            fi
-            echo "::endgroup::"
+            done <<< "$SUBJECTS"
           done
 
-      - name: Redraw the chart
+      - name: Redraw the charts
         shell: bash
-        run: dotnet run scripts/benchmark-history.cs -- render --history "$HISTORY" --out "$CHART"
+        run: |
+          set -euo pipefail
+          while IFS='|' read -r subject history chart filter; do
+            [ -n "$subject" ] || continue
+            if [ ! -f "$history" ]; then
+              echo "No history for $subject yet; nothing to draw."
+              continue
+            fi
+            dotnet run scripts/benchmark-history.cs -- render \
+              --subject "$subject" --history "$history" --out "$chart" </dev/null
+          done <<< "$SUBJECTS"
 
       - name: Commit the history and the chart
         shell: bash
@@ -183,7 +226,7 @@ jobs:
           # Staged first, then compared against the index: on the first run these files are new,
           # and `git diff` alone does not see an untracked file, so the run would push nothing and
           # still report success.
-          git add "$HISTORY" "${CHART%.svg}"*.svg
+          git add docs/benchmarks/
           if git diff --cached --quiet; then
             echo "Nothing changed."
             exit 0

CLAUDE.md

@@ -45,6 +45,7 @@ The opt-in lives in `.sonarlint/sonar-local.props` (analyzer package) and `.sona
 | `Semantics.Quantities` | Hand-written runtime types (`IPhysicalQuantity<TSelf, T>`, `PhysicalQuantityCore`, `IVector0`..`IVector4`, `UnitSystem`) plus generator output under `Generated/`. Every generated quantity is a `readonly record struct`. |
 | `Semantics.SourceGenerators` | Roslyn incremental generators that emit quantity types, units, conversions, magnitudes, physical constants, and storage-type helpers from metadata. Only the physics-specific half lives here — `Models/`, `Metadata/`, `Generators/`, and the bindings in `SemanticsGenerator`/`SemanticsDiagnostics`/`Emit`. The C# syntax templates come from `ktsu.CodeBlocker.Templates`; the metadata-driven generator base, metadata loading and the diagnostic catalogue come from `ktsu.SourceGeneratorToolkit` (#181, #192). |
 | `Semantics.Quantities.{Double,Float,Decimal,Precise}` | Props-only satellite packages. Each ships a `buildTransitive` props file (generated by `scripts/Generate-AliasProps.ps1`) that injects global-using aliases binding every quantity to one storage type, so consumers write `Mass` instead of `Mass<double>`. `Precise` binds to `ktsu.PreciseNumber.PreciseNumber` and is the one whose storage type comes from a package rather than being a C# keyword, so it carries a `PackageReference` the others do not; the core `Semantics.Quantities` still has no PreciseNumber dependency. |
+| `Semantics.Benchmarks` | BenchmarkDotNet suite covering quantities, strings and paths. Not shipped and not covered by tests, so it carries `SonarQubeExclude`. Feeds the per-release charts in `docs/benchmarks/`. |
 | `Semantics.Vocabulary` | **Shared source, not a project.** Resolves `dimensions.json` into the quantities and operators it describes and separates out what cannot be honoured. Compiled into both `Semantics.SourceGenerators` and `Semantics.Cpp` via `Compile Include`; see its README for why source rather than an assembly, and what that costs. |
 | `Semantics.Cpp` | The C++ projection of the quantity vocabulary, in its own project because `ktsu.Coder` ships no `net8.0`. Reads `dimensions.json` and emits one C++ class per dimension, per vector form and per named overload, plus the declared relationships as operators. |
 | `Semantics.Cpp.Test` | Its tests, including ones that compile the whole generated vocabulary with `g++`/`clang++`, assert what it means through `static_assert`, and check that a dimensionally wrong product — scalar or componentwise — is refused by the compiler. Also `SevenTargetProjectionTests`, which is not about C++ at all: see below. |
@@ -321,6 +322,38 @@ since this is a package with a compatibility baseline and the shape is frozen on
 `double` and `decimal`, exactly where the answer terminates and to a relative tolerance where it
 does not. Adding a storage type is one derived class.
 
+### Benchmarks and the release charts
+
+`Semantics.Benchmarks` measures three libraries. `.github/workflows/benchmark-history.yml` runs a
+fixed set once per release, appends to a history file per subject under `docs/benchmarks/`, and
+redraws the chart the README shows. Four things about it are not guessable from the code:
+
+- **`BenchmarkAgainstVersion` must be set in the environment, never with `-p:`.** BenchmarkDotNet
+  generates and builds a project of its own for each run, which a property passed on the command line
+  never reaches: the benchmark assembly would build against the version asked for and the harness
+  against the one pinned centrally, which fails to compile if a type changed shape between them.
+  MSBuild reads environment variables as properties in every project, so the environment form reaches
+  both. It swaps all four shipped packages at once, which is correct because this repository ships one
+  version across every package.
+- **The histories and the SVG files are committed output that a bot pushes.** A local `render` must be
+  diffed before commit rather than assumed. The quantities chart in particular is a regression test:
+  a change to the renderer that moves a byte in it has broken something.
+- **Nothing here touches an internal member.** The `InternalsVisibleTo` that would expose one names
+  only the test assembly, and a benchmark built on internals could only ever measure the working copy,
+  never a published package — which would make the release history impossible to backfill.
+- **All three subjects' benchmarks live in one project, so a compile error in any one file blocks
+  every subject's run regardless of `--filter`.** `AbstractionCostBenchmarks.cs` demonstrated this:
+  it held a `Length<T>` field with no initializer, valid only once `Length<T>` became a
+  `readonly record struct`, so against a pre-4.0.0 package it failed to build and took every
+  subject's backfill down with it. The fields carry `default!` now and all three histories reach
+  3.3.1. Keep the pattern when adding a field to any benchmark class the backfill compiles: an
+  error introduced for one subject costs the other two their history, and the failure reads as a
+  build error in a file the person backfilling was not touching.
+
+`BaselineBenchmarks.ReferenceWork` measures a fixed workload that touches none of this library, so
+timings from different CI runners can be compared. **Its body must never change.** Editing it silently
+rescales every comparison drawn against history recorded before the edit.
+
 ### Operators and physics relationships
 
 Cross-dimensional relationships are also declared in `dimensions.json` (`integrals`, `derivatives`, `dotProducts`, `crossProducts`). The generator emits operators like: