Keep only the metadata of any idle conversation, sent or not

CLAUDE.md

@@ -69,7 +69,7 @@ internal/view/conversation-view/ Horizon's conversation renderer, built from the
 internal/export/otlp/           sends landed files and rounds to an OpenTelemetry logs receiver, one record per file, and the metrics spool
 internal/metrics/               derives the runtime's token metric from landed Session Data into the spool; collector side; testdata holds a redacted capture of the runtime's exporter
 internal/repack/                re-cuts landed files into a new root under another budget
-internal/retention/             removes idle sent conversations' landed files, a skeleton in each one's place (collector side)
+internal/retention/             keeps only idle conversations' metadata: removes landed files, a skeleton in each one's place (collector side)
 internal/adapters/claudecode/   the claude-code-local adapter
 internal/adapters/claudecodeprovider/ the claude-code-provider adapter: the request and response bodies Claude Code writes, landed into their sessions
 internal/adapters/kiro/         the kiro-local adapter: the sessions Kiro's V3 engine writes, for kiro-cli and Kiro Crew
@@ -183,9 +183,9 @@ Every source file carries the Apache-2.0 header; `make license-fix` inserts miss
 - **asz never removes a real session whole.** Only a session a scenario build marked is removed
   whole, by the policy in its marker, and only once all of it was sent to the one receiver the
   pipeline sends to. A removal renames each directory away before it deletes it, so no reader sees
-  half a session. `storage.retention`, off by default, removes the landed files of an idle sent
-  conversation and keeps its structure: a skeleton in each file's place, and the index, cursors,
-  state and chain untouched. It never runs over a scenario root.
+  half a session. `storage.retention.metadata_only_after`, off by default, removes the landed
+  files of an idle conversation, sent or not, and keeps its structure: a skeleton in each file's
+  place, and the index, cursors, state and chain untouched. It never runs over a scenario root.
 - **A landed file is cut once.** A round addresses records by `{seq, row}` and binds to file
   digests, so a referenced file is never re-cut. A new budget applies to new files; an existing
   root is brought under it with `asz repack` into a new root, where the chains are built again.

asz.yaml

@@ -31,12 +31,17 @@ storage:
   root: ./data
 
   # How long a conversation may be idle, with nothing new landed in any of
-  # its streams, before its landed files are removed from this root. Each
-  # file is replaced by a skeleton that keeps the conversation's structure, so
-  # new activity continues the same conversation; the text is kept only by
-  # the receiver it was sent to. Only a conversation that was sent is removed.
-  # A duration such as 72h, or days such as 30d. -1 keeps everything.
-  retention: -1
+  # its streams, before this root keeps less of it. Whether it was sent, and
+  # whether an endpoint is set, does not matter. Each value is a duration
+  # such as 72h, or days such as 30d, and -1 is off. Only off keeps
+  # everything.
+  retention:
+    # To save disk, keep only the conversation's metadata: what was said and
+    # run is removed from every landed file, and who did what, when, in what
+    # order and at what cost stays. New activity continues the same
+    # conversation. The removed content cannot be restored here; only a
+    # receiver that accepted it still has it.
+    metadata_only_after: -1
 
 adapters:
   - name: claude-code-local

cmd/asz/main.go

@@ -588,7 +588,7 @@ func cmdVerify(cfg *config.Config, _ config.Adapter, _ bool) error {
 	}
 
 	if n := removed[sessiondata.ReceiverAccepted] + removed[sessiondata.ReceiverRejected] + removed[sessiondata.ReceiverUnconfirmed]; n > 0 {
-		// Removed on purpose, by storage.retention. A skeleton keeps every
+		// Removed on purpose, by retention. A skeleton keeps every
 		// record's place and the file's digest, so the checks go on through
 		// it, but the content is gone and is not verified here.
 		fmt.Printf("%d file(s) were removed by retention and are not verified: the receiver accepted %d, accepted %d with rejected records, and confirmed nothing of %d\n",
@@ -820,7 +820,7 @@ func cmdCollect(cfg *config.Config, ads []config.Adapter, once bool) error {
 	}
 	defer closeClient()
 	ref.pusher = pusher
-	keep, err := cfg.Storage.RetentionDuration()
+	keep, err := cfg.Storage.Retention.MetadataOnlyAfterDuration()
 	if err != nil {
 		return err
 	}

cmd/asz/push.go

@@ -106,6 +106,7 @@ func cmdPush(cfg *config.Config, _ config.Adapter, _ bool) error {
 			for _, e := range ms.Errors {
 				fmt.Fprintf(os.Stderr, "  error: %v\n", e)
 			}
+			sayUnread(ms.Unread)
 			derived = ms.Errors
 			return nil
 		}

cmd/asz/refresh.go

@@ -164,9 +164,9 @@ type refresher struct {
 	// only while this pipeline holds the scenario lock. A root no build wrote
 	// has no marker, so nothing there is ever removed.
 	remover *remove.Remover
-	// retention removes the landed files of conversations idle longer than
-	// storage.retention, and finishes a removal an earlier pass left in
-	// progress. It never runs over a scenario root, which has its own.
+	// retention keeps only the metadata of conversations idle longer than
+	// storage.retention.metadata_only_after, and finishes a removal an
+	// earlier pass left in progress. It never runs over a scenario root, which has its own.
 	retention *retention.Remover
 	// scenarioLock is held for the life of the process once the root is a
 	// scenario root. scenarioWait is how long a single pass waits for it.
@@ -434,11 +434,11 @@ func present(dir string, optional bool) (bool, error) {
 	return true, nil
 }
 
-// retain sets the retention: how long a conversation may be idle before its
-// landed files are removed. Zero removes nothing, though a removal an
-// earlier pass left in progress is still finished.
+// retain sets the retention: how long a conversation may be idle before only
+// its metadata is kept. Zero removes nothing, though a removal an earlier
+// pass left in progress is still finished.
 func (r *refresher) retain(keep time.Duration) {
-	r.retention = &retention.Remover{Zone: r.zone, Retention: keep, Metrics: r.deriver != nil}
+	r.retention = &retention.Remover{Zone: r.zone, MetadataOnlyAfter: keep, Metrics: r.deriver != nil}
 }
 
 // forget drops what the page cached of sessions whose files were removed,
@@ -718,6 +718,7 @@ func (r *refresher) pass() error {
 			for _, id := range ms.PendingSessions {
 				r.deferred[id] = true
 			}
+			sayUnread(ms.Unread)
 		}
 	}
 	rounds := 0
@@ -942,3 +943,19 @@ func (r *refresher) loop() {
 		_ = r.pass()
 	}
 }
+
+// sayUnread says, for each session, how many distinct execution observations
+// its skeletons keep that no pass counted. A skeleton says neither whether one
+// was an MCP call nor why it was not counted, so the line says what is there,
+// not what was lost. None is an error.
+func sayUnread(unread map[string]int) {
+	sessions := make([]string, 0, len(unread))
+	for id := range unread {
+		sessions = append(sessions, id)
+	}
+	sort.Strings(sessions)
+	for _, id := range sessions {
+		fmt.Fprintf(os.Stderr, "  metrics: %s: %d execution observations lack retained details; their MCP metrics cannot be derived\n",
+			id, unread[id])
+	}
+}

cmd/asz/retention_test.go

@@ -25,7 +25,9 @@ import (
 	"time"
 
 	"github.com/apache/skywalking-ai-sessionizer/internal/config"
+	"github.com/apache/skywalking-ai-sessionizer/internal/export/otlp"
 	"github.com/apache/skywalking-ai-sessionizer/internal/storage"
+	"github.com/apache/skywalking-ai-sessionizer/internal/view"
 	"github.com/apache/skywalking-ai-sessionizer/pkg/sessiondata"
 )
 
@@ -203,3 +205,140 @@ func TestShowAndVerifySayWhatWasRemoved(t *testing.T) {
 		t.Fatalf("verify does not report the removed files:\n%s", out)
 	}
 }
+
+// TestUnreadObservationsAreSaidBySession. What a deriver could not derive
+// is said once a session, in session order, and nothing when it is empty.
+func TestUnreadObservationsAreSaidBySession(t *testing.T) {
+	out := stderrOf(t, func() { sayUnread(map[string]int{"s-b": 2, "s-a": 1}) })
+	want := "  metrics: s-a: 1 execution observations lack retained details; their MCP metrics cannot be derived\n" +
+		"  metrics: s-b: 2 execution observations lack retained details; their MCP metrics cannot be derived\n"
+	if out != want {
+		t.Errorf("said %q, want %q", out, want)
+	}
+	if out := stderrOf(t, func() { sayUnread(nil) }); out != "" {
+		t.Errorf("said %q for nothing unread", out)
+	}
+}
+
+// unsentRemovedRoot is a root whose conversation a pipeline with no endpoint
+// collected and then kept only the metadata of. Nothing was sent, so no
+// metrics were derived either.
+func unsentRemovedRoot(t *testing.T, scenarioName string) (string, string) {
+	t.Helper()
+	root, b := scenarioRoot(t, scenarioName)
+	if err := os.RemoveAll(filepath.Join(root, storage.ScenarioDir)); err != nil {
+		t.Fatal(err)
+	}
+	zone := storage.NewZone(root)
+	ref, err := newRefresher(view.New(zone, nil), zone, bothAdapters(root, ""), pipelineMetrics(&config.Config{Metrics: allMetrics}), 2<<20, true)
+	if err != nil {
+		t.Fatal(err)
+	}
+	t.Cleanup(ref.close)
+	ref.retain(time.Nanosecond)
+	stdoutOf(t, func() {
+		if err := ref.pass(); err != nil {
+			t.Fatal(err)
+		}
+	})
+	if sk, landed := skeletonsOf(t, root, b.Session); sk == 0 || landed != 0 {
+		t.Fatalf("a pipeline with no endpoint kept %d landed files and wrote %d skeletons", landed, sk)
+	}
+	return root, b.Session
+}
+
+// A pipeline with no endpoint keeps only the metadata of an idle
+// conversation it never sent. asz show says no receiver confirmed the file,
+// and asz verify counts it among the removed.
+func TestAPipelineWithNoEndpointKeepsOnlyMetadata(t *testing.T) {
+	root, session := unsentRemovedRoot(t, "growth.yaml")
+	saved := positional
+	t.Cleanup(func() { positional = saved })
+	cfg := config.Default()
+	cfg.Storage.Root = root
+	files, err := storage.LandedFiles(storage.NewZone(root), session)
+	if err != nil {
+		t.Fatal(err)
+	}
+	f, err := sessiondata.Open(files[0].Path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	_, recs, err := sessiondata.All(f)
+	f.Close()
+	if err != nil {
+		t.Fatal(err)
+	}
+	id := ""
+	for _, r := range recs {
+		if r.ID != "" {
+			id = r.ID
+			break
+		}
+	}
+	positional = []string{session, id}
+	out := stdoutOf(t, func() {
+		if err := cmdShow(cfg, config.Adapter{}, false); err != nil {
+			t.Fatal(err)
+		}
+	})
+	if !strings.Contains(out, "no receiver confirmed the file") {
+		t.Errorf("show output does not say no receiver confirmed the file:\n%s", out)
+	}
+	positional = nil
+	out = stdoutOf(t, func() {
+		if err := cmdVerify(cfg, config.Adapter{}, false); err != nil {
+			t.Fatalf("verify over a root with nothing sent: %v", err)
+		}
+	})
+	if !strings.Contains(out, "accepted 0, accepted 0 with rejected records, and confirmed nothing of") {
+		t.Fatalf("verify does not report the removed files as confirmed by no receiver:\n%s", out)
+	}
+}
+
+// Metrics turned on after a conversation kept only its metadata: the first
+// pass of a pipeline says how many of its execution observations their
+// metrics cannot be derived for, and the pass after it does not. asz push,
+// a process of its own, says it again.
+func TestLateMetricsAreSaidByThePipelineAndByPush(t *testing.T) {
+	root, session := unsentRemovedRoot(t, "mcp-calls.yaml")
+	line := "  metrics: " + session + ": "
+	ref := pipelineOver(t, root, bothAdapters(root, ""))
+	var out string
+	stdoutOf(t, func() {
+		out = stderrOf(t, func() {
+			if err := ref.pass(); err != nil {
+				t.Fatal(err)
+			}
+		})
+	})
+	if !strings.Contains(out, line) || !strings.Contains(out, " execution observations lack retained details; their MCP metrics cannot be derived") {
+		t.Fatalf("the first pass with metrics on said:\n%s", out)
+	}
+	stdoutOf(t, func() {
+		out = stderrOf(t, func() {
+			if err := ref.pass(); err != nil {
+				t.Fatal(err)
+			}
+		})
+	})
+	if strings.Contains(out, line) {
+		t.Fatalf("the next pass said it again:\n%s", out)
+	}
+
+	rcv := receiver(t)
+	cfg := &config.Config{Metrics: allMetrics}
+	cfg.Storage.Root = root
+	o := rcv.Options(otlp.ProtocolGRPC)
+	cfg.Export.OTLP.Protocol, cfg.Export.OTLP.Endpoint, cfg.Export.OTLP.ServiceName = o.Protocol, o.Endpoint, "Push Test"
+	stdoutOf(t, func() {
+		out = stderrOf(t, func() {
+			if err := cmdPush(cfg, config.Adapter{}, false); err != nil {
+				t.Fatal(err)
+			}
+		})
+	})
+	if !strings.Contains(out, line) {
+		t.Fatalf("asz push did not say it:\n%s", out)
+	}
+}

cmd/asz/view.go

@@ -166,7 +166,7 @@ func cmdServer(cfg *config.Config, ads []config.Adapter, once bool) error {
 	}
 	defer closeClient()
 	ref.pusher = pusher
-	keep, err := cfg.Storage.RetentionDuration()
+	keep, err := cfg.Storage.Retention.MetadataOnlyAfterDuration()
 	if err != nil {
 		return err
 	}
@@ -262,11 +262,11 @@ func printPipeline(ref *refresher, cfg *config.Config) {
 		}
 		fmt.Printf("export   : %s, at the end of every pass\n", endpoint)
 	}
-	if ref.retention != nil && ref.retention.Retention > 0 {
-		fmt.Printf("retention: a sent conversation idle for %s loses its landed files; a skeleton keeps its structure, and the receiver its text\n", ref.retention.Retention)
+	if ref.retention != nil && ref.retention.MetadataOnlyAfter > 0 {
+		fmt.Printf("retention: a conversation idle for %s keeps only its metadata; its content is removed, and only a receiver that accepted it still has it\n", ref.retention.MetadataOnlyAfter)
 	}
 	// A root a claude-code scenario build wrote. Only there does a pipeline
-	// remove anything, and only what a build marked.
+	// remove a session whole, and only one a build marked.
 	if fi, err := os.Stat(filepath.Join(ref.zone.Root(), storage.ScenarioDir)); err == nil && fi.IsDir() {
 		fmt.Printf("scenario : %s\n", ref.removalSays())
 	}

docs/en/adapters/claude-code-glossary.md

@@ -92,7 +92,7 @@ than a plausible word that would send a reader looking for a field that does not
 | `in_segment` | the activity window a Talk falls in |
 | `input_of` | unavailable; the serialized provider request is not written locally |
 | `omitted` | deliberately not carried |
-| `removed` | no equivalent: asz removes the payloads of a conversation idle longer than storage.retention from its own root; Claude Code keeps its transcript as before |
+| `removed` | no equivalent: asz removes the payloads of a conversation idle longer than storage.retention.metadata_only_after from its own root; Claude Code keeps its transcript as before |
 | `retries` | unavailable; retry identity is not recorded |
 | `segment` | an activity window |
 | `size_only` | only the length was kept |

docs/en/adapters/kiro-glossary.md

@@ -90,7 +90,7 @@ not there.
 | `notification` | — | derived here | no equivalent |
 | `omitted` | — | derived here | no equivalent |
 | `parent` | — | derived here | no equivalent: a record names no record before it |
-| `removed` | — | derived here | no equivalent: asz removes the payloads of a conversation idle longer than storage.retention from its own root |
+| `removed` | — | derived here | no equivalent: asz removes the payloads of a conversation idle longer than storage.retention.metadata_only_after from its own root |
 | `reports` | — | derived here | no equivalent |
 | `retries` | — | derived here | no equivalent |
 | `runtime.notification` | — | derived here | no equivalent: a subagent's end is the result of its call |

docs/en/adapters/langsmith-glossary.md

@@ -88,7 +88,7 @@ not there.
 | `notification` | — | derived here | no equivalent |
 | `omitted` | — | derived here | a graph's own run: its repeated message list is not landed |
 | `redacted` | — | derived here | no equivalent: nothing on the wire says it redacted anything |
-| `removed` | — | derived here | no equivalent: asz removes the payloads of a conversation idle longer than storage.retention from its own root |
+| `removed` | — | derived here | no equivalent: asz removes the payloads of a conversation idle longer than storage.retention.metadata_only_after from its own root |
 | `reports` | — | derived here | derived |
 | `retries` | — | derived here | no equivalent yet: the client does not say a run retried another |
 | `runtime.notification` | — | derived here | no equivalent |

docs/en/changes/changes.md

@@ -283,13 +283,21 @@
 
 ## Retention
 
-- A storage root no longer only grows. `storage.retention`, `-1` and off by default, removes the
-  landed files of a conversation that was sent and has had nothing new land for that long, such as
-  `72h` or `30d`. On one root, 7,235 landed files of 847.5 MB had skeletons of 29.6 MB, 3.5% of
-  them, measured on 2026-10-13. `asz collect` and `asz server` remove at
-  the end of a pass, after the send, the whole conversation's files at once, and only once its head
-  round covers them and, with metrics derived, every file of it is derived. A conversation never
-  sent is kept. See [Retention](../formats/storage-root.md#retention).
+- A storage root no longer only grows. `storage.retention.metadata_only_after`, `-1` and off by
+  default, keeps only the metadata of a conversation that has had nothing new land for that long,
+  such as `72h` or `30d`, to save disk: it removes the landed files, and a skeleton of each keeps
+  who did what, when, in what order and at what cost. It applies whether the conversation was sent
+  or not, and whether an endpoint is set or not; the content of one never sent is then gone
+  everywhere. On one root, 7,235 landed files of 847.5 MB had skeletons of 29.6 MB, 3.5% of them,
+  measured on 2026-10-13. `asz collect` and `asz server` remove at the end of a pass, after the
+  send, the whole conversation's files at once, and only once its head round covers them and, with
+  metrics derived, every file of it is derived. See [Retention](../formats/storage-root.md#retention).
+- Metrics turned on after a conversation kept only its metadata count its tokens from the
+  skeletons, which keep no execution record's protocol or server, so no MCP call is derived from
+  them. The first pass of a process that saves its state says, for each session, how many distinct
+  execution observations its skeletons keep that no pass counted, as `metrics: <session>: <n>
+  execution observations lack retained details`. The count does not say how many were MCP calls, or
+  why one was not counted. Only execution files are read for it.
 - A provider body landed after a removal is cut exactly as with nothing removed. The cutter holds
   each removed body by its skeleton, and the session's store, under `provider/`, keeps the bytes of
   every chain tip at a removal and of each removed piece a later body cites, so the same pieces are