Pin the things you can't afford to lose (exact commands, env and config, hard rules) so context compaction can't summarize them away.
~100% of safety rules kept across compaction · 0 extra tokens per turn · offline classifier, macro-F1 0.88
You spend ten minutes teaching Claude the exact way to reach the staging box (
ssh -A -J bastion deploy@10.0.3.7), the three env vars it has to export, and the rule to never run migrations against prod. An hour later the context fills, Claude auto-compacts, and it is gone. It SSHes with the wrong flags, forgetsDATABASE_URL, and offers to "clean up" the prod database.
That is the Compaction Cliff. A summarizer shrinks a hard rule at the same rate as a chat log. On real agent configs it keeps about 53% of rules after one compaction and about 10% after five. Knowledge Triage sorts every line of your agent's memory by type and keeps the ones that must stay exact. Rules and commands are pinned word for word; notes and history get compressed or dropped.
Good for coding agents with CLAUDE.md / AGENTS.md (SSH commands, env vars, "always run the linter", "never force-push main"), and for clinical, legal, and financial assistants where a dropped rule is a real incident.
Contents: Install · Living with auto-compaction · What it does · The classifier · Numbers · FAQ · Development · Cite
/plugin marketplace add searchsim-org/knowledge-triage-skill
/plugin install knowledge-triage@knowledge-triage
That registers the PreCompact hook that keeps your rules across compaction. To choose a scope, or to try it without touching shared config, use the script instead:
git clone https://github.com/searchsim-org/knowledge-triage-skill.git \
~/.claude/skills/knowledge-triage
~/.claude/skills/knowledge-triage/install.sh # asks: trial / project / usertrial writes the hook to a gitignored .claude/settings.local.json; everything the skill writes lives under one knowledge-triage folder, and ./uninstall.sh --purge removes it.
The SKILL.md and the engine have no Claude dependency. They work with any agent that reads the portable skill format (Codex, Gemini CLI, Cursor, GitHub Copilot, OpenCode, Windsurf). Clone the repo into that agent's skills or extensions directory. Automatic extraction at compaction uses a Claude Code PreCompact hook; on other agents, point the extractor at the rules file the agent reloads and trigger it from that agent's lifecycle:
python3 scripts/triage.py extract <transcript> --rules-dir <agent-rules-dir>The engine (scripts/triage.py) is Python 3.8+ with no dependencies and no network. Run any part by hand:
python3 scripts/triage.py classify examples/CLAUDE.md # label every line by type
python3 scripts/triage.py compact examples/CLAUDE.md --budget 120 # rules pinned, filler compressedTwo things happen automatically here, and neither is the model choosing to act: a hook the Claude Code runtime fires, and the runtime re-reading files from disk.
- The skill extracts your rules. A hook (extract-rules.sh) runs on two runtime events,
PreCompact(right before a compaction) andSessionEnd(when a session ends without ever compacting). Rules are captured either way. It reads the session transcript, asks an LLM which new directives are durable (the paper's SafetyMargin test: if the agent forgot this, would it later do the wrong thing?), types each survivor with the trained classifier, keeps constraints and commands verbatim, and writes them to.claude/rules/knowledge-triage/. It sends only new turns to the LLM and consolidates duplicates, which keeps it cheap.claude -pby default;--modelsets a cheaper one. - The runtime feeds them back. Claude Code re-reads
.claude/rules/from disk at the start of every session and after each compaction. This is built into the tool, not a decision the model makes and not something you have to ask for. The rules are back in context with no per-turn injection and no per-turn token cost. A rule tagged with a path glob becomes a path-scoped rule file that loads only when you work on matching files (the paper's TypeRetrieve).
turn N rules sit in .claude/rules/ (loaded from disk by the runtime)
|
compaction the hook extracts new rules to .claude/rules/,
| then the conversation is summarized
|
turn N+1 the runtime re-reads .claude/rules/ from disk [rules are back, free]
You do not disable auto-compaction or raise its threshold. Full model, scopes, path-scoping, and cross-agent use: references/memory.md.
Every line is sorted into one of five types, ordered by how much editing it can take:
| Type | Under compression |
|---|---|
| Constraint (a rule; breaking it is a failure) | pinned word for word |
| Procedural (a step or exact command) | kept word for word |
| Belief (a fact) | compressed within meaning |
| Preference (a soft guideline) | summarized or merged |
| Episodic (a past event) | dropped |
Three operators apply this across the three ways an agent manages context:
- TypeCompact pins constraints and procedures, fills the rest by tolerance, and returns
COMPACTION_UNSAFE(exit 2) if a rule will not fit the budget. - TypeDecompose splits a base into partitions and copies each rule into every partition it governs.
- TypeRetrieve puts in-scope rules ahead of relevance. A rule cannot score its way out of the results.
Reference: taxonomy · operators · classifier · memory.
The model in the repo classifies a line by what it means. It catches rules written as plain facts:
The patient is allergic to penicillin and all beta-lactam antibiotics.
A keyword matcher reads that as a belief and lets it be summarized away. The trained model marks it a constraint. A rule written as a fact like this is about 50 to 61 percent of real safety text in clinical and legal sources; it is common there.
Three tiers, in order of cost:
- Trained model (default): a small tf-idf and linear model in scripts/model.json. Pure-Python inference, no dependency at runtime, milliseconds per line. Retrain on your own data with scripts/build_training_data.py and scripts/train_classifier.py.
- Regex baseline: pass
--model /dev/nullfor a zero-file fallback that catches imperative rules only. - LLM SafetyMargin: one small-model call per line that scores whether removing the line makes an action unsafe. Highest recall; use it to label your corpus and retrain the model above.
A summarizer has no signal for which lines are rules and edits them at the same rate as prose. From the paper's experiments on five public corpora:
| Operation | Best type-blind baseline | Knowledge Triage |
|---|---|---|
| Compaction at 10% ratio | 0.24 rule recall | 0.80 |
| Compaction over 5 rounds | 0.10 | 0.96 |
| Decomposition locality violations | 93% of configs | 0% |
| Retrieval recall@50 (in-scope rules) | 0.73 | 1.00 |
| Cost per operator call | 5 to 420 s, 36k to 57k LLM tokens | under 1 ms, 0 tokens |
The classifier that ties this to a live conversation scores macro-F1 0.88 (constraint F1 0.92) in 5-fold cross-validation on its training set. The compaction, decomposition, and retrieval figures come from the paper's corpora; the skill itself has not been A/B tested on live sessions.
Does it cost extra tokens? Almost none. The skill does its work at two moments, before a compaction and at session end: one LLM call over new turns only. Between those, rules sit in .claude/rules/, which the Claude Code runtime re-reads from disk on its own; nothing is re-injected per turn. --model at install time sets a cheaper extractor.
What if the extraction LLM is unavailable? The hook writes nothing. It does not fall back to a keyword match, since guessing is what fills the file with noise.
Does it edit my CLAUDE.md? No. It writes only under .claude/rules/knowledge-triage/. Delete that folder, or run ./uninstall.sh --purge, to undo it.
Is my conversation sent anywhere? The extraction prompt goes to whatever KT_LLM_CMD points at (claude -p by default, which is your own Claude Code). No third party.
Does it survive /compact and auto-compact? Yes, both reload .claude/rules/ from disk. Path-scoped rules reload when you next read a matching file.
Does it work without the hook? Yes. Put rules in .claude/rules/ yourself; the hook only automates the extraction.
How is this different from Claude Code auto memory? Auto memory decides what to save on its own. This types each rule and keeps constraints and commands verbatim, which auto memory does not. Use both.
knowledge-triage-skill/
SKILL.md agent-facing instructions
scripts/
triage.py engine: classify / compact / decompose / retrieve / pin / extract
train_classifier.py, build_training_data.py, model.json
hooks/ PreCompact extractor + plugin hooks.json
references/ taxonomy, operators, classifier, memory
install.sh, uninstall.sh, .claude-plugin/ installer and plugin manifests
Check the engine:
python3 scripts/triage.py --self-checkRetrain the classifier on your own labelled data (needs scikit-learn only at train time):
python3 scripts/build_training_data.py --corpus /path/to/agent/configs
python3 scripts/train_classifier.py data/train.jsonl -o scripts/model.jsonIssues and pull requests welcome.
Built on the paper The Compaction Cliff in Long-Running AI Agent Memory (CIKM 2026). If it helps your work, a citation is appreciated.
@inproceedings{zerhoudi2026compaction,
title = {The Compaction Cliff in Long-Running AI Agent Memory},
booktitle = {Proceedings of the 35th ACM International Conference on Information and Knowledge Management},
series = {CIKM '26},
year = {2026},
doi = {10.1145/3799682.3840567},
}Paper: https://doi.org/10.1145/3799682.3840567 · Dataset: https://huggingface.co/datasets/searchsim/AgentArtifactCorpus · Reference implementation: https://github.com/searchsim-org/cikm26-knowledge-triage
MIT. See LICENSE.