feat(agent): opt-in append-note concept_update_mode to avoid full-page rewrites on ingest

[sebastianbraun25]

Problem

openkb add currently regenerates the FULL body of every existing concept/entity page it touches. _gen_update/_gen_entity_update in openkb/agent/compiler.py read the entire current page from disk and send it to the LLM as existing_content, with the instruction "Rewrite the full page incorporating the new information naturally — do not just append." For a short document (e.g. a support ticket) that touches 8-10 already-existing concepts/entities, this means 8-10 full-page rewrites per ingest, even when the actual new information is just 1-2 sentences. Cost scales with page size, and page size only grows as more documents are ingested over time — so the cost of ingesting the Nth document into a mature knowledge base is far higher than the Nth document's own information content would justify.

Kontext

• openkb/agent/compiler.py: _gen_update/_gen_entity_update (full-page LLM rewrite calls), _CONCEPT_UPDATE_USER/_ENTITY_UPDATE_USER (prompt templates instructing a full rewrite), _write_concept/_write_entity (replace the full page body on disk).
• The concepts/entities identification step (_CONCEPTS_PLAN_USER) already only sends lightweight one-line briefs of existing pages, not their full content — that step is not the cost driver.
• openkb/config.py: DEFAULT_CONFIG / insert_mode (see feat(cli): configurable insert_mode (fail-fast / fail-at-end) for strict compile-failure handling #239) is the existing precedent for an opt-in per-KB config toggle that changes compiler behavior without touching the CLI surface.

Vorschlag

Add an opt-in concept_update_mode config key (per-KB, default "rewrite" = current behavior, fully unchanged) with a new "append" mode:

• On update, instead of sending the full existing page and asking for a full rewrite, generate only a short 1-2 sentence note describing how the new document relates to this concept/entity. The LLM never sees the existing page content for this call.
• Deterministically (no LLM call for the write itself) upsert that note as a dated, source-linked line under a ## Notes section in the same concept/entity page — keyed by the source document name, so re-ingesting an updated version of the same document replaces its own note line instead of duplicating it.
• sources: frontmatter tracking, backlinks, and index.md updates keep working unchanged.
• Reconciling the accumulated notes back into polished prose (see "Follow-up" below) is a separate, occasional maintenance step — not run on every openkb add.

Follow-up: fold accumulated notes into prose (openkb consolidate)

Once a page has accumulated notes, they eventually need folding back into
curated prose. Added to this issue's scope (not automatic — a manual,
opt-in maintenance command):

• New openkb consolidate [PAGE_NAME] [--all] [--min-notes N] [--dry-run]
  command (mirrors recompile's CLI shape). Never triggered automatically
  during openkb add — the user (or a cron job) decides the cadence.
• Per page, a single LLM call folds the ## Notes section into the
  existing prose — no new source document, no concept/entity
  classification, since the page is already fixed.
• Contradictions between notes (or notes vs. existing prose) are described
  directly in the rewritten text rather than silently resolved.
• The full wikilink whitelist is sent here (once per page per run, not per
  ticket) so consolidated prose can properly cross-link to other pages.
• The ## Notes section is replaced entirely; sources:/type: are
  preserved, only description: may be refreshed. A later note-append
  call re-creates a fresh ## Notes section, so the next consolidation
  run only ever sees what changed since the last one — no extra tracking
  state needed.

This keeps the existing per-concept/per-entity parallel generation pattern (same concurrency/retry machinery), just with a much cheaper prompt on the update path, and is fully backward compatible since the default mode is unchanged.

---

_{This issue was drafted with the assistance of an AI assistant.}