feat(cortex): org:<uid> root — tenant-relative hosted paths and a retired user: root for the transition

README.md

@@ -177,6 +177,24 @@ route has no keyword/vector switch, so both wires declare hybrid fetch only. See
 [`docs/architecture/cortex.md`](docs/architecture/cortex.md) and
 [`crates/tinymemory-integrations/src/cortex/README.md`](crates/tinymemory-integrations/src/cortex/README.md).
 
+### Where a person's memory lives
+
+One root per person, `org:<id>`, and no `user:` segment below it (`user:<id>`
+is the person's actor, never a scope):
+
+```text
+org:<id>/app:learnings                         learnings
+org:<id>/app:brain/source:gmail                a connector's documents
+org:<id>/ws:main/app:conversations             chats
+org:<id>/ws:main/app:flows/service:<flow>/…    a workflow's memory
+```
+
+A direct `cortexdb` engine gets `scope_root = "org:<id>"`; a `tinyhumans`
+engine gets `tenant_root = true` and sends paths relative to the tenant root
+the backend pins. Memory an earlier layout wrote below `user:<id>` stays
+readable and forgettable through `retired_scope_root` until it has moved. See
+[`docs/architecture/cortex-layout.md`](docs/architecture/cortex-layout.md).
+
 ### Adding an engine
 
 An engine is a module of `tinymemory-integrations` behind a feature named after

crates/tinymemory-integrations/src/config/mod.rs

@@ -86,7 +86,8 @@ pub struct EngineSettings {
     #[serde(default, skip_serializing_if = "Option::is_none")]
     pub consolidation: Option<Consolidation>,
     /// The scope every item is laid out below (layout v3), such as one
-    /// person's `user:<id>`; unset keeps the legacy `app:tinymemory` tree.
+    /// person's `org:<id>`; unset keeps the legacy `app:tinymemory` tree
+    /// (unless [`EngineSettings::tenant_root`]).
     /// Switching it moves nothing: memory under the other layout is no
     /// longer read until a host moves it.
     #[serde(default, skip_serializing_if = "Option::is_none")]
@@ -96,6 +97,17 @@ pub struct EngineSettings {
     /// write. Ignored without a root.
     #[serde(default, skip_serializing_if = "Option::is_none")]
     pub scope_owner: Option<String>,
+    /// Layout v3 relative to the hosted tenant's own root, which the
+    /// TinyHumans backend pins (`org:<id>`): no root segment is sent.
+    /// `tinyhumans` engine only; overrides [`EngineSettings::scope_root`].
+    #[serde(default, skip_serializing_if = "std::ops::Not::not")]
+    pub tenant_root: bool,
+    /// A root an earlier v3 layout wrote below (`user:<id>`), still read
+    /// and forgotten, never written, until its memory has moved. Needs a
+    /// v3 layout ([`EngineSettings::scope_root`] or
+    /// [`EngineSettings::tenant_root`]).
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub retired_scope_root: Option<String>,
     /// Attribute events to who actually said or did them (CortexDB's
     /// `observed_actor`, with the memory's owner as `subject`): an assistant
     /// turn to its agent, a user turn or an item naming a

crates/tinymemory-integrations/src/cortex/README.md

@@ -89,10 +89,15 @@ app:tinymemory/team:acme/agent:writer/app:learnings                     a team m
 
 That is the legacy layout, the default. With a scope root
 (`EngineSettings::scope_root`, `CortexEngine::with_scope_root`), such as one
-person's `user:<id>`, every item is laid out below that root instead, each
-kind under a leaf of its own (`user:42/ws:main/app:conversations`,
-`user:42/app:brain/source:gmail`), and a direct engine registers the root
-with its owner before the first write. See
+person's `org:<id>`, every item is laid out below that root instead, each
+kind under a leaf of its own (`org:42/ws:main/app:conversations`,
+`org:42/app:brain/source:gmail`), and a direct engine registers the root
+with its owner (the actor `user:<id>`) before the first write. On the
+hosted wire the root is the tenant's own (`EngineSettings::tenant_root`):
+the engine sends `ws:main/app:conversations` and the backend stores it at
+`org:<id>/ws:main/app:conversations`. A retired root
+(`EngineSettings::retired_scope_root`, the earlier `user:<id>`) is still
+read and forgotten, never written, until its memory has moved. See
 [cortex-layout.md](../../../../docs/architecture/cortex-layout.md).
 
 The hosted backend also re-roots every scope under the caller's tenant.

crates/tinymemory-integrations/src/cortex/engine/fetch.rs

@@ -266,14 +266,19 @@ impl CortexEngine {
 }
 
 /// Merges per-scope rankings rank by rank: every scope's best, then every
-/// scope's second, and so on.
+/// scope's second, and so on, each item once.
 pub(super) fn interleave(mut lists: Vec<Vec<Envelope>>) -> Vec<Envelope> {
     let longest = lists.iter().map(Vec::len).max().unwrap_or(0);
     let mut iters: Vec<_> = lists.iter_mut().map(|list| list.drain(..)).collect();
     let mut out = Vec::new();
+    // An item held below both the root and a retired root is one hit, at its
+    // best rank.
+    let mut seen = HashSet::new();
     for _ in 0..longest {
         for iter in &mut iters {
-            if let Some(envelope) = iter.next() {
+            if let Some(envelope) = iter.next()
+                && seen.insert(envelope.id.clone())
+            {
                 out.push(envelope);
             }
         }

crates/tinymemory-integrations/src/cortex/engine/fetch_tests.rs

@@ -103,3 +103,16 @@ fn a_pack_budget_fits_each_event_whole_up_to_a_ceiling() {
     assert_eq!(whole_items_budget(MAX_PACK_EVENTS), MAX_PACK_TOKENS);
     assert_eq!(whole_items_budget(usize::MAX), MAX_PACK_TOKENS);
 }
+
+#[test]
+fn an_item_ranked_in_two_scopes_is_one_hit_at_its_best_rank() {
+    // The same item below the root and below a retired root.
+    let envelope =
+        |text: &str, id: &str| Envelope::for_item(&doc(text, None), id).unwrap().remove(0);
+    let merged = interleave(vec![
+        vec![envelope("a", "same"), envelope("b", "b")],
+        vec![envelope("c", "c"), envelope("a again", "same")],
+    ]);
+    let ids: Vec<_> = merged.iter().map(|e| e.id.as_str()).collect();
+    assert_eq!(ids, vec!["same", "c", "b"]);
+}

crates/tinymemory-integrations/src/cortex/engine/items.rs

@@ -13,7 +13,7 @@ use tinymemory_api::{
 use super::CortexEngine;
 use super::scopes::KindScope;
 use crate::cortex::envelope::{Decoded, Envelope, decode_event, labels, rebuild, rebuild_whole};
-use crate::cortex::error::Result;
+use crate::cortex::error::{Error, Result};
 
 /// The kinds `filter` admits, in the fixed order
 /// [`ItemKind::ALL`] lists them.
@@ -168,7 +168,9 @@ impl CortexEngine {
             for (id, events) in self.item_events(&scope, &ids).await? {
                 let envelopes: Vec<Envelope> = events.into_iter().map(|d| d.envelope).collect();
                 if let Some(item) = rebuild_whole(&envelopes) {
-                    found.insert(ItemId::new(id.clone()), hit(&id, &item, 0.0));
+                    found
+                        .entry(ItemId::new(id.clone()))
+                        .or_insert_with(|| hit(&id, &item, 0.0));
                 }
             }
         }
@@ -199,15 +201,31 @@ impl CortexEngine {
         }
         let lookups: Vec<(KindScope, Vec<String>)> = by_node
             .into_iter()
-            .map(|(namespace, ids)| (KindScope::new(&self.layout, namespace.clone(), kind), ids))
+            .flat_map(|(namespace, ids)| {
+                KindScope::read(&self.layout, namespace, kind)
+                    .into_iter()
+                    .map(move |scope| (scope, ids.clone()))
+            })
             .collect();
-        let found: Vec<HashMap<String, Vec<Decoded>>> = stream::iter(lookups)
-            .map(|(scope, ids)| async move { self.item_events(&scope, &ids).await })
+        let mut found: Vec<(usize, HashMap<String, Vec<Decoded>>)> = stream::iter(lookups)
+            .enumerate()
+            .map(|(order, (scope, ids))| async move {
+                Ok::<_, Error>((order, self.item_events(&scope, &ids).await?))
+            })
             .buffer_unordered(LOOKUPS_AT_ONCE)
             .try_collect()
             .await?;
+        // An item held below both the root and a retired root is rebuilt from
+        // the events of one of them, never from both at once: the root's
+        // (lookups are in read order, root first), whichever lookup answered
+        // first; the retired root's only when the root's copy cannot be
+        // rebuilt.
+        found.sort_by_key(|(order, _)| *order);
         let mut out = HashMap::new();
-        for (id, events) in found.into_iter().flatten() {
+        for (id, events) in found.into_iter().flat_map(|(_, found)| found) {
+            if out.contains_key(&id) {
+                continue;
+            }
             let envelopes: Vec<Envelope> = events.into_iter().map(|d| d.envelope).collect();
             if let Some(item) = rebuild_whole(&envelopes) {
                 out.insert(id, item);

crates/tinymemory-integrations/src/cortex/engine/list.rs

@@ -173,16 +173,22 @@ impl CortexEngine {
                 if len > 0 {
                     filled_pages += 1;
                 }
+                let shadowed = self.shadowed(scope, &page.items).await?;
                 for (position, event) in page.items.iter().enumerate().skip(at.offset) {
                     at.offset = position + 1;
                     let id = event.get("id").and_then(Value::as_str);
                     if id.is_some() && id == at.last.as_deref() {
                         continue;
                     }
                     at.last = id.map(str::to_owned);
-                    if let Some(found) =
-                        self.admit(kind, &req, event, &mut seen, walk == Walk::Preview)
-                    {
+                    if let Some(found) = self.admit(
+                        kind,
+                        &req,
+                        event,
+                        &mut seen,
+                        &shadowed,
+                        walk == Walk::Preview,
+                    ) {
                         pending.push(found);
                         if pending.len() == req.limit {
                             let exhausted = at.offset == len
@@ -213,6 +219,25 @@ impl CortexEngine {
         Ok((self.resolve(pending).await?, next))
     }
 
+    /// The ids of the items on `events` (a page of `scope`) that the root
+    /// also holds, when `scope` is below the retired root: the root's copy is
+    /// the one listed, so an item held below both is one item on every page,
+    /// not one per root. Empty for any other scope.
+    async fn shadowed(&self, scope: &KindScope, events: &[Value]) -> Result<HashSet<String>> {
+        if !self.layout.is_retired(&scope.path) {
+            return Ok(HashSet::new());
+        }
+        let ids: Vec<String> = events
+            .iter()
+            .filter_map(decode_event)
+            .map(|decoded| decoded.envelope.id)
+            .collect::<HashSet<_>>()
+            .into_iter()
+            .collect();
+        let active = KindScope::new(&self.layout, scope.namespace.clone(), scope.kind);
+        Ok(self.item_events(&active, &ids).await?.into_keys().collect())
+    }
+
     /// Whether one raw event starts an item this listing returns; with
     /// `preview`, the item is taken from that event alone.
     fn admit(
@@ -221,10 +246,11 @@ impl CortexEngine {
         req: &ListRequest,
         event: &Value,
         seen: &mut HashSet<String>,
+        shadowed: &HashSet<String>,
         preview: bool,
     ) -> Option<Pending> {
         let envelope = decode_event(event)?.envelope;
-        if !keeps(&req.filter, kind, &envelope) {
+        if !keeps(&req.filter, kind, &envelope) || shadowed.contains(&envelope.id) {
             return None;
         }
         let starts = envelope.part().is_none_or(|index| index == 0);

crates/tinymemory-integrations/src/cortex/engine/mod.rs

@@ -117,7 +117,7 @@ impl CortexEngine {
 
     /// The same engine, laying its scopes out below `root` (layout v3)
     /// instead of the legacy `app:tinymemory` tree: one person's memory as
-    /// one subtree (`user:<id>`), each kind under a leaf of its own (see the
+    /// one subtree (`org:<id>`), each kind under a leaf of its own (see the
     /// `envelope` module docs). With an `owner` actor (`user:<id>`), a
     /// direct engine registers the root as owned by it before its first
     /// write, so the person owns their root rather than whichever key wrote
@@ -141,6 +141,43 @@ impl CortexEngine {
         Ok(self)
     }
 
+    /// The same engine in layout v3 relative to the hosted tenant's root:
+    /// no root segment is sent, and the TinyHumans backend pins its own
+    /// (`org:<id>`), so a person's chats are stored at
+    /// `org:<id>/ws:main/app:conversations`. See the `envelope` module docs.
+    ///
+    /// # Errors
+    ///
+    /// [`Error::Config`] on the direct wire, where nothing pins a root and
+    /// every install would share one tree.
+    pub fn with_tenant_root(mut self) -> Result<Self> {
+        if self.wire() != CortexWire::TinyHumans {
+            return Err(Error::Config(
+                "only the TinyHumans wire has a tenant root; name a scope root".to_string(),
+            ));
+        }
+        self.layout = ScopeLayout::tenant();
+        self.owner = None;
+        self.registered = Arc::new(AtomicBool::new(false));
+        Ok(self)
+    }
+
+    /// The same v3 engine, also reading and forgetting below `retired`
+    /// (`user:<id>`, where an earlier layout wrote), while writing only
+    /// below its root: the transition until that memory has moved. Every
+    /// read merges both by item id; a forget or erasure removes from both.
+    ///
+    /// # Errors
+    ///
+    /// [`Error::Config`] without a scope root (call
+    /// [`CortexEngine::with_scope_root`] or [`CortexEngine::with_tenant_root`]
+    /// first), or for a retired root that is not a valid v3 root or is the
+    /// root itself.
+    pub fn with_retired_root(mut self, retired: &str) -> Result<Self> {
+        self.layout = self.layout.with_retired_root(retired)?;
+        Ok(self)
+    }
+
     /// Registers the v3 root as owned by its owner, once, before a write.
     /// Already registered (`409`), the root's record is read and the owner
     /// added to its members when it is not an owner yet, so a `409` never
@@ -154,6 +191,9 @@ impl CortexEngine {
         let (ScopeLayout::V3 { root, .. }, Some(owner)) = (&self.layout, &self.owner) else {
             return;
         };
+        if root.is_empty() {
+            return;
+        }
         if self.wire() != CortexWire::Direct || self.registered.load(Ordering::Acquire) {
             return;
         }
@@ -479,3 +519,7 @@ mod layout_tests;
 #[cfg(test)]
 #[path = "engine_test_support.rs"]
 mod test_support;
+
+#[cfg(test)]
+#[path = "mod_retired_root_tests.rs"]
+mod retired_root_tests;