diff --git a/crates/tinymemory-integrations/src/cortex/README.md b/crates/tinymemory-integrations/src/cortex/README.md index aa55c140..b78d6a89 100644 --- a/crates/tinymemory-integrations/src/cortex/README.md +++ b/crates/tinymemory-integrations/src/cortex/README.md @@ -243,12 +243,15 @@ as prefixes, so they cannot be labelled and are filtered only client-side. - **Forget.** `Ids` looks the items' labels up in every scope the engine holds. `Filter` (which must be non-empty) walks the scopes it reads and matches the full filter. Either way the matched events are then removed with - `selector.memory_ids`, in batches of 100. An empty selector is never sent, - and neither is `confirm_all`. `forgotten` counts items. -- **Erase.** Hosted erases only the whole tree (`whole_tree`), in one - `DELETE memory` that erases the caller's entire hosted memory, and refuses - anything narrower with `Unsupported`, because the backend proxies no - per-scope erasure. Direct lists the registered kind scopes in reach + `selector.memory_ids` and `cascade: "redact_events"`, in batches of 100. + The cascade is always named: CortexDB's default, `derived_only`, keeps the + events. An empty selector is never sent, and neither is `confirm_all`. `forgotten` counts items. +- **Erase.** Hosted erases the whole tree (`whole_tree`) in one + `DELETE memory` that erases the caller's entire hosted memory. Anything + narrower goes scope by scope, as on Direct, through the backend's + `memory/v1/erasures` passthrough (`{scope, audit_note}`, memory-api's + synchronous scoped erasure, unwrapped; a retriable `502 + ERASURE_INCOMPLETE` is retried; a missing route is `Unsupported`). Direct lists the registered kind scopes in reach with the complete scope listing, then sends `v1/erasures` with `confirm_all` (never a selector) once per scope, deepest first, and returns the erasure ids as receipts. CortexDB deletes an erased scope's diff --git a/crates/tinymemory-integrations/src/cortex/descriptor/mod.rs b/crates/tinymemory-integrations/src/cortex/descriptor/mod.rs index 179fbd43..b82b724f 100644 --- a/crates/tinymemory-integrations/src/cortex/descriptor/mod.rs +++ b/crates/tinymemory-integrations/src/cortex/descriptor/mod.rs @@ -122,7 +122,8 @@ pub enum CortexWire { /// and a bulk append route. Direct, /// CortexDB behind the TinyHumans backend's `/memory/*` routes: - /// `{success,data}` envelopes, typed `errorCode` failures, a strict answer + /// `{success,data}` envelopes (except the `/memory/v1/*` passthrough, + /// which answers in CortexDB's dialect), typed `errorCode` failures, a strict answer /// schema, `Idempotency-Key` claims on writes, and no bulk, wait or health /// route. TinyHumans, @@ -168,10 +169,10 @@ impl CortexWire { (Self::TinyHumans, Route::BuildBeliefs) => "memory/beliefs/build", // Never sent: hosted beliefs are read through recall only. (Self::TinyHumans, Route::Beliefs) => "memory/beliefs", - // Never sent: the backend proxies no per-scope erasure route, so - // an `erase` narrower than the whole tree refuses on this wire - // without a request. - (Self::TinyHumans, Route::Erasures) => "memory/erasures", + // The backend's CortexDB-dialect passthrough (no envelope); + // memory-api pins the scope under the tenant's root. An `erase` + // narrower than the whole tree goes here, scope by scope. + (Self::TinyHumans, Route::Erasures) => "memory/v1/erasures", // `DELETE memory`: erases the caller's entire hosted memory. (Self::TinyHumans, Route::EraseAll) => "memory", // Never sent: the hosted backend keeps its own tenancy. diff --git a/crates/tinymemory-integrations/src/cortex/engine/engine_test_support.rs b/crates/tinymemory-integrations/src/cortex/engine/engine_test_support.rs index c9f57c6e..fb860f78 100644 --- a/crates/tinymemory-integrations/src/cortex/engine/engine_test_support.rs +++ b/crates/tinymemory-integrations/src/cortex/engine/engine_test_support.rs @@ -11,6 +11,7 @@ impl CortexEngine { visibility, settle: visibility, poll: Duration::from_millis(5), + erasure: Duration::from_millis(300), }; self.log.client.set_read_backoff(Duration::from_millis(5)); self diff --git a/crates/tinymemory-integrations/src/cortex/engine/erase.rs b/crates/tinymemory-integrations/src/cortex/engine/erase.rs index fb2e2dbc..9b3ca797 100644 --- a/crates/tinymemory-integrations/src/cortex/engine/erase.rs +++ b/crates/tinymemory-integrations/src/cortex/engine/erase.rs @@ -9,29 +9,26 @@ //! first, so a layout that ever put a scope below another would not strand //! redacted events behind held keys. //! -//! On the TinyHumans wire only the whole tree erases (`whole_tree`: the -//! root, its descendants, every kind), in one `DELETE memory` that erases the +//! On the TinyHumans wire the whole tree (`whole_tree`: the root, its +//! descendants, every kind) erases in one `DELETE memory` that erases the //! caller's entire hosted memory, every scope under its tenant (including any -//! another layout or client wrote there). The backend proxies no per-scope -//! erasure, so a narrower request refuses with `Unsupported` and sends -//! nothing. +//! another layout or client wrote there). A narrower request erases scope by +//! scope, as Direct does, through the backend's `memory/v1/erasures` +//! passthrough, which memory-api pins under the tenant's root (see +//! `log::erase`). A backend without that route answers `Unsupported`, so a +//! caller can fall back to `forget`. use tinymemory_api::{EraseReport, EraseRequest, ItemKind}; use super::CortexEngine; use crate::cortex::descriptor::CortexWire; -use crate::cortex::error::{Error, Result}; +use crate::cortex::error::Result; impl CortexEngine { /// See the module docs. pub(super) async fn erase_scopes(&self, req: EraseRequest) -> Result { req.validate()?; - if self.log.client.wire() == CortexWire::TinyHumans { - if !is_whole_tree(&req) { - return Err(Error::Unsupported( - "the TinyHumans backend erases only the whole memory (whole_tree)".to_string(), - )); - } + if self.log.client.wire() == CortexWire::TinyHumans && is_whole_tree(&req) { let erased_scopes = self.log.erase_all().await?; log::debug!("[cortex] erased the whole hosted memory ({erased_scopes} scopes)"); return Ok(EraseReport { @@ -48,10 +45,10 @@ impl CortexEngine { scopes.sort_by_key(|scope| std::cmp::Reverse(scope.path.matches('/').count())); let mut report = EraseReport::default(); for scope in scopes { - let receipt = self.log.erase(&scope.path).await?; - log::debug!("[cortex] erased {} ({receipt})", scope.path); - report.erased_scopes += 1; - report.receipts.push(receipt); + let erased = self.log.erase(&scope.path).await?; + log::debug!("[cortex] erased {} ({erased:?})", scope.path); + report.erased_scopes = report.erased_scopes.saturating_add(erased.scopes); + report.receipts.extend(erased.ids); } Ok(report) } diff --git a/crates/tinymemory-integrations/src/cortex/engine/mod_direct_tests.rs b/crates/tinymemory-integrations/src/cortex/engine/mod_direct_tests.rs index a7d1f630..6111369c 100644 --- a/crates/tinymemory-integrations/src/cortex/engine/mod_direct_tests.rs +++ b/crates/tinymemory-integrations/src/cortex/engine/mod_direct_tests.rs @@ -124,6 +124,26 @@ async fn forget_batches_event_ids_at_one_hundred() { assert_eq!(state.event_count(), 0); } +#[tokio::test] +async fn every_forget_names_the_cascade_that_removes_the_events() { + // CortexDB's default cascade (`derived_only`) keeps the events, so a + // forget that left it to the default would delete nothing written. + let (endpoint, state) = direct_double().await; + let engine = direct_engine(&endpoint); + let item = sample_items().remove(0); + engine.store(item.clone()).await.unwrap(); + engine + .forget(ForgetTarget::Ids(vec![item.fingerprint().into()])) + .await + .unwrap(); + let forgets = state.seen.lock().unwrap().forgets.clone(); + assert!(!forgets.is_empty()); + for body in &forgets { + assert_eq!(body["cascade"], "redact_events", "{body}"); + } + assert_eq!(state.event_count(), 0); +} + #[tokio::test] async fn a_partially_applied_conversation_completes_on_retry() { let (endpoint, state) = direct_double().await; @@ -279,3 +299,18 @@ async fn reads_wait_out_a_scope_authorization_change() { "six attempts, then it gives up" ); } + +#[test] +fn the_double_refuses_a_cascade_that_is_not_a_known_string() { + let mut log = crate::cortex::testing::CortexLog::default(); + for cascade in [ + serde_json::json!(null), + serde_json::json!(123), + serde_json::json!("nope"), + ] { + let (code, answer) = log.forget(&serde_json::json!({ + "scope": "app:x", "confirm_all": true, "cascade": cascade + })); + assert_eq!(code, 400, "{cascade}: {answer}"); + } +} diff --git a/crates/tinymemory-integrations/src/cortex/engine/mod_erase_tests.rs b/crates/tinymemory-integrations/src/cortex/engine/mod_erase_tests.rs index 663dfd5d..de49afe6 100644 --- a/crates/tinymemory-integrations/src/cortex/engine/mod_erase_tests.rs +++ b/crates/tinymemory-integrations/src/cortex/engine/mod_erase_tests.rs @@ -1,13 +1,16 @@ -//! Erase on the Direct wire: deepest scope first, so every event is deleted -//! and every write key released; kinds narrow it. The hosted wire erases -//! only the whole tree, in one `DELETE /memory`, and refuses anything -//! narrower without a request. +//! Erase: deepest scope first, so every event is deleted and every write +//! key released; kinds narrow it. Hosted, the whole tree is one `DELETE +//! /memory` and anything narrower goes scope by scope through the backend's +//! `memory/v1/erasures` passthrough (memory-api's synchronous scoped +//! erasure, retried while incomplete). use tinymemory_api::{ EraseRequest, ItemKind, LearningKind, ListRequest, MemoryMeta, MetaFilter, Namespace, Reach, StoreItem, }; +use std::sync::atomic::Ordering; + use super::*; use crate::cortex::testing::{direct_double, direct_engine, hosted_double, hosted_engine}; @@ -124,17 +127,122 @@ async fn an_erasure_with_nothing_registered_sends_nothing() { } #[tokio::test] -async fn the_hosted_wire_refuses_to_erase_without_a_request() { +async fn the_hosted_wire_erases_a_subtree_through_the_backend_passthrough() { let (endpoint, state) = hosted_double().await; let engine = hosted_engine(&endpoint); - let refused = engine - .erase(EraseRequest::new(Reach::subtree(node("agent:x")))) - .await; + let (gone, kept) = (node("ws:main/agent:flow"), node("ws:main/agent:chat")); + let erased = learning("flow fact", &gone); + engine.store(erased.clone()).await.unwrap(); + engine.store(learning("chat fact", &kept)).await.unwrap(); + + let report = engine + .erase(EraseRequest::new(Reach::subtree(gone))) + .await + .unwrap(); + assert_eq!(report.erased_scopes, 1, "{report:?}"); + assert_eq!(state.count("POST /memory/v1/erasures"), 1); + assert_eq!(state.count("DELETE /memory"), 0, "never the whole memory"); + let bodies = state.seen.lock().unwrap().erasures.clone(); + assert!(bodies[0].get("selector").is_none(), "{}", bodies[0]); + assert_eq!(listed(&engine).await, vec!["chat fact".to_string()]); + let again = engine.store(erased).await.unwrap(); + assert!(!again.replayed, "an erased item's key was released"); +} + +#[tokio::test] +async fn the_hosted_erasure_names_only_the_fields_memory_api_accepts() { + let (endpoint, state) = hosted_double().await; + let engine = hosted_engine(&endpoint); + let at = node("agent:assistant"); + engine.store(learning("a fact", &at)).await.unwrap(); + engine + .erase(EraseRequest::new(Reach::exact(at))) + .await + .unwrap(); + let body = state.seen.lock().unwrap().erasures[0].clone(); + let mut keys: Vec<&String> = body.as_object().unwrap().keys().collect(); + keys.sort(); + assert_eq!(keys, ["audit_note", "scope"], "{body}"); +} + +#[tokio::test] +async fn an_incomplete_hosted_erasure_is_retried() { + let (endpoint, state) = hosted_double().await; + let engine = hosted_engine(&endpoint); + let at = node("agent:assistant"); + engine.store(learning("a fact", &at)).await.unwrap(); + state.erasure_incomplete_for.store(1, Ordering::SeqCst); + + let report = engine + .erase(EraseRequest::new(Reach::exact(at))) + .await + .unwrap(); + assert_eq!(report.erased_scopes, 1); + assert_eq!(state.count("POST /memory/v1/erasures"), 2); + assert!(listed(&engine).await.is_empty()); +} + +#[tokio::test] +async fn a_hosted_erasure_that_stays_incomplete_is_unavailable() { + let (endpoint, state) = hosted_double().await; + let engine = hosted_engine(&endpoint); + let at = node("agent:assistant"); + engine.store(learning("a fact", &at)).await.unwrap(); + state.erasure_incomplete_for.store(10, Ordering::SeqCst); + + let failed = engine.erase(EraseRequest::new(Reach::exact(at))).await; assert!( - matches!(refused, Err(tinymemory_api::Error::Unsupported(_))), + matches!(failed, Err(tinymemory_api::Error::Unavailable(_))), + "{failed:?}" + ); +} + +#[tokio::test] +async fn a_running_direct_erasure_is_polled_until_it_completes() { + let (endpoint, state) = direct_double().await; + let engine = direct_engine(&endpoint); + let at = node("agent:assistant"); + engine.store(learning("a fact", &at)).await.unwrap(); + // The POST and the first two polls answer `running`. + state.erasure_running_for.store(3, Ordering::SeqCst); + + let report = engine + .erase(EraseRequest::new(Reach::exact(at))) + .await + .unwrap(); + assert_eq!(report.receipts, vec!["erasure_1".to_string()]); + assert_eq!(state.count("GET /v1/erasures/erasure_1"), 3); +} + +#[tokio::test] +async fn a_direct_erasure_that_does_not_complete_is_an_error() { + let (endpoint, state) = direct_double().await; + let engine = direct_engine(&endpoint); + let at = node("agent:assistant"); + engine.store(learning("a fact", &at)).await.unwrap(); + state.erasure_running_for.store(1, Ordering::SeqCst); + *state.erasure_ends.lock().unwrap() = Some("failed"); + + let refused = engine.erase(EraseRequest::new(Reach::exact(at))).await; + assert!( + matches!(&refused, Err(tinymemory_api::Error::Engine(m)) if m.contains("failed")), "{refused:?}" ); - assert!(state.seen.lock().unwrap().requests.is_empty()); +} + +#[tokio::test] +async fn a_backend_without_the_scoped_erasure_route_is_unsupported() { + let (endpoint, state) = hosted_double().await; + state.scoped_erase_missing.store(true, Ordering::SeqCst); + let engine = hosted_engine(&endpoint); + let at = node("agent:assistant"); + engine.store(learning("a fact", &at)).await.unwrap(); + let refused = engine.erase(EraseRequest::new(Reach::exact(at))).await; + assert!( + matches!(refused, Err(tinymemory_api::Error::Unsupported(_))), + "a caller falls back to forget: {refused:?}" + ); + assert_eq!(listed(&engine).await, vec!["a fact".to_string()]); } fn whole_tree() -> EraseRequest { @@ -168,20 +276,19 @@ async fn the_hosted_wire_erases_the_whole_memory_in_one_request() { } #[tokio::test] -async fn the_hosted_wire_refuses_a_narrower_erase_without_a_request() { +async fn a_narrower_hosted_erase_goes_scope_by_scope_not_whole_memory() { let (endpoint, state) = hosted_double().await; let engine = hosted_engine(&endpoint); + engine + .store(learning("a fact", &node("agent:a"))) + .await + .unwrap(); let mut learnings = whole_tree(); learnings.kinds = vec![ItemKind::Learning]; - let mut exact = EraseRequest::new(Reach::exact(Namespace::ROOT)); - exact.whole_tree = true; - for req in [learnings, exact] { - let refused = engine.erase(req).await; - assert!( - matches!(refused, Err(tinymemory_api::Error::Unsupported(_))), - "{refused:?}" - ); - } + engine.erase(learnings).await.unwrap(); + assert_eq!(state.count("DELETE /memory"), 0); + assert_eq!(state.count("POST /memory/v1/erasures"), 1); + let missing_interlock = engine .erase(EraseRequest::new(Reach::subtree(Namespace::ROOT))) .await; @@ -192,7 +299,6 @@ async fn the_hosted_wire_refuses_a_narrower_erase_without_a_request() { ), "{missing_interlock:?}" ); - assert!(state.seen.lock().unwrap().requests.is_empty()); } #[tokio::test] @@ -357,3 +463,161 @@ fn the_double_refuses_an_erasure_without_a_scope() { let (status, _) = log.erase(&serde_json::json!({ "confirm_all": true })); assert_eq!(status, 422); } + +#[tokio::test] +async fn a_direct_erasure_that_stays_running_times_out_as_unavailable() { + let (endpoint, state) = direct_double().await; + let engine = direct_engine(&endpoint); + let at = node("agent:assistant"); + engine.store(learning("a fact", &at)).await.unwrap(); + // Running for far longer than the test's erasure deadline. + state.erasure_running_for.store(100_000, Ordering::SeqCst); + + let refused = engine.erase(EraseRequest::new(Reach::exact(at))).await; + assert!( + matches!(&refused, Err(tinymemory_api::Error::Unavailable(m)) if m.contains("pending")), + "{refused:?}" + ); + assert!(state.count("GET /v1/erasures/erasure_1") >= 1); +} + +#[tokio::test] +async fn a_polled_erasure_without_a_status_is_not_completed() { + let (endpoint, state) = direct_double().await; + let engine = direct_engine(&endpoint); + let at = node("agent:assistant"); + engine.store(learning("a fact", &at)).await.unwrap(); + state.erasure_running_for.store(1, Ordering::SeqCst); + state + .erasure_poll_omits_status + .store(true, Ordering::SeqCst); + + let refused = engine.erase(EraseRequest::new(Reach::exact(at))).await; + assert!( + matches!(&refused, Err(tinymemory_api::Error::Engine(m)) if m.contains("status")), + "{refused:?}" + ); +} + +#[tokio::test] +async fn a_direct_erasure_that_keeps_a_non_terminal_status_is_not_done() { + let (endpoint, state) = direct_double().await; + let engine = direct_engine(&endpoint); + let at = node("agent:assistant"); + engine.store(learning("a fact", &at)).await.unwrap(); + // Settles as `cancelled`: neither pending nor completed. + state.erasure_running_for.store(1, Ordering::SeqCst); + *state.erasure_ends.lock().unwrap() = Some("cancelled"); + + let refused = engine.erase(EraseRequest::new(Reach::exact(at))).await; + assert!( + matches!(&refused, Err(tinymemory_api::Error::Engine(m)) if m.contains("cancelled")), + "{refused:?}" + ); +} + +#[tokio::test] +async fn a_hosted_erasure_reports_the_scopes_the_backend_erased() { + let (endpoint, state) = hosted_double().await; + let engine = hosted_engine(&endpoint); + let at = node("agent:assistant"); + engine.store(learning("a fact", &at)).await.unwrap(); + // The scope emptied between the listing and the erasure. + *state.scoped_erase_answer.lock().unwrap() = + Some(serde_json::json!({ "erased": true, "scopes": 0, "erasure_ids": [] })); + + let report = engine + .erase(EraseRequest::new(Reach::exact(at))) + .await + .unwrap(); + assert_eq!(report.erased_scopes, 0, "{report:?}"); + assert!(report.receipts.is_empty()); +} + +#[tokio::test] +async fn a_malformed_hosted_erasure_answer_is_an_error() { + for answer in [ + serde_json::json!({ "erased": true, "scopes": 1, "erasure_ids": [123] }), + serde_json::json!({ "erased": true, "scopes": 1, "erasure_ids": "x" }), + serde_json::json!({ "erased": true, "erasure_ids": [] }), + serde_json::json!({ "erased": true, "scopes": "1", "erasure_ids": [] }), + // Scopes erased, but no receipt for them. + serde_json::json!({ "erased": true, "scopes": 1, "erasure_ids": [] }), + serde_json::json!({ "erased": true, "scopes": 2, "erasure_ids": null }), + serde_json::json!({ "erased": true, "scopes": 1, "erasure_ids": [""] }), + ] { + let (endpoint, state) = hosted_double().await; + let engine = hosted_engine(&endpoint); + let at = node("agent:assistant"); + engine.store(learning("a fact", &at)).await.unwrap(); + *state.scoped_erase_answer.lock().unwrap() = Some(answer.clone()); + let refused = engine.erase(EraseRequest::new(Reach::exact(at))).await; + assert!( + matches!(refused, Err(tinymemory_api::Error::Engine(_))), + "{answer}: {refused:?}" + ); + } +} + +#[tokio::test] +async fn a_synchronous_erasure_answer_without_a_status_is_completed() { + let (endpoint, state) = direct_double().await; + let engine = direct_engine(&endpoint); + let at = node("agent:assistant"); + engine.store(learning("a fact", &at)).await.unwrap(); + state + .erasure_post_omits_status + .store(true, Ordering::SeqCst); + + let report = engine + .erase(EraseRequest::new(Reach::exact(at))) + .await + .unwrap(); + assert_eq!(report.erased_scopes, 1); + assert_eq!( + state.count("GET /v1/erasures/erasure_1"), + 0, + "nothing to poll" + ); +} + +#[tokio::test] +async fn a_polled_status_that_is_not_a_known_string_never_completes_the_erasure() { + for status in [ + serde_json::json!(7), + serde_json::json!(null), + serde_json::json!("weird"), + ] { + let (endpoint, state) = direct_double().await; + let engine = direct_engine(&endpoint); + let at = node("agent:assistant"); + engine.store(learning("a fact", &at)).await.unwrap(); + state.erasure_running_for.store(1, Ordering::SeqCst); + *state.erasure_poll_status.lock().unwrap() = Some(status.clone()); + let refused = engine.erase(EraseRequest::new(Reach::exact(at))).await; + assert!( + matches!(refused, Err(tinymemory_api::Error::Engine(_))), + "{status}: {refused:?}" + ); + } +} + +#[tokio::test] +async fn a_hosted_scope_is_erased_by_an_engine_that_never_held_it() { + let (endpoint, state) = hosted_double().await; + let at = node("agent:assistant"); + hosted_engine(&endpoint) + .store(learning("a fact", &at)) + .await + .unwrap(); + // A fresh engine has no memory of the write: the scopes come from the + // backend's registry. + let other = hosted_engine(&endpoint); + let report = other + .erase(EraseRequest::new(Reach::exact(at))) + .await + .unwrap(); + assert_eq!(report.erased_scopes, 1, "{report:?}"); + assert_eq!(state.count("POST /memory/v1/erasures"), 1); + assert!(listed(&other).await.is_empty()); +} diff --git a/crates/tinymemory-integrations/src/cortex/log/erase.rs b/crates/tinymemory-integrations/src/cortex/log/erase.rs index e226e7e5..cbf7cd4a 100644 --- a/crates/tinymemory-integrations/src/cortex/log/erase.rs +++ b/crates/tinymemory-integrations/src/cortex/log/erase.rs @@ -6,38 +6,216 @@ //! (deleted, not redacted, their write keys released). CortexDB only //! redacts the scopes below it, which keep their keys; `engine::erase` //! names kind scopes only (leaves), deepest first, so that never happens. +//! +//! Both wires erase a scope. Direct posts `{scope, confirm_all}` to +//! CortexDB's own `v1/erasures`; hosted posts `{scope, audit_note}` to the +//! TinyHumans backend's `memory/v1/erasures` passthrough (memory-api's +//! scoped erasure, which takes no `confirm_all` and refuses an unknown +//! field). memory-api pins the scope under the caller's tenant root, erases +//! it and every scope below it with the tenant's own user-actor token, and +//! answers synchronously in its own dialect (no `{success,data}` envelope; +//! see `transport`): `{erased: true, scopes, erasure_ids}`, where `scopes: +//! 0` (nothing stored) is still success. `502 ERASURE_INCOMPLETE` is +//! the only failure retried (any other leaves the outcome unknown); re-erasing a scope is safe. A backend without the +//! route answers 404, reported as [`Error::Unsupported`]: nothing was erased. +//! +//! A Direct erasure is a job. CortexDB answers `completed` when it ran to +//! the end before answering; a `running` answer is polled at +//! `v1/erasures/{id}` until it settles, and anything but `completed` is an +//! error: an erasure that did not finish must never read as done. + +use std::time::Instant; use serde_json::{Value, json}; -use super::Log; +use super::{HOSTED_POLL_CEILING, Log}; use crate::cortex::descriptor::{CortexWire, Route}; -use crate::cortex::error::{Error, Result}; -use crate::cortex::transport::Attempts; +use crate::cortex::error::{Error, Result, error_code}; +use crate::cortex::transport::{Attempts, urlencode}; + +/// How many times a hosted erasure is sent before an incomplete or +/// transient failure surfaces. +const HOSTED_ERASE_ATTEMPTS: u32 = 3; + +/// The code of the retriable `502` that says an erasure did not finish. +const ERASURE_INCOMPLETE: &str = "ERASURE_INCOMPLETE"; + +/// The one status that means the scope is gone. +const COMPLETED: &str = "completed"; + +/// Statuses of an erasure that has not settled yet. +const PENDING: [&str; 4] = ["running", "pending", "queued", "accepted"]; + +/// The audit note every erasure carries. +const AUDIT_NOTE: &str = "tinymemory: erase"; + +/// What one scope's erasure removed. +#[derive(Debug)] +pub(crate) struct Erased { + /// How many scopes the backend erased: one on Direct; on a hosted wire + /// what the backend reports (`0` when nothing was left to erase). + pub(crate) scopes: usize, + /// The erasure ids, once completed. + pub(crate) ids: Vec, +} impl Log { - /// Erases `scope`, returning CortexDB's erasure id. The execute runs to - /// completion before it answers; sending it again erases what is there - /// then, so it is sent once and a failure surfaces. - pub(crate) async fn erase(&self, scope: &str) -> Result { + /// Erases `scope`, returning what was erased once it has completed. + pub(crate) async fn erase(&self, scope: &str) -> Result { + match self.client.wire() { + CortexWire::Direct => self.erase_direct(scope).await.map(|id| Erased { + scopes: 1, + ids: vec![id], + }), + CortexWire::TinyHumans => self.erase_hosted(scope).await, + } + } + + /// Hosted: memory-api's synchronous scoped erasure. Retried on a + /// transient failure (`502 ERASURE_INCOMPLETE` among them), since + /// erasing a scope again erases only what is left. + async fn erase_hosted(&self, scope: &str) -> Result { + let body = json!({ "scope": scope, "audit_note": AUDIT_NOTE }); + let path = self.client.wire().path(Route::Erasures); + let mut attempt = 0; + let answer = loop { + attempt += 1; + match self + .client + .json(reqwest::Method::POST, path, Some(&body), Attempts::Once) + .await + { + Ok(answer) => break answer, + Err(Error::NotFound(message)) => { + return Err(Error::Unsupported(format!( + "the TinyHumans backend has no scoped erasure route ({message})" + ))); + } + // Only an answer that proves the erasure incomplete is + // retried (re-erasing erases what is left). Any other + // failure leaves the outcome unknown, and a repeated + // destructive request would hide it, so it surfaces. + Err(error) + if error_code(&error) == Some(ERASURE_INCOMPLETE) + && attempt < HOSTED_ERASE_ATTEMPTS => + { + log::debug!("[cortex] erasure of {scope} incomplete; retrying ({attempt})"); + tokio::time::sleep(self.timing.poll * 2_u32.pow(attempt - 1)).await; + } + Err(error) => return Err(error), + } + }; + if answer.get("erased").and_then(Value::as_bool) != Some(true) { + return Err(Error::Engine(format!( + "the erasure of {scope} answered without `erased: true`" + ))); + } + let scopes = answer + .get("scopes") + .and_then(Value::as_u64) + .ok_or_else(|| { + Error::Engine(format!( + "the erasure of {scope} answered without a numeric `scopes` count" + )) + })?; + let ids = match answer.get("erasure_ids") { + None | Some(Value::Null) => Vec::new(), + Some(Value::Array(ids)) => ids + .iter() + .map(|id| id.as_str().map(str::to_owned)) + .collect::>>() + .ok_or_else(|| { + Error::Engine(format!( + "the erasure of {scope} answered a non-string entry in `erasure_ids`" + )) + })?, + Some(_) => { + return Err(Error::Engine(format!( + "the erasure of {scope} answered `erasure_ids` that is not an array" + ))); + } + }; + // The receipts are the audit trail of a destructive call: scopes + // erased without one is an incompatible answer, not a success. + if scopes > 0 && ids.iter().all(|id| id.trim().is_empty()) { + return Err(Error::Engine(format!( + "the erasure of {scope} erased {scopes} scope(s) but answered no `erasure_ids`" + ))); + } + Ok(Erased { + scopes: usize::try_from(scopes).map_err(|_| { + Error::Engine(format!( + "the erasure of {scope} answered a `scopes` count ({scopes}) beyond this platform's range" + )) + })?, + ids, + }) + } + + /// Direct: CortexDB's `v1/erasures`, sent once (a failure surfaces), + /// then polled while it runs. + async fn erase_direct(&self, scope: &str) -> Result { let body = json!({ "scope": scope, "confirm_all": true, - "audit_note": "tinymemory: erase", + "audit_note": AUDIT_NOTE, }); + let path = self.client.wire().path(Route::Erasures); let answer = self .client - .json( - reqwest::Method::POST, - self.client.wire().path(Route::Erasures), - Some(&body), - Attempts::Once, - ) + .json(reqwest::Method::POST, path, Some(&body), Attempts::Once) .await?; - answer + let id = answer .get("erasure_id") .and_then(Value::as_str) .map(str::to_owned) - .ok_or_else(|| Error::Engine(format!("the erasure of {scope} answered no erasure_id"))) + .ok_or_else(|| { + Error::Engine(format!("the erasure of {scope} answered no erasure_id")) + })?; + // The synchronous answer may omit its status (it ran to the end). + let mut status = status_of(&answer, true)?; + let limit = self.timing.erasure; + let deadline = Instant::now() + limit; + let mut gap = self.timing.poll; + while PENDING.contains(&status.as_str()) { + let unavailable = || { + Error::Unavailable(format!( + "the erasure of {scope} ({id}) was still pending after {}ms", + limit.as_millis() + )) + }; + let remaining = deadline.saturating_duration_since(Instant::now()); + if remaining.is_zero() { + return Err(unavailable()); + } + let poll = async { + tokio::time::sleep(gap).await; + self.client + .json( + reqwest::Method::GET, + &format!("{path}/{}", urlencode(&id)), + None, + Attempts::RetryTransient, + ) + .await + }; + // The deadline bounds the sleep and every retry of the request, + // not just the gap between polls. + let job = tokio::time::timeout(remaining, poll) + .await + .map_err(|_| unavailable())??; + gap = (gap * 2).min(HOSTED_POLL_CEILING); + // A polled job must say where it stands: a missing status is + // not a completed erasure. + status = status_of(&job, false)?; + log::debug!("[cortex] erasure {id} of {scope} is {status}"); + } + if status != COMPLETED { + return Err(Error::Engine(format!( + "the erasure of {scope} ({id}) ended {status}, not {COMPLETED}" + ))); + } + Ok(id) } /// Erases the caller's entire hosted memory (`DELETE memory`), every @@ -85,3 +263,18 @@ impl Log { Ok(usize::try_from(scopes).unwrap_or(usize::MAX)) } } + +/// An erasure answer's status. CortexDB's synchronous POST answer may omit +/// it (`default_completed`: an id with no status ran to completion); a +/// polled job must carry a string status, and a status of any other type is +/// malformed either way. +fn status_of(answer: &Value, default_completed: bool) -> Result { + match answer.get("status") { + Some(Value::String(status)) => Ok(status.clone()), + None if default_completed => Ok(COMPLETED.to_string()), + other => Err(Error::Engine(format!( + "an erasure answer carried no usable status ({})", + other.map_or_else(|| "missing".to_string(), |v| v.to_string()) + ))), + } +} diff --git a/crates/tinymemory-integrations/src/cortex/log/forget.rs b/crates/tinymemory-integrations/src/cortex/log/forget.rs index 0f46a818..72a43fbb 100644 --- a/crates/tinymemory-integrations/src/cortex/log/forget.rs +++ b/crates/tinymemory-integrations/src/cortex/log/forget.rs @@ -6,6 +6,11 @@ //! needs `confirm_all`, and `confirm_all` with a selector is refused), and //! this crate never writes `confirm_all` at all, so the two mistakes cannot //! meet. +//! +//! Every removal names its cascade, [`FORGET_CASCADE`]. CortexDB's default +//! is `derived_only`, which drops what was derived from the events and +//! leaves the events themselves in place, so a removal that left the +//! cascade out would not remove anything a user wrote. use serde_json::json; @@ -17,6 +22,11 @@ use crate::cortex::transport::Attempts; /// The most event ids one removal names, so each body stays small. pub(crate) const FORGET_BATCH: usize = 100; +/// The cascade every removal sends: the named events go, with everything +/// derived from them. Never left to the engine's default (`derived_only`), +/// which keeps the events. +pub(crate) const FORGET_CASCADE: &str = "redact_events"; + /// How many times a hosted removal is sent before a transient fault surfaces. const HOSTED_FORGET_ATTEMPTS: u32 = 3; @@ -41,6 +51,7 @@ impl Log { "scope": scope, "layers": ["events"], "selector": { "memory_ids": ids }, + "cascade": FORGET_CASCADE, "audit_note": "tinymemory: forget", }); let path = self.client.wire().path(Route::Forget); diff --git a/crates/tinymemory-integrations/src/cortex/log/mod.rs b/crates/tinymemory-integrations/src/cortex/log/mod.rs index 22a3a261..608e89e5 100644 --- a/crates/tinymemory-integrations/src/cortex/log/mod.rs +++ b/crates/tinymemory-integrations/src/cortex/log/mod.rs @@ -76,6 +76,9 @@ pub(crate) struct Timing { /// First gap between polls. Direct polls at this gap; hosted doubles it /// up to [`HOSTED_POLL_CEILING`]. pub(crate) poll: Duration, + /// How long a running Direct erasure is polled before it is reported + /// as not done (the status requests included). + pub(crate) erasure: Duration, } /// Longest gap between hosted polls. A fixed 250ms poll would spend a fifth @@ -88,6 +91,7 @@ impl Default for Timing { visibility: Duration::from_secs(30), settle: Duration::from_secs(10), poll: Duration::from_millis(250), + erasure: Duration::from_secs(300), } } } diff --git a/crates/tinymemory-integrations/src/cortex/testing/log.rs b/crates/tinymemory-integrations/src/cortex/testing/log.rs index 7410fc1e..3d921666 100644 --- a/crates/tinymemory-integrations/src/cortex/testing/log.rs +++ b/crates/tinymemory-integrations/src/cortex/testing/log.rs @@ -12,6 +12,9 @@ //! - the forget selector reads only `memory_ids`; an empty selector without //! `confirm_all` is refused, and a selector with `confirm_all` is refused //! as ambiguous; +//! - the forget `cascade` defaults to `derived_only`, which drops the +//! beliefs built from the named events and **keeps the events**; only +//! `redact_events` removes them, and any other cascade is a 400; //! - recall returns each event with its stored text (no `[role] ` marker, //! as 0.10.3 and 0.10.4 do in `layers.events`), honours `view: //! "descend"`, metadata label filters and the events budget; @@ -169,6 +172,16 @@ impl CortexLog { }) .unwrap_or_default(); let selective = !ids.is_empty(); + let cascade = match body.get("cascade") { + None => "derived_only", + Some(Value::String(cascade)) => cascade.as_str(), + // Present but not a string (null, a number): invalid, never + // the default. + Some(_) => "", + }; + if !matches!(cascade, "derived_only" | "redact_events") { + return (400, json!({ "error_code": "INVALID_CASCADE" })); + } if selective && confirm_all { return ( 400, @@ -181,6 +194,25 @@ impl CortexLog { json!({ "error_code": "EMPTY_SELECTOR_WITHOUT_CONFIRMATION" }), ); } + if cascade == "derived_only" { + // What was derived goes; the events stay. + let named = |e: &Value| { + str_of(e, "/scope") == scope + && (!selective || ids.iter().any(|id| id == str_of(e, "/id"))) + }; + let sources: Vec = self + .events + .iter() + .filter(|e| named(e)) + .map(|e| str_of(e, "/id").to_string()) + .collect(); + self.beliefs + .retain(|belief| !sources.iter().any(|id| id == str_of(belief, "/source"))); + return ( + 200, + json!({ "deleted": { "events": 0 }, "requested": ids.len() }), + ); + } let before = self.events.len(); if selective { let (gone, kept): (Vec, Vec) = diff --git a/crates/tinymemory-integrations/src/cortex/testing/mod.rs b/crates/tinymemory-integrations/src/cortex/testing/mod.rs index e8bd1326..2d81ee32 100644 --- a/crates/tinymemory-integrations/src/cortex/testing/mod.rs +++ b/crates/tinymemory-integrations/src/cortex/testing/mod.rs @@ -133,6 +133,25 @@ pub(crate) struct Double { pub(crate) legacy_conflict_400: AtomicBool, /// Replaces the `data` of the `DELETE /memory` answer (a malformed one). pub(crate) erase_all_answer: Mutex>, + /// The hosted double has no `/memory/v1/erasures` passthrough (an older + /// backend): it answers the router's bare 404. + pub(crate) scoped_erase_missing: AtomicBool, + /// How many hosted erasures answer `502 ERASURE_INCOMPLETE` (retriable) + /// before one completes. + pub(crate) erasure_incomplete_for: AtomicUsize, + /// How many `running` answers a Direct erasure gives (its POST, then + /// its status polls) before it settles. + pub(crate) erasure_running_for: AtomicUsize, + /// Replaces the `status` of polled Direct erasure answers. + pub(crate) erasure_poll_status: Mutex>, + /// The Direct erasure POST answer omits `status` (CortexDB may). + pub(crate) erasure_post_omits_status: AtomicBool, + /// Polled Direct erasure answers (`GET v1/erasures/{id}`) omit `status`. + pub(crate) erasure_poll_omits_status: AtomicBool, + /// Replaces the answer of a hosted scoped erasure that succeeded. + pub(crate) scoped_erase_answer: Mutex>, + /// The status a settled erasure ends with; `completed` when unset. + pub(crate) erasure_ends: Mutex>, } /// The shared handle the routes and tests hold. diff --git a/crates/tinymemory-integrations/src/cortex/testing/routes.rs b/crates/tinymemory-integrations/src/cortex/testing/routes.rs index 1eac4656..4040eabc 100644 --- a/crates/tinymemory-integrations/src/cortex/testing/routes.rs +++ b/crates/tinymemory-integrations/src/cortex/testing/routes.rs @@ -372,8 +372,129 @@ async fn erase( return refused; } state.seen.lock().unwrap().erasures.push(body.clone()); - let result = state.log.lock().unwrap().erase(&body); - relay(&state, result) + let (code, mut answer) = state.log.lock().unwrap().erase(&body); + if code < 300 { + let status = erasure_status(&state); + if !state.erasure_post_omits_status.load(Ordering::SeqCst) { + answer["status"] = json!(status); + } + } + relay(&state, (code, answer)) +} + +/// The erasure's status as this answer reports it: `running` while the +/// test's running budget lasts, then how it ends. +fn erasure_status(state: &Shared) -> &'static str { + if take_one(&state.erasure_running_for) { + "running" + } else { + state.erasure_ends.lock().unwrap().unwrap_or("completed") + } +} + +/// `POST /memory/v1/erasures`: the backend's passthrough to memory-api's +/// scoped erasure, answered in its dialect (no envelope): `{scope, +/// audit_note}` only (any other field is `400 UNKNOWN_FIELD`), the root is +/// `422 ROOT_ERASURE_REFUSED`, and a synchronous `{erased, scope, scopes, +/// erasure_ids}`; an incomplete erasure is a retriable `502`. +async fn hosted_erase( + State(state): State, + uri: Uri, + headers: HeaderMap, + Json(body): Json, +) -> Reply { + if let Some(early) = gate(&state, "POST", &uri, &headers) { + return early; + } + if state.scoped_erase_missing.load(Ordering::SeqCst) { + return ( + StatusCode::NOT_FOUND, + Json(json!({ "message": "Not Found" })), + ); + } + if let Some(unknown) = body.as_object().and_then(|map| { + map.keys() + .find(|k| !matches!(k.as_str(), "scope" | "audit_note")) + }) { + return ( + StatusCode::BAD_REQUEST, + Json(json!({ "error_code": "UNKNOWN_FIELD", "message": unknown })), + ); + } + // A scope that is not a string is a malformed request, not the root. + let Some(scope) = body + .get("scope") + .and_then(Value::as_str) + .map(str::to_string) + else { + return ( + StatusCode::BAD_REQUEST, + Json(json!({ "error_code": "INVALID_SCOPE" })), + ); + }; + if scope.is_empty() || scope == "/" { + return ( + StatusCode::UNPROCESSABLE_ENTITY, + Json(json!({ "error_code": "ROOT_ERASURE_REFUSED" })), + ); + } + if let Some(refused) = refuse_scope(&state, &scope) { + return refused; + } + state.seen.lock().unwrap().erasures.push(body.clone()); + if take_one(&state.erasure_incomplete_for) { + return ( + StatusCode::BAD_GATEWAY, + Json(json!({ "erased": false, "error_code": "ERASURE_INCOMPLETE", "retriable": true })), + ); + } + let held = state.log.lock().unwrap().scopes(&scope).len(); + let mut ids = Vec::new(); + if held > 0 { + let (code, answer) = state + .log + .lock() + .unwrap() + .erase(&json!({ "scope": scope, "confirm_all": true })); + if code >= 300 { + return (status(code), Json(answer)); + } + ids.push(answer["erasure_id"].clone()); + } + if let Some(answer) = state.scoped_erase_answer.lock().unwrap().clone() { + return (StatusCode::OK, Json(answer)); + } + ( + StatusCode::OK, + Json(json!({ "erased": true, "scope": scope, "scopes": ids.len(), "erasure_ids": ids })), + ) +} + +/// `GET {erasures}/{id}`: the erasure's status, unwrapped. +async fn erasure( + State(state): State, + axum::extract::Path(id): axum::extract::Path, + uri: Uri, + headers: HeaderMap, +) -> Reply { + if let Some(early) = gate(&state, "GET", &uri, &headers) { + return early; + } + // A forced status leaves the running budget alone. + if let Some(forced) = state.erasure_poll_status.lock().unwrap().clone() { + return ( + StatusCode::OK, + Json(json!({ "erasure_id": id, "status": forced })), + ); + } + let status = erasure_status(&state); + if state.erasure_poll_omits_status.load(Ordering::SeqCst) { + return (StatusCode::OK, Json(json!({ "erasure_id": id }))); + } + ( + StatusCode::OK, + Json(json!({ "erasure_id": id, "status": status })), + ) } /// The backend's `DELETE /memory`: erases the caller's entire memory. @@ -648,6 +769,7 @@ pub(super) fn direct(state: Shared) -> Router { .route("/v1/recall", post(recall)) .route("/v1/forget", post(forget)) .route("/v1/erasures", post(erase)) + .route("/v1/erasures/{id}", get(erasure)) .route("/v1/answer", post(answer)) .route("/v1/admin/health", get(health)) .route("/v1/admin/version", get(version)) @@ -669,5 +791,7 @@ pub(super) fn hosted(state: Shared) -> Router { .route("/memory/answer", post(answer)) .route("/memory/scopes", get(scopes)) .route("/memory", axum::routing::delete(erase_all)) + .route("/memory/v1/erasures", post(hosted_erase)) + .route("/memory/v1/erasures/{id}", get(erasure)) .with_state(state) } diff --git a/crates/tinymemory-integrations/src/cortex/transport/failure.rs b/crates/tinymemory-integrations/src/cortex/transport/failure.rs index 543ffeee..a49ec93b 100644 --- a/crates/tinymemory-integrations/src/cortex/transport/failure.rs +++ b/crates/tinymemory-integrations/src/cortex/transport/failure.rs @@ -177,10 +177,17 @@ pub(crate) fn hosted_status_error( let parsed: Option = serde_json::from_str(body).ok(); let code = parsed .as_ref() - .and_then(|v| v.get("errorCode")) - .and_then(Value::as_str) - .map(clean_code) - .filter(|c| !c.is_empty()) + // The backend's own `errorCode`, or memory-api's `error_code` on a + // route the backend passes through unwrapped (`memory/v1/*`). + // A code that is null, not a string or empty once cleaned does not + // shadow the other key. + .and_then(|v| { + ["errorCode", "error_code"] + .iter() + .filter_map(|key| v.get(*key).and_then(Value::as_str)) + .map(clean_code) + .find(|c| !c.is_empty()) + }) .unwrap_or_else(|| default_code(status)); let message = parsed.as_ref().and_then(|v| v.get("error")).map_or_else( || body.to_string(), diff --git a/crates/tinymemory-integrations/src/cortex/transport/failure_tests.rs b/crates/tinymemory-integrations/src/cortex/transport/failure_tests.rs index 5d3ab90f..d5a3f0cc 100644 --- a/crates/tinymemory-integrations/src/cortex/transport/failure_tests.rs +++ b/crates/tinymemory-integrations/src/cortex/transport/failure_tests.rs @@ -251,3 +251,23 @@ fn a_write_the_indexer_has_not_reached_is_transient() { let hosted = hosted_status_error("h", "memory/experience", StatusCode::REQUEST_TIMEOUT, body); assert!(matches!(hosted, Error::Unavailable(_)), "{hosted:?}"); } + +#[test] +fn an_unusable_error_code_falls_back_to_the_other_key() { + for body in [ + r#"{"errorCode":null,"error_code":"INVALID_BODY"}"#, + r#"{"errorCode":7,"error_code":"INVALID_BODY"}"#, + r#"{"errorCode":"-- ","error_code":"INVALID_BODY"}"#, + ] { + let error = hosted_status_error("h", "memory/x", StatusCode::BAD_REQUEST, body); + assert_eq!(error_code(&error), Some("INVALID_BODY"), "{body}"); + } + // The primary key still wins when it is usable. + let error = hosted_status_error( + "h", + "memory/x", + StatusCode::BAD_REQUEST, + r#"{"errorCode":"FIRST","error_code":"SECOND"}"#, + ); + assert_eq!(error_code(&error), Some("FIRST")); +} diff --git a/crates/tinymemory-integrations/src/cortex/transport/mod.rs b/crates/tinymemory-integrations/src/cortex/transport/mod.rs index 1b155cd8..465e9473 100644 --- a/crates/tinymemory-integrations/src/cortex/transport/mod.rs +++ b/crates/tinymemory-integrations/src/cortex/transport/mod.rs @@ -358,9 +358,11 @@ impl HttpClient { } let bytes = body::read_capped(response, label).await?; match self.wire { - CortexWire::TinyHumans => failure::unwrap_envelope(self.host(), label, status, &bytes), - CortexWire::Direct if bytes.is_empty() => Ok(Value::Null), - CortexWire::Direct => serde_json::from_slice(&bytes).map_err(|_| { + CortexWire::TinyHumans if !speaks_cortex(path) => { + failure::unwrap_envelope(self.host(), label, status, &bytes) + } + _ if bytes.is_empty() => Ok(Value::Null), + _ => serde_json::from_slice(&bytes).map_err(|_| { Error::Engine(format!( "memory API {label} on {} returned invalid JSON", self.host() @@ -370,6 +372,17 @@ impl HttpClient { } } +/// The TinyHumans backend's passthrough of CortexDB's own `/v1` dialect, +/// mounted under `memory/`: memory-api's status and body, no +/// `{success,data}` envelope. +pub(crate) const HOSTED_CORTEX_PREFIX: &str = "memory/v1/"; + +/// Whether a hosted `path` is answered in CortexDB's own dialect rather +/// than the backend's envelope. +fn speaks_cortex(path: &str) -> bool { + path.starts_with(HOSTED_CORTEX_PREFIX) +} + /// The route part of a path, for messages: query strings carry scopes and /// cursors, which are noise in an error. fn label(path: &str) -> &str { diff --git a/crates/tinymemory-integrations/tests/live_cortexdb.rs b/crates/tinymemory-integrations/tests/live_cortexdb.rs index 64d37e9c..bb57d75f 100644 --- a/crates/tinymemory-integrations/tests/live_cortexdb.rs +++ b/crates/tinymemory-integrations/tests/live_cortexdb.rs @@ -432,7 +432,8 @@ async fn every_kind_is_exported_whole_across_pages() { } } -/// Direct only: the hosted wire has no erasure route. +/// Direct only for now: the hosted `memory/v1/erasures` passthrough is +/// covered by the wire double until it is live on the backend. #[tokio::test] async fn an_erased_node_is_gone_and_its_items_store_anew() { let _alone = ONE_AT_A_TIME.lock().await; diff --git a/docs/architecture/cortex-wire.md b/docs/architecture/cortex-wire.md index 0b786065..3dbf2a06 100644 --- a/docs/architecture/cortex-wire.md +++ b/docs/architecture/cortex-wire.md @@ -45,6 +45,8 @@ for. Both fail with `Error::Unsupported` before any request. | Health | `v1/admin/health` | `memory/scopes` | GET | | Scopes (registered scopes under a prefix) | `v1/scopes/list` | `memory/scopes` | GET | | Build beliefs (one scope) | `v1/beliefs/build` | none (never sent) | POST | +| Erase (one scope) | `v1/erasures` | `memory/v1/erasures` (unwrapped) | POST | +| Erasure status (Direct only) | `v1/erasures/{id}` | | GET | | Whoami (Direct only) | `v1/auth/whoami` | | GET | The endpoint is joined with the route, so a base URL with a path prefix keeps @@ -215,9 +217,16 @@ strict (an unknown key, or a `null` instructions, is a 400). ```json { "scope": "...", "layers": ["events"], "selector": { "memory_ids": ["evt_1", "evt_2"] }, + "cascade": "redact_events", "audit_note": "tinymemory: forget" } ``` +The cascade is always named. CortexDB's default, `derived_only`, removes what +was derived from the events and keeps the events, so a forget that left it out +would not remove anything written. memory-api tombstones a forget by +`memory_ids` (not one by labels or time range), so this is the forget to use +for a real delete. + At most 100 ids per request. The id field is exactly `memory_ids`: an unrecognised or empty selector means *the whole scope* to CortexDB (an empty selector needs `confirm_all`, and `confirm_all` beside a selector is refused). @@ -246,7 +255,7 @@ scopes. There is no cursor (v0.10.5): `limit` defaults to 50 and is clamped to 1000, and `prefix` matches whole segments. At 1000 paths a read logs a warning and an export refuses, since some scopes may be missing. -### Erase: `v1/erasures` (Direct only) +### Erase: `v1/erasures` and `memory/v1/erasures` ```json { "scope": "app:tinymemory/agent:assistant/app:learnings", "confirm_all": true, "audit_note": "tinymemory: erase" } @@ -262,19 +271,36 @@ Scopes below it are only redacted and keep their keys for 24 hours, so a re-sent write there replays and stores nothing. A whole-scope erasure of six 240 KB events took about 8 s on v0.10.5; small scopes take well under a second. Every erasure drops every recall pack the server holds, as a forget -does. +does. A `running` answer is polled at `GET v1/erasures/{id}` until it +settles (five minutes at most); any final status but `completed` is an error. + +Hosted, an erase narrower than the whole tree posts each kind scope to the +backend's `memory/v1/erasures` passthrough of memory-api's scoped erasure: + +```json +{ "scope": "app:tinymemory/agent:assistant/app:learnings", "audit_note": "tinymemory: erase" } +``` + +No `confirm_all` (memory-api refuses an unknown field with `400 +UNKNOWN_FIELD`). memory-api pins the scope under the tenant root, erases it and +everything below it with the tenant's user-actor token, and answers +synchronously and unwrapped: +`{"erased": true, "scope": "...", "scopes": , "erasure_ids": [...]}` +(`scopes: 0` means nothing was stored, still a success). `502 +ERASURE_INCOMPLETE` (`retriable: true`) is retried up to three times; the root +is `422 ROOT_ERASURE_REFUSED`. A backend without the route (404) is reported as +`Unsupported`, so a caller can fall back to a forget. ### Erase everything: `DELETE memory` (TinyHumans only) -The backend proxies no per-scope erasure, so the hosted engine erases only -the whole tree (`EraseRequest` with `whole_tree`, the root with its -descendants, every kind), in one `DELETE memory` with no body. It erases the +The hosted engine erases the whole tree (`EraseRequest` with `whole_tree`, +the root with its descendants, every kind) in one `DELETE memory` with no body. It erases the caller's **entire** hosted memory, every scope under their tenant, whichever layout or client wrote it, and answers `{"success": true, "data": {"erased": true, "scopes": }}`; `EraseReport.erased_scopes` is `n` and there are no receipts. It is sent once. -A narrower erase refuses with `Unsupported` before any request, and a backend -without the route (404) is reported as `Unsupported` too. +A narrower erase goes through `memory/v1/erasures` (above). A backend without +the route (404) is reported as `Unsupported`. ### Build beliefs: `v1/beliefs/build` (Direct only) diff --git a/docs/specs/memory-v2.md b/docs/specs/memory-v2.md index 2c85cd08..4d515171 100644 --- a/docs/specs/memory-v2.md +++ b/docs/specs/memory-v2.md @@ -228,7 +228,10 @@ pub struct EraseReport { pub erased_scopes: usize, pub receipts: Vec } - **CortexDB, Direct wire:** sends one `/v1/erasures` (`confirm_all`) per registered kind scope, deepest first, and returns the erasure ids as `receipts`. -- **CortexDB, hosted wire:** refuses (the backend proxies no erasure route). +- **CortexDB, hosted wire:** the whole tree in one `DELETE memory`; anything + narrower the same as Direct, through the TinyHumans backend's + `memory/v1/erasures` passthrough (memory-api's synchronous scoped erasure, + pinned under the tenant's root). ### Bulk store