Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
aa48fc6
refactor(sources): split source modules into mod and test files
senamakel Oct 8, 2026
e9d49a2
chore(sources): remove unused reader and type re-exports
senamakel Oct 8, 2026
69630cd
feat(tinymemory-integrations): add reader dispatch for source types
senamakel Oct 8, 2026
a318b2a
refactor(items): drop per-source metadata mapping
senamakel Oct 8, 2026
095f10d
feat(meta): expose source layout metadata
senamakel Oct 8, 2026
0a2f881
test: add unit tests for conformance fixtures, meta filters, and cort…
senamakel Oct 8, 2026
4fc4482
test(integrations): add coverage for item source helpers
senamakel Oct 8, 2026
6f48a50
test(sources): drop removed fields from fixtures and update kind asse…
senamakel Oct 8, 2026
f9da351
test(conformance): use import source kind in feed fixture
senamakel Oct 8, 2026
6752696
test(tools): drop removed source kinds from tool contract fixtures
senamakel Oct 8, 2026
63174a6
docs(sources): trim network and composio sections
senamakel Oct 8, 2026
26f62cc
docs: drop network and composio sources from feature docs
senamakel Oct 8, 2026
bb0c72a
docs: drop sources-network feature from docs and ci
senamakel Oct 8, 2026
f736afb
ci: rename sources feature matrix entry
senamakel Oct 8, 2026
aa078bf
feat(sources): add network readers and Composio normalisers
senamakel Oct 8, 2026
d554811
feat(composio): add per-toolkit source modules for composio integrations
senamakel Oct 8, 2026
85f35bd
feat(integrations): add items source and reader modules
senamakel Oct 8, 2026
3f88e90
refactor(sources): retire the composio source kind
senamakel Oct 8, 2026
58ffac5
docs: remove Composio source support from documentation
senamakel Oct 8, 2026
cb8876d
Merge remote-tracking branch 'origin/main' into remove-composio-memor…
senamakel Oct 8, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ else on request, so a host pays only for what it uses.
| `documents` | `documents` | Format sniffing and conversion to markdown, producing `StoreItem::Document` |
| `documents-office` | `documents::OfficeConverter` | PDF, DOCX, PPTX and XLSX to markdown (implies `documents`) |
| `brain` | `brain` | Files into `tinymemory_tools::BrainDocument`s, by the source type their format implies (implies `documents`) |
| `sources` | `sources` | Folder, file and conversation readers, Composio normalisers (implies `documents`) |
| `sources` | `sources` | Folder, file and conversation readers (implies `documents`) |
| `sources-network` | `sources::fetch` and the network readers | GitHub, RSS and web-page readers and `fetch_url`, behind the SSRF guard (implies `sources`) |
| `safety` | `safety` | Secret and PII scrubbing of a `StoreItem` |
| `legacy-import` | `import` | Migrating a v1 (embedded TinyCortex) workspace into any engine |
Expand Down
10 changes: 5 additions & 5 deletions crates/tinymemory-api/src/meta/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -156,26 +156,27 @@ pub enum SourceKind {
Github,
/// An RSS or Atom feed.
Rss,
/// A Composio toolkit payload (Gmail, Slack, Notion, ...).
Composio,
/// A host conversation.
Conversation,
/// Written by an agent directly; the default.
#[default]
Agent,
/// Imported from a legacy store.
///
/// The retired `composio` kind decodes to this variant, so items stored
/// under it stay readable.
#[serde(alias = "composio")]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve legacy source labels when filtering imports

When an existing CortexDB event is decoded, this alias changes its metadata from composio to Import, but the event still carries the original tm:k:digest("composio") label. labels::narrowing now generates only tm:k:digest("import") for a MetaFilter selecting Import, so CortexDB discards those legacy items before decoding them. Consequently an unfiltered explore can report an import bucket that becomes empty when narrowed, and source-filtered fetch/list/forget cannot reach the supposedly readable stored Composio data; include the legacy label when narrowing Import or otherwise migrate/preserve the stored label identity.

AGENTS.md reference: AGENTS.md:L26-L29

Useful? React with 👍 / 👎.

Import,
}

impl SourceKind {
/// Every kind, in declaration order.
pub const ALL: [Self; 9] = [
pub const ALL: [Self; 8] = [
Self::Folder,
Self::File,
Self::Link,
Self::Github,
Self::Rss,
Self::Composio,
Self::Conversation,
Self::Agent,
Self::Import,
Expand All @@ -190,7 +191,6 @@ impl SourceKind {
Self::Link => "link",
Self::Github => "github",
Self::Rss => "rss",
Self::Composio => "composio",
Self::Conversation => "conversation",
Self::Agent => "agent",
Self::Import => "import",
Expand Down
11 changes: 11 additions & 0 deletions crates/tinymemory-api/src/meta/mod_tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -43,3 +43,14 @@ fn source_kind_wire_strings_match_serde() {
assert_eq!(json, serde_json::json!(kind.as_str()));
}
}

#[test]
fn the_retired_composio_source_kind_decodes_as_import() {
let kind: SourceKind =
serde_json::from_value(serde_json::json!("composio")).expect("legacy kind decodes");
assert_eq!(kind, SourceKind::Import);
assert_eq!(
serde_json::to_value(SourceKind::Import).expect("serialise"),
serde_json::json!("import")
);
}
4 changes: 2 additions & 2 deletions crates/tinymemory-integrations/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ tinymemory-api = { path = "../tinymemory-api" }
# `MemoryEngine`, `BearerSource`, `DocumentConverter` and `SourceReader` are
# object-safe async traits.
async-trait = { version = "0.1", optional = true }
# Every wire body, envelope, cursor, Composio payload and checkpoint is JSON.
# Every wire body, envelope, cursor and checkpoint is JSON.
serde = { version = "1", features = ["derive"], optional = true }
serde_json = { version = "1", optional = true }
# The typed errors of `documents`, `sources` and `import`.
Expand Down Expand Up @@ -122,7 +122,7 @@ documents-office = ["documents", "dep:pdf-extract", "dep:calamine", "dep:quick-x
# placed under the source type their format implies.
brain = ["documents", "dep:tinymemory-tools"]
# `sources`: readers that turn folders, files and conversations into
# `StoreItem`s, and the Composio payload normalisers. Links no HTTP stack.
# `StoreItem`s. Links no HTTP stack.
sources = ["documents", "dep:schemars", "dep:regex", "dep:walkdir", "dep:chrono", "dep:log", "dep:tracing"]
# The readers that fetch over the network — GitHub, RSS, web pages — and
# `sources::fetch::fetch_url`, all behind the shared SSRF guard.
Expand Down
2 changes: 1 addition & 1 deletion crates/tinymemory-integrations/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ wants one integration enables one feature and links nothing else.
| `cortex`, `registry`, `config` | `cortex` (default) | `CortexEngine` over two wires (`cortexdb`, `tinyhumans`); `list_engines`, `build_engine`, `EngineCredential`; `MemoryConfig` | [`src/cortex/README.md`](src/cortex/README.md) |
| `documents` | `documents` | Format sniffing and conversion to markdown, producing `StoreItem::Document`. No I/O. | [`src/documents/README.md`](src/documents/README.md) |
| `documents::OfficeConverter` | `documents-office` | PDF, DOCX, PPTX and XLSX to markdown, in process | (same) |
| `sources` | `sources` | Readers for folders, files and conversations; Composio payload normalisers; `collect_items`. Links no HTTP stack. | [`src/sources/README.md`](src/sources/README.md) |
| `sources` | `sources` | Readers for folders, files and conversations; `collect_items`. Links no HTTP stack. | [`src/sources/README.md`](src/sources/README.md) |
| `sources::fetch`, GitHub, RSS and web-page readers | `sources-network` | The network readers and `fetch_url`, all behind the SSRF guard | (same) |
| `safety` | `safety` | Secret and PII scrubbing of a `StoreItem` before it is stored | [`src/safety/README.md`](src/safety/README.md) |
| `import` | `legacy-import` | Reads a v1 (embedded TinyCortex) workspace and migrates it into any engine, resumably | [`src/import/README.md`](src/import/README.md) |
Expand Down
2 changes: 1 addition & 1 deletion crates/tinymemory-integrations/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
//! | [`cortex`], [`registry`], [`config`] | `cortex` (default) | The CortexDB engine over its two wires, and building one from configuration |
//! | `documents` | `documents`, `documents-office` | Format sniffing and conversion to markdown, emitting `StoreItem::Document` |
//! | `brain` | `brain` | Converting files into the brain documents `tinymemory_tools::Brain` ingests, by source type |
//! | `sources` | `sources`, `sources-network` | Readers turning folders, files, links, GitHub, RSS, Composio payloads and conversations into `StoreItem`s |
//! | `sources` | `sources`, `sources-network` | Readers turning folders, files, links, GitHub, RSS, and conversations into `StoreItem`s |
//! | `safety` | `safety` | Secret and PII scrubbing for a `StoreItem` before it is stored |
//! | `import` | `legacy-import` | Migrating a legacy v1 (embedded TinyCortex) workspace into any engine |
//!
Expand Down
6 changes: 2 additions & 4 deletions crates/tinymemory-integrations/src/sources/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

`tinymemory_integrations::sources` (features `sources` and `sources-network`):
readers that turn a source into `StoreItem`s — a folder, a single file, a web
page, a GitHub repository, an RSS feed, a Composio toolkit payload, or the
page, a GitHub repository, an RSS feed, or the
host's local conversation threads. Conversion to markdown and language
detection come from the sibling [`documents`](../documents/README.md) module.
Architecture overview:
Expand All @@ -21,7 +21,6 @@ checks it with `MemorySourceEntry::validate`, nothing more.
| `fetch` | one URL into a `RawDocument` or a link item (`sources-network`); the RSS and web-page readers fetch through it with their own body caps |
| `fetch::ssrf` | the SSRF guard: scheme and host policy, one address classifier for literal and resolved addresses, a public-only DNS resolver, per-hop redirect checks, and a capped body reader |
| `items` | reader output to `StoreItem`s with `MemoryMeta` filled per kind; `collect_items` drives a reader end to end |
| `composio` | toolkit normalisers (Gmail, Slack, GitHub, Linear, Notion, ClickUp), the `fields::pick_str` lookup they share, `normalise_payload` and `payload_items`. `readers::composio::ComposioReader` is only a placeholder reader |
| `error` | the module `Error`, mapped onto `tinymemory_api::Error` |

## Kinds and metadata
Expand All @@ -35,7 +34,6 @@ Every item's `meta.source` is `SourceRef { kind, id: Some(entry.id) }`.
| `web_page` | `Link` | document | `url` |
| `github_repo` | `Github` | document | `repo` (`owner/name`), `commit` (commits), `url` (issues, PRs), `observed_at` |
| `rss_feed` | `Rss` | document | `url` (the entry's link), `observed_at` (published) |
| `composio` | `Composio` | document | `tags = [toolkit]`; payloads add `url`, `observed_at`, `thread_id`, `repo` |
| `conversation` | `Conversation` | conversation | `workspace`, `thread_id`, `turns`, `observed_at` (last turn) |

## Folder selection
Expand Down Expand Up @@ -74,7 +72,7 @@ named and numeric entities decode the same way everywhere.

## Features

- `sources` — the local readers, `items`, `composio` and `types`; implies
- `sources` — the local readers, `items` and `types`; implies
`documents`. Links no HTTP stack.
- `sources-network` — adds the GitHub, RSS and web-page readers, `fetch`, and
the SSRF guard.
69 changes: 0 additions & 69 deletions crates/tinymemory-integrations/src/sources/composio/clickup/mod.rs

This file was deleted.

This file was deleted.

Loading
Loading