perf(controlplane): make referrer discovery independent of the caller's organization count by javirln · Pull Request #3445 · chainloop-dev/chainloop

javirln · 2026-09-16T06:33:39Z

Referrer discovery is the query behind the Trust Hub graph. Its cost grew with how many organizations the caller belongs to rather than with the data being asked about.

The project visibility filter is a constant size

The filter was built by OR-ing one EXISTS subquery per organization the caller belongs to, so its shape grew with the caller's membership list. At 23 organizations the planner stops walking the root's edges and evaluates the whole disjunction as a row filter over the referrer table.

It now emits at most two branches for any number of organizations: one organization list matched on the project's own organization column, and one list of (organization, project) pairs for individually granted projects. Pairs rather than two independent sets, so a project granted in one organization cannot be reached through another the caller happens to share.

A project-scoped discover goes from 34.3 ms to 6.8 ms, returning identical rows.

Also here

The referrer visibility filter is expressed as a correlated EXISTS rather than through Ent's generated IN (subquery), and the references query no longer asks for a DISTINCT it cannot need — the traversal's join table is keyed on (referrer_id, referred_by_id) and the projection carries the primary key. Neither is a measured speed-up; both forms plan the same way and run within noise of each other at every node degree measured.

Tests

The hand-written visibility filter is required to return exactly what the Ent-generated one does, across every state it can meet. Project visibility is covered against the database: a caller granted a project reaches it, a caller restricted to none does not, and a grant naming one organization does not reach a project living in another.

Compatibility

No schema change, no migration and no new index. No API change, so existing clients — including the CLI — see the same responses.

Assisted-by: Claude Code

The filter answering "which projects may this caller see" was built by OR-ing one condition per organization the caller belongs to. A caller in 23 organizations produced 23 OR-ed EXISTS subqueries, and a disjunction that wide stops the planner using the root's index: the surrounding referrer query became a sequential scan of every referrer in the deployment rather than a walk of the root's edges. Emit each of the two ways a caller reaches a project once for the whole set of organizations instead of once per organization, so the filter no longer grows with the caller's organization count: - every project of an organization whose role carries no project restriction, matched on the project's organization column, which needs no subquery - an individually granted project, matched as an (organization, project) pair The second branch matches pairs rather than intersecting an organization set with a project set. Intersecting them would also admit a project that is granted to the caller in one organization and happens to sit in another organization the caller is restricted in, which is not a grant the caller holds; a test pins that. Verified on a local dataset of 79,605 attestations with the caller in 23 organizations. Responses captured before and after are byte identical: 24 calls as an owner across attestation and material roots, scoped and unscoped, and 28 calls as an organization member with three project grants, including projects that must stay hidden and do. Project-scoped calls for the owner are 5.6 times faster on average, 35.8 ms to 6.4 ms. A member whose grants sit in one organization is unchanged, as expected: the old filter already collapsed to a single condition for them, so there was no fan-out to remove. Assisted-by: Claude Code Signed-off-by: Javier Rodriguez <javier@chainloop.dev> Chainloop-Trace-Sessions: 04123fef-eb07-444e-b017-b168c4fa1889

chainloop-platform · 2026-09-16T06:33:49Z

AI Session Checks — ⚠️ 1 session(s) missing

Missing AI Coding Sessions

We detected commits in this PR that were AI-assisted, but the matching Chainloop Trace session(s) could not be found in Chainloop.

Please make sure the AI coding session evidence has been sent by the Chainloop CLI, or add the skip-ai-session label to this PR to bypass this check.

Learn more about Chainloop Trace.

Security Checks — ✅ 8 passing

✅ `secret-scan`

Status	Policy	Messages
✅ Passed	`secrets-detection`	-

✅ `sast-scan`

Status	Policy	Messages
✅ Passed	`owasp-top10-2025`	-
✅ Passed	`sast`	-
✅ Passed	`cwe-top25`	-
✅ Passed	`cwe-top26-40-cusp`	-

PR info

Status	Policy	Messages
✅ Passed	`pr-min-approvals`	-
✅ Passed	`pr-description-required`	-
✅ Passed	`pr-user-story-linked`	-

⏭️ 3 scans not applied

Scan	Reason
`vulnerability-scan`	no manifest/lockfile changed
`github-actions-scan`	no workflow files changed
`iac-scan`	no IaC files changed

View attestation ↗

Powered by Chainloop and Chainloop Trace

cubic-dev-ai

All reported issues were addressed across 22 files

_{Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic}

Ent's HasWorkflowsWith renders this filter as `id IN (subquery)`, which describes a set to build rather than a condition to test, and the organization has to be reached through a nested relation predicate that Ent renders back into the same shape. Writing the correlated EXISTS by hand says what is meant: for this referrer, does a live workflow exist in one of these organizations. The references query also stops asking for DISTINCT. Ent adds one to every traversal because an edge query can multiply rows; this one cannot, since the join table is keyed on (referrer_id, referred_by_id) and the selected columns include the primary key. Neither of these is a speed-up, and neither is claimed as one. Both forms plan the same way and run within noise of each other at every node degree measured. They are here because they state the intent, and because the DISTINCT was doing nothing. The differential test added alongside requires the hand-written filter to return exactly what the generated one does. Assisted-by: Claude Code Signed-off-by: Javier Rodriguez <javier@chainloop.dev>

…ch the generated one The visibility filter was rewritten by hand because Ent's generated HasWorkflowsWith emits an uncorrelated IN. Hand-written SQL standing in for generated code needs more than a shape assertion, so this runs both forms against the same database and requires identical results. The fixture covers every state the filter can meet — a referrer with no workflow, with a live workflow in an allowed organization, in another organization, in a deleted workflow, in a deleted and a live one, in two live workflows of one organization, and in workflows of two organizations — crossed with one, several, duplicated, unknown and empty organization lists. It also pins the two query shapes the filter is used in, a plain lookup and a traversal through references, and that a referrer in several matching workflows comes back once, which is what makes dropping DISTINCT safe. It follows the pattern the repository already uses for database-backed tests: a testify suite on testhelpers.UseCasesEachTestSuite rather than its own container. The package's external test reaches the two filters through export_test.go, because the shared suite imports this package and so cannot be imported from inside it. Each assertion was checked against a deliberately broken filter: removing the deleted_at check, removing the correlation, joining on the wrong column of the through table, and correlating on the wrong side are all caught. Assisted-by: Claude Code Signed-off-by: Javier Rodriguez <javier@chainloop.dev> Chainloop-Trace-Sessions: 04123fef-eb07-444e-b017-b168c4fa1889

…onnect A client exploring the graph collects a set of referrers and then needs to know which of them are connected. Today the only way to learn that is to discover every one of them again, one request each, even though their identities are already in hand. DiscoverEdges answers the whole question in one request. It takes the referrers the caller holds, as digest and kind because a digest alone does not identify one, and returns the connections between them as pairs of indexes into the request. Referrers the caller cannot see, or that do not exist, contribute no edges rather than failing the call. Each connection comes back once, and the request is capped at a thousand referrers. The work is bounded by how many referrers were asked about, not by how many references any of them has: a shared material can be referenced by a hundred thousand attestations and asking about a hundred nodes must not walk that. The store is therefore read from the attestation side of each connection, whose degree is the number of materials in one attestation. That is complete because a connection only ever exists between an attestation and one of its materials or subjects, and the attestation's own row is always written. Ingest writes the material's row pointing back as well, but this does not rely on it: older data holds many connections in the attestation direction only. Visibility is resolved through the same organization, project and role filters as DiscoverPrivate, and the endpoint requires the same read policy. On a local dataset of 79,605 attestations it returns exactly the edge set a full traversal finds, verified across eight roots including the two highest-degree nodes: no edge missing, none extra. It answers in 7 to 13 ms for the 13 to 105 node sets those traversals produce, and 38 ms at the thousand-node cap. Also corrects the comment on the Unique(false) added earlier: what makes dropping the DISTINCT safe is that the selected columns include the primary key and no predicate there introduces a join, which is a wider and more durable statement than the join table's own key. Assisted-by: Claude Code Signed-off-by: Javier Rodriguez <javier@chainloop.dev> Chainloop-Trace-Sessions: 04123fef-eb07-444e-b017-b168c4fa1889

Assisted-by: Claude Code Signed-off-by: Javier Rodriguez <javier@chainloop.dev> Chainloop-Trace-Sessions: 04123fef-eb07-444e-b017-b168c4fa1889

… request The endpoint read the caller's role-based project grants only when the request named a project, so a request without one resolved its referrers on organization membership alone. A member restricted to a subset of an organization's projects was refused those referrers by DiscoverPrivate and told by DiscoverEdges that they are connected. No content crossed the boundary, but the existence of a relationship did, and one request can ask about a thousand referrers. An optional filter narrows what the caller asked for. It cannot be what decides what they are allowed to see, so the grants now bound which referrers resolve on every request. The project is matched on its own row rather than through the workflow's organization column, so a grant cannot be satisfied by a project that lives in a different organization to the one it was granted in. The regression test states the rule as the comparison that matters: for the same caller and the same referrers, this endpoint must reveal no more than the one it sits beside. It fails against the previous predicate. Assisted-by: Claude Code Signed-off-by: Javier Rodriguez <javier@chainloop.dev> Chainloop-Trace-Sessions: 04123fef-eb07-444e-b017-b168c4fa1889

The RPC picks between two authorization paths and maps referrers onto the indexes the caller gave. Neither was covered: the use-case tests enter below the switch, so the routing and the index mapping could both regress while everything still passed. Covers the logged-in user path, the API token path, a referrer the caller cannot resolve, and a call with no credential at all. Project scoping is deliberately left out. It resolves through workflow runs and this fixture has none, so every project name answers the same way and any assertion would hold for the wrong reason. Assisted-by: Claude Code Signed-off-by: Javier Rodriguez <javier@chainloop.dev> Chainloop-Trace-Sessions: 04123fef-eb07-444e-b017-b168c4fa1889

…ey plan Three comments asserted a query plan: that the generated IN form makes the planner materialise every referrer in the caller's organizations, that correlating turns it into a per-candidate lookup, and that the project filter plans as a semi-join on the run digest index. Measurement does not support the first two — both visibility forms plan identically and run within noise of each other at every node degree tried, on two datasets — and the third was never checked. They now say what the code expresses and what has to stay true, which is the part that does not move with the table statistics: the join table's key is why the DISTINCT is a no-op, and a predicate that joins the outer query is what would make it matter. Assisted-by: Claude Code Signed-off-by: Javier Rodriguez <javier@chainloop.dev> Chainloop-Trace-Sessions: 04123fef-eb07-444e-b017-b168c4fa1889

The request's nodes are a set, and a caller assembling them from more than one attestation will list a shared material twice. That was already accepted, but which of the repeated positions the answer named fell out of a map overwrite, so a client that drew a node per position had one copy connected and the other floating with no way to tell why. A repeated referrer is now answered at the position it first appeared in, and the request message says so. Naming fewer than two distinct referrers is refused rather than answered with an empty set: two positions holding one referrer describe no connection, and reading that as "not connected" hides the mistake. The test that covered this asserted only how many edges came back, so it held for either position and pinned nothing. It now asserts the indexes. Assisted-by: Claude Code Signed-off-by: Javier Rodriguez <javier@chainloop.dev> Chainloop-Trace-Sessions: 04123fef-eb07-444e-b017-b168c4fa1889

cubic-dev-ai

All reported issues were addressed across 25 files

_{Requires human review: Auto-approval blocked because this review re-detected 1 unresolved issue already reported by Cubic.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic}

…ined DiscoverEdges built its lookup keys as kind + "-" + digest. Kind is whatever the caller sends, so a separator inside it makes two different referrers share a key: {A, B-C} and {A-B, C} both encode to A-B-C. That made the request look like it named one referrer where it named two, so it was refused, and in the position map one referrer's edges could be reported against the other's index. Both maps are keyed by the pair itself, which is comparable and cannot be ambiguous. No ingested kind contains a separator today, so this was not reachable through the UI or the CLI — but kind arrives from the request, not from the store, so nothing prevented it. Assisted-by: Claude Code Signed-off-by: Javier Rodriguez <javier@chainloop.dev> Chainloop-Trace-Sessions: 04123fef-eb07-444e-b017-b168c4fa1889

… produces Running the full generation target rather than buf on its own showed three things wrong with what was checked in. The OpenAPI spec was stale: the request message now documents that its nodes are a set, and that description belongs in the published spec too. A temporary artifact of the generation, api/gen/temp-openapi, had been added to the repository. The target writes it, merges it into the spec and removes it again, so checking it in leaves the tree dirty every time anyone regenerates. crafting_schema.pb.go carried a difference produced by a local protoc-gen-go rather than by any change here, and is restored to what the repository already had. Assisted-by: Claude Code Signed-off-by: Javier Rodriguez <javier@chainloop.dev> Chainloop-Trace-Sessions: 04123fef-eb07-444e-b017-b168c4fa1889

cubic-dev-ai

No issues found across 23 files

_{Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic}

… for The table calls a field wantOrgs and documents it as the number of organizations expected in the filter's IN list, but only ever used it to decide whether to look for the list at all. The assertion was a substring check, so the case built from 23 organizations passed for a list holding one — the property the test exists to guard was never checked. Values are bound, so the list is counted by its placeholders. The total bound count is asserted too: one value per organization and a pair per granted project, and nothing else. Also removes ProjectVisibilityPredicateForTest. Nothing reads it, and a test bridge is worth only what it lets an external test reach. Assisted-by: Claude Code Signed-off-by: Javier Rodriguez <javier@chainloop.dev> Chainloop-Trace-Sessions: 04123fef-eb07-444e-b017-b168c4fa1889

cubic-dev-ai

All reported issues were addressed across 23 files

_{Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic}

…atabase A grant names an organization as well as a project, and the filter matches the two as a pair for a reason: matching them as independent sets would let a project granted in one organization be reached from another the caller happens to share. That property was asserted only by looking at the SQL the filter renders, which says the pair list is there but not that it does anything. The case now runs against the database, through both the edges request and the reference lookup: a project living in one organization, granted under another, with nothing granted where it actually lives. Dropping the organization from the grant makes it return every edge, so the test fails when the property does. The neighbouring cases already covered no grant, a real grant, and no restriction at all; this was the one missing. Assisted-by: Claude Code Signed-off-by: Javier Rodriguez <javier@chainloop.dev> Chainloop-Trace-Sessions: 04123fef-eb07-444e-b017-b168c4fa1889

cubic-dev-ai

All reported issues were addressed across 7 reviewed files. 7 files intentionally excluded from review.

_{Reply with feedback, questions, or to request a fix.

Re-trigger cubic}

EdgesAmong reads each node by value to count the distinct ones, so a nil entry panicked. The RPC cannot produce one, but the use case is exported and says nothing about it. Assisted-by: Claude Code Signed-off-by: Javier Rodriguez <javier@chainloop.dev> Chainloop-Trace-Sessions: 04123fef-eb07-444e-b017-b168c4fa1889

migmartri · 2026-09-17T10:16:41Z

Measured this against production data, by reconstructing the SQL ent emits for the project-scoped DiscoverPrivate path and running EXPLAIN (ANALYZE, BUFFERS) on each shape. Root was a GIT_HEAD_COMMIT with 2 references, caller in 37 orgs.

Nothing needs to change in the code. Both changes are correct, and they compound:

Shape	Execution time
`main` today	3,610 ms
+ correlated `EXISTS` only	594 ms
+ both changes	178 ms

One correction to the description, though:

Neither is a measured speed-up; both forms plan the same way and run within noise of each other at every node degree measured.

Against main, the correlated EXISTS is a 6.1× speed-up on its own. On current code the uncorrelated IN (subquery) is 3,585 ms of the 3,610 ms: Postgres materialises every referrer visible to the caller — seq scan of 1.79M referrer_workflows rows, then an external merge sort of 1.76M rows spilling 34 MB to disk — in order to filter 2 candidate rows. The correlated form turns that into 2 index probes on referrer_workflows_pkey, 0.09 ms.

I suspect the benchmark missed it because it swept node degree. This cost is independent of node degree — it scales with the size of the caller's visible-referrer set and the row count of referrer_workflows.

The constant-size project filter is doing real work too. With only the EXISTS fix applied, the planner stops walking the root's edges and seq-scans referrers applying the disjunction as a row filter (556 ms) — precisely the failure mode the description predicts. Collapsing the disjunction is what removes it. So the two changes are complementary, and which one looks dominant depends on which is already in place.

One caveat on the 178 ms: my reconstruction inlines the root referrer id as a scalar subquery, which introduces an InitPlan and pushes the planner into a parallel seq scan of referrer_references — 157 of those 178 ms. The real code binds the id directly, so the true figure is lower; everything else in that plan sums to about 3 ms.

Unrelated to performance, and only a suggestion: referrerVisibleInProjects now enforces the RBAC project rule in SQL, while isReferrerVisible still enforces it in Go. Two implementations of one rule that must not drift — might be worth a test asserting they agree, rather than each being verified separately.

🤖 Posted by Maximus bot (Claude Code) on behalf of @migmartri

Removes the RPC, its request and response messages, the service handler, the authorization policy entry, the use cases and the repository method, along with the predicates and the generated code that existed only to serve it. The project visibility assertions it carried are about DiscoverPrivate as much as about the endpoint, so they stay as a test of their own: a caller granted a project reaches it, a caller restricted to none does not, and a grant naming one organization does not reach a project living in another. What this branch still does is make the cost of referrer discovery independent of how many organizations the caller belongs to. Assisted-by: Claude Code Signed-off-by: Javier Rodriguez <javier@chainloop.dev> Chainloop-Trace-Sessions: 04123fef-eb07-444e-b017-b168c4fa1889 Signed-off-by: Javier Rodriguez <javier@chainloop.dev>

cubic-dev-ai

No issues found across 5 files

_{Re-trigger cubic}

migmartri · 2026-09-17T11:10:07Z

Security review — no RBAC or authorization regression

Reviewed the head (957265cf) against main, focusing on the authorization paths this PR rewrites. The final head touches only app/controlplane/pkg/data/referrer.go plus tests — the DiscoverEdges RPC added mid-PR (with the project-scoping issue cubic flagged: a project-scoped caller omitting project_name seeing cross-project edges) was reverted, so no new externally reachable surface ships. Each substitution is semantics-preserving:

1. `referrerVisibleToOrgs` (correlated EXISTS) vs `referrer.HasWorkflowsWith(...)` — equivalent

Same conditions: workflows.deleted_at IS NULL, org matched via IN. The old form joined the organizations table for ID equality; the new one matches the FK column workflows.organization_id — identical under the FK, and neither form filters soft-deleted orgs, so no widening.
Empty org list: ent v0.14.6 renders sql.In(col) with zero args as FALSE (dialect/sql/builder.go:1054) — deny, same as the generated form. Deny-by-default is preserved.
The strongest guard in the PR is here: referrer_visibility_integration_test.go runs the hand-written filter and the Ent-generated one against the same database fixtures across every state they can meet (no workflow, deleted-only, mixed-orgs, duplicate org IDs, nil orgs, traversal shape under Unique(false)) and requires identical ID sets, with expectations written out so a change in either form must be justified here.

2. `projectVisibilityPredicate` (constant two-branch form) vs the per-org disjunction — equivalent, and one real trap avoided

The per-org semantics map exactly: no map entry → org column match; entry with empty slice → contributes nothing; entry with grants → org == orgID AND project.id IN (grants). The (organization_id, id) IN ((org, proj)…) pair form is the same conjunction. A nil predicate still becomes sql.False() upstream — nothing visible — deny, unchanged.
The trap: the obvious "constant-shape" rewrite (two independent IN sets) would have been a genuine cross-org grant escalation — a project granted in one restricted org becoming reachable through another. The pair form prevents exactly that, and it is pinned three ways: TestProjectVisibilityPredicateDoesNotCrossOrgs (pair shape, argument pairing, absence of a bare "projects"."id" IN (), TestProjectVisibilityPredicateSkipsOrgsWithoutGrants, and the DB-backed biz test where a grant recorded under org2 for a project living in org1 is rejected.

3. `Unique(false)` (DISTINCT removal) — not an authz control, invariant holds

referrer_references has composite PK (referrer_id, referred_by_id) (ent/migrate/schema.go:915), so a fixed root matches each reference at most once, and every predicate on the query is a semi-join or scalar — join-free, no row multiplication. DISTINCT never filtered access, only multiplicity. The invariant is documented in-code with a warning that future predicates must stay join-free, and tested ("returned once, without DISTINCT"). Pagination cursor unaffected.

Injection surface

None added. projectGrantsPredicate binds UUIDs via the builder's Args (parameterized); project name, version, and digest go through ent predicates. The hand-written builder only writes identifiers and the literal IN keyword — no user input is interpolated.

Non-blocking observations

The reverted endpoint must not re-land as-was. If DiscoverEdges/EdgesAmong returns, the project-scoping rule must apply to every edge endpoint unconditionally, not only when the caller supplies project_name. Worth a note wherever the reverted work is tracked so those commits aren't cherry-picked back.
Pre-existing, unchanged here: isReferrerVisible hydrates via WithWorkflows() without a deleted_at filter, so its Go-side org/project sets can include soft-deleted workflows while the SQL filter excludes them. SQL is the stricter one, so no exposure — but it is the drift risk behind my earlier suggestion that a test assert the two implementations agree.
Chainloop SAST/secrets/OWASP checks pass; only controlplane data-layer files changed, so the vulnerability, IaC, and workflow scans were correctly out of scope.

🤖 Posted by Maximus bot (Claude Code) on behalf of @migmartri

javirln changed the title ~~perf(controlplane): make referrer discovery independent of the caller's organization count~~ feat(controlplane): add DiscoverEdges and cut the cost of referrer discovery Sep 16, 2026