Skip to content

Migrate warehouse queries to effect-orm - #1258

Merged
Makisuo merged 4 commits into
mainfrom
claude/maple-orm-clickhouse-migration-16dcd5
Oct 5, 2026
Merged

Makisuo merged 4 commits into
mainfrom
claude/maple-orm-clickhouse-migration-16dcd5

Conversation

@Makisuo

@Makisuo Makisuo commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

Moves Maple's ClickHouse query layer from @maple-dev/effect-clickhouse to @maple-dev/effect-orm 0.2.0.

What changed

  • Imports: the old root and /types imports are now @maple-dev/effect-orm/clickhouse. /expr, /sql and /benchmark keep their subpaths.
  • Tables: declared with CH.table(name, { external: true, columns, tenantColumn: "OrgId" }). The DDL still lives in datasources.ts, so effect-orm's migrations don't manage them.
  • Branded tenant id: the OrgId column is branded, and effect-orm compares branded columns strictly.
    • Every tenant filter uses the shared orgIdParam from @maple/query-engine/ch/tables.
    • Warehouse query signatures and the observability executor take orgId: OrgId.
    • Local mode shares one LOCAL_ORG_ID from @maple/query-engine/local.
  • Typed compile params: compile now checks params against the query's type. Queries whose variant is chosen at runtime compile per branch, so each pairs with its own params:
    • trace time probe, split into traceTimeProbeQuery / recentTraceTimeProbeQuery
    • session activity
    • AI session summary
    • PlanetScale branch stats
    • AI tools / sessions window bounds
  • Conditional filters: optional filters written as cond ? $.X.eq(param…) : undefined now use CH.whenTrue, so the builder knows the param is conditional.

The emitted SQL is unchanged: the SQL catalog baseline test passes.

Verification

  • bun typecheck: 49/49 tasks plus tsconfig.alchemy.json.
  • Vitest:
    • query-engine: 1621 pass
    • query-engine-integrations: 446 pass
    • backend ai-sessions, warehouse, audit and alerts dirs: 567 pass
    • api routes: 376 pass
    • local-ui: 52 pass
  • bun test cli server: 19 pass.
  • Live ClickHouse: bun run ch:test passes (424 catalog queries execute and decode, plus 88 service tests). All 9 backend *.clickhouse.e2e files pass (182 tests).
  • Local mode end to end: maple start from this branch, fed by the examples/effect-todo services. All 20 CLI query commands work. local-ui loads every page with data and no console errors: traces, trace and span detail, logs, metrics, services, service map, errors.

🤖 Generated with Claude Code

Imports move from @maple-dev/effect-clickhouse to @maple-dev/effect-orm
(/clickhouse replaces the root and /types entries). Warehouse tables are
external `CH.table` definitions, since their DDL stays in datasources.ts.

OrgId columns are branded, so every tenant filter compares against the
shared `orgIdParam` and warehouse query signatures take `OrgId`. Queries
whose variant is chosen at runtime compile per branch, so each pairs
with the params it actually needs.

The dependency is a local `link:` until effect-orm is published.
@coderabbitai

coderabbitai Bot commented Oct 5, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Too many files!

This PR contains 154 files, which is 4 over the limit of 150.

To get a review, reduce the PR to 150 files or fewer by splitting it into smaller PRs or changing its base branch.

Upgrade to a paid plan to raise the limit.

This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: e65cfa7f-fccf-4f0a-bccb-e772a19751ee
📥 Commits

Reviewing files that changed from the base of the PR and between a9d45c7 and 898c193.

⛔ Files ignored due to path filters (1)
  • bun.lock is excluded by !**/*.lock
📒 Files selected for processing (154)
  • CLAUDE.md
  • apps/api/package.json
  • apps/api/scripts/bench-queries.ts
  • apps/api/scripts/bench-suites/services.ts
  • apps/api/scripts/query-bench/catalog.clickhouse.e2e.test.ts
  • apps/api/src/routes/queries.ts
  • apps/cli/src/server/service-map-rollup.ts
  • apps/local-ui/src/lib/constants.ts
  • apps/local-ui/src/lib/query.ts
  • docs/query-benchmarking.md
  • packages/backend/package.json
  • packages/backend/src/services/ai-sessions/ai-session-reads.ts
  • packages/backend/src/services/auth/tinybird-jwt.ts
  • packages/backend/src/services/warehouse/WarehouseQueryService.clickhouse.e2e.test.ts
  • packages/backend/src/services/warehouse/WarehouseQueryService.test.ts
  • packages/backend/src/services/warehouse/ai-tools.clickhouse.e2e.test.ts
  • packages/backend/src/services/warehouse/ai-trace-index-materialization.clickhouse.e2e.test.ts
  • packages/backend/src/services/warehouse/series-cap.clickhouse.e2e.test.ts
  • packages/domain/package.json
  • packages/domain/src/raw-sql.ts
  • packages/domain/src/tinybird/ai-crawler-columns.ts
  • packages/domain/src/tinybird/gen-ai-columns.ts
  • packages/domain/src/tinybird/semconv-renames.test.ts
  • packages/domain/src/tinybird/semconv-renames.ts
  • packages/domain/src/tinybird/span-display-name.ts
  • packages/query-engine-integrations/package.json
  • packages/query-engine-integrations/src/ai/ai-sessions.test.ts
  • packages/query-engine-integrations/src/ai/ai-sessions.ts
  • packages/query-engine-integrations/src/ai/ai-span-columns.test.ts
  • packages/query-engine-integrations/src/ai/ai-span-columns.ts
  • packages/query-engine-integrations/src/ai/ai-tools.test.ts
  • packages/query-engine-integrations/src/ai/ai-tools.ts
  • packages/query-engine-integrations/src/benchmark/catalog.test.ts
  • packages/query-engine-integrations/src/benchmark/index.ts
  • packages/query-engine-integrations/src/cloudflare/cloudflare-infra-breakdowns.test.ts
  • packages/query-engine-integrations/src/cloudflare/cloudflare-infra-breakdowns.ts
  • packages/query-engine-integrations/src/cloudflare/cloudflare-infra-extended.test.ts
  • packages/query-engine-integrations/src/cloudflare/cloudflare-infra-extended.ts
  • packages/query-engine-integrations/src/cloudflare/cloudflare-infra-filters.test.ts
  • packages/query-engine-integrations/src/cloudflare/cloudflare-infra-filters.ts
  • packages/query-engine-integrations/src/cloudflare/cloudflare-infra.test.ts
  • packages/query-engine-integrations/src/cloudflare/cloudflare-infra.ts
  • packages/query-engine-integrations/src/cloudflare/cloudflare-map.test.ts
  • packages/query-engine-integrations/src/cloudflare/cloudflare-map.ts
  • packages/query-engine-integrations/src/cloudflare/cloudflare-usage.test.ts
  • packages/query-engine-integrations/src/cloudflare/cloudflare-usage.ts
  • packages/query-engine-integrations/src/planetscale/planetscale-infra.test.ts
  • packages/query-engine-integrations/src/planetscale/planetscale-infra.ts
  • packages/query-engine-integrations/src/planetscale/planetscale-map.test.ts
  • packages/query-engine-integrations/src/planetscale/planetscale-map.ts
  • packages/query-engine-integrations/src/planetscale/planetscale-storage.test.ts
  • packages/query-engine-integrations/src/product/billing-usage.test.ts
  • packages/query-engine-integrations/src/product/billing-usage.ts
  • packages/query-engine-integrations/src/product/internal.ts
  • packages/query-engine-integrations/src/product/setup-audit.test.ts
  • packages/query-engine-integrations/src/product/setup-audit.ts
  • packages/query-engine-integrations/src/railway/railway-infra.ts
  • packages/query-engine/package.json
  • packages/query-engine/src/benchmark/builders.ts
  • packages/query-engine/src/benchmark/catalog.test.ts
  • packages/query-engine/src/benchmark/catalog.ts
  • packages/query-engine/src/benchmark/index.ts
  • packages/query-engine/src/ch/ch.test.ts
  • packages/query-engine/src/ch/endpoint-types.test-d.ts
  • packages/query-engine/src/ch/index.ts
  • packages/query-engine/src/ch/pipe-dispatch.ts
  • packages/query-engine/src/ch/queries/activity.test.ts
  • packages/query-engine/src/ch/queries/activity.ts
  • packages/query-engine/src/ch/queries/alert-checks.test.ts
  • packages/query-engine/src/ch/queries/alert-checks.ts
  • packages/query-engine/src/ch/queries/anomaly.test.ts
  • packages/query-engine/src/ch/queries/anomaly.ts
  • packages/query-engine/src/ch/queries/attribute-keys.test.ts
  • packages/query-engine/src/ch/queries/attribute-keys.ts
  • packages/query-engine/src/ch/queries/audit-log.ts
  • packages/query-engine/src/ch/queries/containers.test.ts
  • packages/query-engine/src/ch/queries/containers.ts
  • packages/query-engine/src/ch/queries/count-semantics.test.ts
  • packages/query-engine/src/ch/queries/errors.test.ts
  • packages/query-engine/src/ch/queries/errors.ts
  • packages/query-engine/src/ch/queries/format.ts
  • packages/query-engine/src/ch/queries/infra.test.ts
  • packages/query-engine/src/ch/queries/infra.ts
  • packages/query-engine/src/ch/queries/liveness.test.ts
  • packages/query-engine/src/ch/queries/liveness.ts
  • packages/query-engine/src/ch/queries/logs.test.ts
  • packages/query-engine/src/ch/queries/logs.ts
  • packages/query-engine/src/ch/queries/metrics.test.ts
  • packages/query-engine/src/ch/queries/metrics.ts
  • packages/query-engine/src/ch/queries/product-events-explore.test.ts
  • packages/query-engine/src/ch/queries/product-events-explore.ts
  • packages/query-engine/src/ch/queries/product-events-paths.test.ts
  • packages/query-engine/src/ch/queries/product-events-paths.ts
  • packages/query-engine/src/ch/queries/product-events.test.ts
  • packages/query-engine/src/ch/queries/product-events.ts
  • packages/query-engine/src/ch/queries/query-helpers.ts
  • packages/query-engine/src/ch/queries/releases.test.ts
  • packages/query-engine/src/ch/queries/releases.ts
  • packages/query-engine/src/ch/queries/rollup-splice.test.ts
  • packages/query-engine/src/ch/queries/rollup-splice.ts
  • packages/query-engine/src/ch/queries/series-cap.test.ts
  • packages/query-engine/src/ch/queries/series-cap.ts
  • packages/query-engine/src/ch/queries/service-endpoints.test.ts
  • packages/query-engine/src/ch/queries/service-infra.test.ts
  • packages/query-engine/src/ch/queries/service-infra.ts
  • packages/query-engine/src/ch/queries/service-map-rollup.test.ts
  • packages/query-engine/src/ch/queries/service-map-rollup.ts
  • packages/query-engine/src/ch/queries/service-map.test.ts
  • packages/query-engine/src/ch/queries/service-map.ts
  • packages/query-engine/src/ch/queries/service-operations.test.ts
  • packages/query-engine/src/ch/queries/service-operations.ts
  • packages/query-engine/src/ch/queries/services.test.ts
  • packages/query-engine/src/ch/queries/services.ts
  • packages/query-engine/src/ch/queries/session-events.test.ts
  • packages/query-engine/src/ch/queries/session-events.ts
  • packages/query-engine/src/ch/queries/session-replays.test.ts
  • packages/query-engine/src/ch/queries/session-replays.ts
  • packages/query-engine/src/ch/queries/session-tags.ts
  • packages/query-engine/src/ch/queries/signal-presence.test.ts
  • packages/query-engine/src/ch/queries/signal-presence.ts
  • packages/query-engine/src/ch/queries/top-operations.ts
  • packages/query-engine/src/ch/queries/traces.test.ts
  • packages/query-engine/src/ch/queries/traces.ts
  • packages/query-engine/src/ch/queries/web-analytics-ai.test.ts
  • packages/query-engine/src/ch/queries/web-analytics-ai.ts
  • packages/query-engine/src/ch/queries/web-analytics.ts
  • packages/query-engine/src/ch/raw-sql.ts
  • packages/query-engine/src/ch/tables.ts
  • packages/query-engine/src/ch/user-agent.ts
  • packages/query-engine/src/dashboard-variables/interpolate.ts
  • packages/query-engine/src/execution/executor.test.ts
  • packages/query-engine/src/execution/executor.ts
  • packages/query-engine/src/execution/fingerprint.ts
  • packages/query-engine/src/execution/ports.ts
  • packages/query-engine/src/local.ts
  • packages/query-engine/src/observability/WarehouseExecutor.ts
  • packages/query-engine/src/observability/error-detail.test.ts
  • packages/query-engine/src/observability/explore-attributes.test.ts
  • packages/query-engine/src/observability/find-slow-traces.test.ts
  • packages/query-engine/src/observability/inspect-trace.test.ts
  • packages/query-engine/src/observability/inspect-trace.ts
  • packages/query-engine/src/observability/search-logs.test.ts
  • packages/query-engine/src/observability/session-replays.test.ts
  • packages/query-engine/src/observability/span-detail.test.ts
  • packages/query-engine/src/profiles/query-profile.ts
  • packages/query-engine/src/registry/logs.ts
  • packages/query-engine/src/registry/product-events.ts
  • packages/query-engine/src/registry/queries.ts
  • packages/query-engine/src/registry/query-definition.ts
  • packages/query-engine/src/runtime/query-definition-runner.ts
  • packages/query-engine/src/runtime/query-engine.ts
  • packages/query-engine/src/runtime/raw-sql.test.ts
  • packages/query-engine/src/runtime/raw-sql.ts
  • packages/query-engine/src/traces-shared.ts

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

…khouse-migration-16dcd5

# Conflicts:
#	apps/api/src/routes/queries.ts
#	apps/clickhouse-builder-docs/package.json
#	apps/clickhouse-builder-docs/scripts/sync-content.ts
#	bun.lock
#	packages/query-engine/src/ch/queries/infra.ts
#	packages/query-engine/src/registry/queries.ts
@Makisuo
Makisuo marked this pull request as ready for review October 5, 2026 20:59
@maple-review-bot

maple-review-bot Bot commented Oct 5, 2026 •

Copy link
Copy Markdown

Maple review

🟡 Confidence 3/5 · needs attention
quality 100/100 · no findings · tests partial · risk medium

Warning

This review ended early; what follows is what it established.

Mechanical migration of the ClickHouse query layer to @maple-dev/effect-orm, plus a branded OrgId threaded through every tenant filter as the shared orgIdParam. The hunks I read preserve emitted SQL and tenant scoping; I found no defect I can stand behind. Files I could not read are named below.

  • Builder imports move to @maple-dev/effect-orm/{clickhouse,expr,sql}
  • OrgId is branded; every tenant filter becomes orgIdParam
  • compile params are typed per query, so runtime variants compile per branch
  • traceTimeProbeQuery splits into bounded recentTraceTimeProbeQuery; LOCAL_ORG_ID moves to @maple/query-engine/local
What was checked
  • Every $.OrgId.eq(param.string("orgId")) in errors.ts, ai-tools.ts, ai-sessions.ts is now orgIdParam; none dropped
  • traceFacetTiers inlined at errors.ts:703,828,850 keeps the same canUseTraceFacetsRollup union choice
  • planetscaleStatsParams no longer spreads database; the three branch arms pass it explicitly (apps/api/src/routes/queries.ts:346,358,375)
Files not reviewed (78)

The review ended before it read these diffs, so nothing above vouches for them.

  • apps/api/package.json
  • apps/api/scripts/query-bench/catalog.clickhouse.e2e.test.ts
  • apps/cli/src/server/service-map-rollup.ts
  • packages/backend/package.json
  • packages/backend/src/services/auth/tinybird-jwt.ts
  • packages/backend/src/services/warehouse/WarehouseQueryService.clickhouse.e2e.test.ts
  • packages/backend/src/services/warehouse/WarehouseQueryService.test.ts
  • packages/backend/src/services/warehouse/ai-tools.clickhouse.e2e.test.ts
  • packages/backend/src/services/warehouse/ai-trace-index-materialization.clickhouse.e2e.test.ts
  • packages/backend/src/services/warehouse/series-cap.clickhouse.e2e.test.ts
  • packages/domain/package.json
  • packages/domain/src/raw-sql.ts
  • packages/domain/src/tinybird/ai-crawler-columns.ts
  • packages/domain/src/tinybird/gen-ai-columns.ts
  • packages/domain/src/tinybird/semconv-renames.test.ts
  • packages/domain/src/tinybird/semconv-renames.ts
  • packages/domain/src/tinybird/span-display-name.ts
  • packages/query-engine-integrations/package.json
  • packages/query-engine-integrations/src/ai/ai-sessions.test.ts
  • packages/query-engine-integrations/src/ai/ai-span-columns.test.ts
  • packages/query-engine-integrations/src/ai/ai-tools.test.ts
  • packages/query-engine-integrations/src/benchmark/catalog.test.ts
  • packages/query-engine-integrations/src/benchmark/index.ts
  • packages/query-engine-integrations/src/cloudflare/cloudflare-infra-breakdowns.test.ts
  • packages/query-engine-integrations/src/cloudflare/cloudflare-infra-extended.test.ts
  • packages/query-engine-integrations/src/cloudflare/cloudflare-infra-filters.test.ts
  • packages/query-engine-integrations/src/cloudflare/cloudflare-infra.test.ts
  • packages/query-engine-integrations/src/cloudflare/cloudflare-map.test.ts
  • packages/query-engine-integrations/src/cloudflare/cloudflare-usage.test.ts
  • packages/query-engine-integrations/src/planetscale/planetscale-infra.test.ts
  • and 48 more

898c193 · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Devin Review: 2 flags

Not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

Devin Review

@Makisuo
Makisuo merged commit 99b88bd into main Oct 5, 2026
23 of 25 checks passed
@Makisuo
Makisuo deleted the claude/maple-orm-clickhouse-migration-16dcd5 branch October 5, 2026 21:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant