Skip to content

epic: SQE backend — serve SQLite databases via SQE alongside other backends #629

Description

@iheitlager

Use Case

sqlite-rs as a database backend that SQE serves — SQLite files as data sources, combinable with other backends (Postgres, DuckDB, etc.).

Context

SQE federates multiple database backends. Adding SQLite as a first-class backend means:

  • Query SQLite files alongside other data sources
  • Use SQLite for local/edge data that joins with remote sources
  • No C dependency for SQLite backend support

Requirements

Read path

  • Table discovery from SQLite schema
  • Type mapping (SQLite affinity → Arrow)
  • Pushdown (filters, projections, limits)
  • Scan to Arrow batches

Write path (optional)

  • INSERT/UPDATE/DELETE via SQE
  • Transaction semantics

Federation

  • Cross-backend JOINs (SQLite ⋈ Postgres)
  • Unified catalog view

Dependencies

Path

  1. Read-only backend with table discovery + scans
  2. Pushdown optimization
  3. Write support
  4. Federation testing

Success

  • sqe query "SELECT * FROM sqlite.my_table JOIN pg.other ..."
  • Performance within 2× of direct SQLite access
  • Works with existing SQE CLI and connectors

Metadata

Metadata

Assignees

No one assigned

    Labels

    epicTracking issue spanning multiple tickets

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions