[Feat] Community Local: SPA scaffold + SQLite persistence - #1741
Conversation
Nothing here is served on the web; Electron loads the bundle into a BrowserWindow, and the old name collided with surfsense_web, the actual web app. frontend/ matches the workstream that owns it and avoids ui/src/components/ui once shadcn is installed.
Generated by shadcn init --template vite --base radix --preset nova, so the layout matches upstream rather than a hand-rolled one. Radix and lucide match surfsense_web. Dropped the template's nested git repo and its readme. Only exception to the generated config: react-refresh/only-export-components is off for components/ui, where shadcn exports variants beside each component.
Proves the two halves talk before any feature lands. Dev goes through the vite proxy so the API needs no CORS middleware; the packaged app loads from file:// and will need its own answer at packaging time.
Health moves from api/health to modules/health. Routes, schemas, and soon models for one feature sit together, the way automations and notifications are laid out in the cloud backend, rather than being split across folders named after the kind of file. api/ keeps only what assembles the app.
Alembic is pinned below 1.19: autogenerate still reports a phantom remove_constraint for the CHECK constraints behind native_enum=False enums, which makes the drift test fail on a schema that matches.
The naming convention is set before the first migration, not later: SQLite is the one backend that lets constraints stay unnamed, and Alembic's batch mode cannot drop what it cannot name. pysqlite autocommits DDL, so a migration that dies between two create_table calls would leave a half-built database that fails to upgrade on every later launch. Clearing isolation_level and issuing BEGIN ourselves puts DDL back inside the transaction. expire_on_commit is off because the request dependency commits exactly when FastAPI is about to serialize the objects it just loaded.
Chunks keep their embedding so a re-index can rebuild the vector table without paying for embeddings twice, and are unique on position so a retried ingest cannot silently double a document's passages. The enum helper moves to shared/db.py now that chat needs it too. It still forces create_constraint, which SQLAlchemy has defaulted off since 1.4 and which is the only thing keeping these columns from being a bare VARCHAR that accepts anything.
ADR-0003: a deliverable's searchable body is an ordinary Document with document_type ARTIFACT, so this table owns no title, path, markdown or indexing state, and is unique on document_id. Dropped from the cloud shape: created_by_id, since Local has no auth, and storage_backend, since there is one backend and it is the user's disk. thread_id becomes chat_thread_id and clears to NULL rather than cascading, so deleting a conversation cannot delete the deck it wrote.
Migrations are written by hand and autogenerate is off. It cannot see a rename, so moving a column comes out as a drop plus an add that deletes the data silently, and the database it runs against is one user's laptop: unbacked, uninspectable, and impossible to roll forward from. Open WebUI writes all 58 of its SQLite revisions the same way. env.py carries no target_metadata, which makes --autogenerate fail outright rather than leaving the rule to discipline. It also no longer imports the app, so migrating does not drag routers onto the boot path.
A desktop install has no operator to run alembic after an update, so the app migrates itself on boot. The API is the only process allowed to emit DDL; the worker opens the same file to read and write rows. The engine is built in lifespan rather than at import so tests can point create_app() at a throwaway file, and disposed in a finally so a failed migration does not leak the pool.
Hand-written migrations are only safe with something that fails when a model changes and the revision does not. compare_metadata against a database built by migrations is that check; it catches a stray column before it reaches a release. conftest imports every slice's models, since a slice absent from Base.metadata is a slice the drift test silently cannot see. The rollback test fails without the BEGIN listener in shared/db.py.
These assert what the database enforces, not how it was built, so they belong with the slice that owns the table rather than in one test_schema file. Layout follows the cloud backend's tests/unit/<slice>, minus the extra level while there is only one kind of test. Four files share the name test_constraints.py, so pytest runs with --import-mode=importlib instead of scattering empty __init__.py files to satisfy the unique-basename rule.
Says to run alembic revision without --autogenerate, and that anything touching a table with rows in it reads the live schema first. Both are easy to get wrong from habit, since every other project reaches for autogenerate by default.
Locks hand-written migrations and the reasoning behind them, so the question does not get reopened per slice, and fills in the artifacts tables, which the data model named but never gave columns.
Local runs its own release train, so it starts at the bottom of one the way the cloud train did, rather than at the 0.1.0 the standalone packages picked. Nothing has shipped yet; the number should say so.
|
@CREDO23 is attempting to deploy a commit to the Rohan Verma's projects Team on Vercel. A member of the Team first needs to authorize it. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Team Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Follow-up to #1740, which merged the plans and the API skeleton. This adds
the frontend scaffold and the persistence layer underneath the backend.
Everything lands under
surfsense_local/; no cloud package is touched.Frontend
Vite + React + shadcn, generated with
shadcn init --template vite --base radix --preset novaso the layout matches upstream rather than ahand-rolled one. The folder is
frontend/, notweb/: Electron loads thebundle into a BrowserWindow, and the old name collided with
surfsense_web, the actual web app.The health view calls the API through the vite proxy, proving the two
halves talk before any feature lands.
Persistence
SQLAlchemy 2.0 models, one slice per feature, covering workspaces,
documents, chunks, chat threads and messages, and artifacts. Alembic owns
the schema;
create_allis never called, so the database a user gets isthe one migrations built.
Migrations are written by hand and autogenerate is off. It cannot see
a rename, so moving a column comes out as a drop plus an add that deletes
the data silently, and the database it runs against is one user's laptop:
unbacked, uninspectable, and impossible to roll forward from.
env.pycarries no
target_metadata, which makes--autogeneratefail outrightrather than leaving the rule to discipline. Open WebUI writes all 58 of
its SQLite revisions the same way.
Two SQLite-specific details that are hard to retrofit later:
backend that lets constraints stay unnamed, and Alembic's batch mode
cannot drop what it cannot name.
create_tablecalls would strand a half-built database that fails toupgrade on every later launch. Clearing
isolation_leveland issuingBEGINourselves puts DDL back inside the transaction. A test assertsnothing survives a failed upgrade.
The API migrates itself on boot, since a desktop install has no operator
to run
alembic upgrade.Tests
15 passing.
compare_metadataagainst a migration-built database failswhen a model changes and no revision follows, which is what makes
hand-written migrations safe. The rest assert what the database enforces
(cascades, partial unique index on
dedup_key, enum CHECK constraints)and live with the slice that owns the table.
Also
Local starts its own release train at
0.0.1with its own bump script; itdoes not follow the cloud train's shared version.
High-level PR Summary
This PR adds the foundational infrastructure for SurfSense Community Local by introducing a React + Vite frontend scaffold using shadcn/ui components and a SQLAlchemy 2.0 + Alembic persistence layer with hand-written SQLite migrations. The frontend displays a health status badge that communicates with the backend API through a Vite proxy. The persistence layer implements a vertical slice architecture with models for workspaces, documents, chunks, chat threads, and artifacts, enforced by a manually-written initial schema migration. The API now runs migrations on startup and manages sessions per request. All constraints (foreign keys, unique indexes, enums) are tested, and a metadata comparison test ensures models and migrations stay synchronized. The local version is set to
0.0.1with its own release cadence independent from the cloud version.⏱️ Estimated Review Time: 1-3 hours
💡 Review Order Suggestion
plans/community-local/00-umbrella-plan.mdplans/community-local/00b-diagrams.mdplans/community-local/00c-data-model.mdplans/community-local/frontend/01-shell.mdsurfsense_local/VERSIONsurfsense_local/README.mdsurfsense_local/backend/shared/config.pysurfsense_local/backend/shared/db.pysurfsense_local/backend/shared/migrations.pysurfsense_local/backend/alembic.inisurfsense_local/backend/alembic/env.pysurfsense_local/backend/alembic/script.py.makosurfsense_local/backend/alembic/versions/0001_initial_schema.pysurfsense_local/backend/modules/workspaces/models.pysurfsense_local/backend/modules/documents/models.pysurfsense_local/backend/modules/chunks/models.pysurfsense_local/backend/modules/chat/models.pysurfsense_local/backend/modules/artifacts/models.pysurfsense_local/backend/api/dependencies.pysurfsense_local/backend/api/main.pysurfsense_local/backend/tests/conftest.pysurfsense_local/backend/tests/test_migrations.pysurfsense_local/backend/tests/documents/test_constraints.pysurfsense_local/backend/tests/chunks/test_constraints.pysurfsense_local/backend/tests/chat/test_constraints.pysurfsense_local/backend/tests/artifacts/test_constraints.pysurfsense_local/backend/pyproject.tomlsurfsense_local/frontend/package.jsonsurfsense_local/frontend/vite.config.tssurfsense_local/frontend/tsconfig.jsonsurfsense_local/frontend/components.jsonsurfsense_local/frontend/src/index.csssurfsense_local/frontend/src/main.tsxsurfsense_local/frontend/src/lib/api.tssurfsense_local/frontend/src/App.tsxsurfsense_local/frontend/src/components/theme-provider.tsxsurfsense_local/frontend/src/components/ui/badge.tsxsurfsense_local/frontend/src/components/ui/button.tsxsurfsense_local/frontend/src/components/ui/skeleton.tsxsurfsense_local/scripts/bump-version.sh