Phase 5 retrieval

ai-service/.env.example

@@ -36,3 +36,24 @@ LOG_LEVEL=INFO
 
 # Serve interactive API docs at /docs. Keep off in production.
 AI_SERVICE_DOCS=false
+
+# ─── Retrieval: similar issues and suggested resolutions ─────────────────────
+# Connection as the restricted ai_service database role. Leave empty to turn
+# retrieval off (triage and sentiment still work). Use 127.0.0.1, not
+# localhost: the PostgreSQL client tries IPv6 first, which can stall when
+# Docker only listens on IPv4.
+#
+# One-time setup (as a database administrator), then apply migrations:
+#   uv run ai-db bootstrap --admin-url postgresql://ADMIN:PASSWORD@127.0.0.1:5432/newnopdesk
+#   uv run ai-db migrate
+AI_DATABASE_URL=postgresql://ai_service:ai_service_local@127.0.0.1:5432/newnopdesk
+
+# fastembed runs a local ONNX model (no API key). "fake" is for tests only.
+EMBEDDING_PROVIDER=fastembed
+EMBEDDING_MODEL=BAAI/bge-small-en-v1.5
+# Where to keep the downloaded model (~67 MB). Empty = fastembed's temp folder.
+# EMBEDDING_CACHE_DIR=
+
+# Cosine similarity a past issue needs to count as similar, and how many to use
+RETRIEVAL_MIN_SIMILARITY=0.70
+RETRIEVAL_MAX_RESULTS=5

ai-service/README.md

@@ -2,7 +2,7 @@
 
 Python service that adds AI-assisted triage and sentiment analysis to NewnopDesk. The Node backend calls it over internal HTTP; browsers never reach it.
 
-**Stack:** Python 3.12 · FastAPI · Pydantic · Google Gemini (`google-genai` SDK) · uv · pytest · ruff · mypy (strict)
+**Stack:** Python 3.12 · FastAPI · Pydantic · Google Gemini (`google-genai` SDK) · PostgreSQL + pgvector · fastembed · Alembic · uv · pytest · ruff · mypy (strict)
 
 ---
 
@@ -12,7 +12,12 @@ Python service that adds AI-assisted triage and sentiment analysis to NewnopDesk
 |----------|---------|----------------------|
 | `POST /v1/analyze` | A newly created issue | Triage suggestions (impact, urgency, category, owning team, each with a reason) **and** a sentiment assessment |
 | `POST /v1/sentiment` | One client comment on an existing issue | A sentiment assessment, so mood can be tracked across a thread |
-| `GET /healthz` | Liveness check | — |
+| `PUT /v1/documents/{issue_id}` | A resolved issue to index | — (embeds the problem text; re-embeds only when it changed) |
+| `DELETE /v1/documents/{issue_id}?company_id=` | Remove an issue from the index (e.g. reopened) | — |
+| `GET /v1/documents` | Which issues are indexed, for reconciliation | — |
+| `POST /v1/similar` | An issue being viewed | — (vector search: same company, optionally the viewer's products) |
+| `POST /v1/suggest-resolution` | An issue being viewed | A suggested fix grounded in similar past issues, citing their ticket numbers |
+| `GET /healthz` | Liveness check, and whether retrieval is enabled | — |
 
 Triage and sentiment share one call per issue to keep cost down. Every result is a **suggestion**: the backend stores it and an engineer or admin accepts, edits or rejects it. This service never changes an issue.
 
@@ -26,9 +31,18 @@ Sentiment is a separate signal for account managers. The prompt tells the model
 cd ai-service
 uv sync                         # creates .venv with Python 3.12 and all dependencies
 cp .env.example .env            # set AI_SERVICE_TOKEN; add GEMINI_API_KEY, or use AI_PROVIDER=fake
-uv run uvicorn ai_service.main:create_app --factory --host 127.0.0.1 --port 8000
+
+# Retrieval only: one-time database setup, against the Postgres from docker-compose.yml
+uv run ai-db bootstrap --admin-url postgresql://newnopdesk:newnopdesk_local@127.0.0.1:5432/newnopdesk
+uv run ai-db migrate
+
+uv run ai-serve                 # http://127.0.0.1:8000
 ```
 
+`ai-serve` starts uvicorn on a selector-based event loop. psycopg's async mode can't run on the `ProactorEventLoop` that uvicorn picks on Windows by default; on Linux the selector loop is standard, so behaviour is identical everywhere.
+
+If `AI_DATABASE_URL` is unset, or the database or embedding model can't be loaded at startup, retrieval is disabled (its endpoints return `503 RETRIEVAL_UNAVAILABLE`) and triage and sentiment keep working. `/healthz` reports which.
+
 With `AI_PROVIDER=fake` the service runs without an API key. The fake provider uses keyword rules: its output is valid but not accurate, which is enough to develop and demo the rest of the system.
 
 ```bash
@@ -89,6 +103,8 @@ All `/v1` endpoints need `Authorization: Bearer <AI_SERVICE_TOKEN>`. An `X-Reque
 | `LLM_UNAVAILABLE` | 503 | yes | Provider error or unreachable |
 | `LLM_BLOCKED` | 422 | no | Provider safety filters refused |
 | `LLM_REQUEST_REJECTED` | 502 | no | Provider rejected the request (bad key, unknown model) |
+| `RETRIEVAL_UNAVAILABLE` | 503 | no | Retrieval isn't configured, or failed to start |
+| `RETRIEVAL_DATABASE_UNAVAILABLE` | 503 | yes | The vector database is unreachable |
 | `INTERNAL_ERROR` | 500 | yes | Unexpected error; possibly transient, so the caller may retry within its attempt limit |
 
 The service makes **exactly one attempt** per request. Retrying is the Node backend's decision: its job queue owns the retry budget, so retries never stack up across two layers.
@@ -104,8 +120,15 @@ src/ai_service/
 ├── schemas.py           # API request/response and strict LLM output models
 ├── taxonomy.py          # categories and teams, with the descriptions shown to the model
 ├── safety.py            # cleaning, delimiting and quote checks for untrusted text
-├── prompts/             # versioned prompts (analyze-v1, sentiment-v1)
-├── services/analysis.py # one call → validate → check quote → log
+├── embeddings.py        # Embedder interface: fastembed (local ONNX) and a fake for tests
+├── vector_store.py      # VectorStore interface: PostgreSQL + pgvector, and in-memory
+├── serve.py             # `ai-serve`: uvicorn on a selector event loop
+├── db/                  # `ai-db`: bootstrap (role + schema) and migrations
+├── prompts/             # versioned prompts (analyze-v1, sentiment-v1, resolution-v1)
+├── services/
+│   ├── llm_call.py      # one call → validate → feature check → log, shared by every feature
+│   ├── analysis.py      # triage + sentiment, comment sentiment
+│   └── retrieval.py     # indexing, similar issues, suggested resolutions
 ├── llm/
 │   ├── base.py          # LLMProvider interface and error types
 │   ├── gemini.py        # Gemini implementation
@@ -115,6 +138,8 @@ src/ai_service/
 └── api/                 # routes, auth, error handlers, request-ID middleware
 ```
 
+Migrations for the `ai` schema live in `migrations/` (Alembic, plain SQL).
+
 **Swapping providers:** the rest of the service depends only on `llm/base.py`. A new provider is one class implementing `generate_json()` that maps its failures to the `LLMError` types, plus one line in `factory.py`.
 
 ---
@@ -134,6 +159,17 @@ Client-written text is treated as untrusted at every step.
 
 Regex blocklists of "bad phrases" are deliberately not used: they are easy to evade and give a false sense of security. The defences above don't depend on spotting the attack.
 
+### Retrieval and tenant isolation
+
+Similar-issue search must never cross a tenant boundary, so isolation is enforced at four independent layers:
+
+1. **The database role.** The service connects as `ai_service`, which owns the `ai` schema (its tables and the pgvector extension) and has **no privileges** on the application's `public` schema. PostgreSQL itself refuses to let it read `users`, `issues` or anything else; a test proves it.
+2. **The SQL.** Every search, read and delete filters on `company_id` in the query. The `VectorStore` interface has no unscoped method, `company_id` is a required keyword argument, and invalid values (`0`, negative, `True`, strings) raise.
+3. **The viewer's products.** The backend passes the products the viewer can open, so an engineer is never shown (or given a fix based on) an issue from a product outside their access, even within the same company.
+4. **The backend re-checks results.** Every returned issue is checked again against the viewer's normal tenancy rules before it reaches a browser, which also drops issues reopened or deleted since they were indexed.
+
+Suggested resolutions are grounded and checked: the model sees only the retrieved past issues (wrapped in `<past_issue-{boundary}>` delimiters with the same neutralisation as client text), must cite at least one of them when it claims relevant history, and **may only cite tickets that were actually retrieved**. A response citing anything else, including a real ticket from another company, is rejected. When no similar issue clears the threshold, no LLM call is made at all.
+
 **Gemini safety filters** are set to block only high-probability harm. Client complaints are often angry or rude, and the default thresholds could block a legitimate complaint we need to classify. The output is constrained JSON, so this doesn't let the model write harmful free text.
 
 **Data:** on Gemini's free tier, Google may use prompts and responses to improve its products. That's acceptable for seeded demo data; a deployment handling real client tickets should use a paid tier.
@@ -143,11 +179,12 @@ Regex blocklists of "bad phrases" are deliberately not used: they are easy to ev
 ## Testing
 
 ```bash
-uv run pytest             # 158 tests, no API key or network needed
+uv run pytest             # 233 tests; database tests start a throwaway pgvector container (Docker)
 uv run ruff check .       # lint (includes security rules)
 uv run ruff format .      # format
 uv run mypy               # strict type checking of src/ and tests/
 uv run pytest -m live -v  # optional: 3 smoke tests against the real Gemini API (needs GEMINI_API_KEY)
+uv run pytest -m embedding  # optional: the real embedding model (downloads ~67 MB once)
 ```
 
 Tests use a **scripted provider** that returns whatever output a test specifies, valid or malicious, and records every prompt it receives.
@@ -161,8 +198,26 @@ Tests use a **scripted provider** that returns whatever output a test specifies,
 | `test_gemini_provider.py` | Request building (schema, safety settings, thinking level) and mapping of Gemini responses and errors, against a stubbed SDK |
 | `test_safety.py` | Input cleaning and quote matching |
 | `test_schema_and_prompts.py` | Taxonomy, schemas, provider schemas and prompts agree with each other |
+| `test_vector_store.py` | One contract run against **both** stores (in-memory and PostgreSQL): ranking, company isolation, product narrowing, scoped delete, invalid company IDs |
+| `test_database_security.py` | Against real PostgreSQL: `ai_service` can't read, write or create anything in `public`; the extension lives in `ai`; `company_id > 0` is enforced by a `CHECK` |
+| `test_retrieval_api.py` | Indexing and re-embedding, disjoint results per company, product narrowing, citations limited to retrieved tickets, injected past-issue text staying in its block, retrieval disabled gracefully |
+| `test_retrieval_postgres.py` | Full stack: HTTP → service → PostgreSQL + pgvector, isolated per company |
 | `test_live_gemini.py` | Opt-in smoke tests against the real API |
 
-Each safety defence was checked by disabling it and confirming tests fail (between 2 and 8 tests per defence).
+Each safety and isolation defence was checked by disabling it and confirming tests fail.
+
+---
+
+## Design decisions
+
+**Embed the problem, not the fix.** The vector is built from the issue's title and description, so a new issue matches past ones by symptom. The staff resolution notes are stored next to it and given to the model when suggesting a fix, but not embedded: mixing the fix into the vector would blur matching by symptom.
+
+**pgvector in the same PostgreSQL as the application,** in its own schema and role, rather than a separate vector database: no extra server, backups or sync on a 1 GB instance. Prisma only manages `public` and ignores the `ai` schema entirely (verified: its schema diff stays empty).
+
+**Exact search at this scale.** With 300 documents across 3 companies, `EXPLAIN ANALYZE` shows PostgreSQL filtering to the company's rows and taking the top 5 with an exact heap sort in **0.18 ms**, without using the HNSW index. That's faster than the index at this size and exact, so recall measures the embedding model, not index error. The HNSW index is there for when the corpus grows.
+
+**Local embeddings (fastembed, `bge-small-en-v1.5`).** No API key, no per-call cost, and fully reproducible evaluations. 5 ms per embedding, about 200 MB of memory. On real issue text it scored the same problem at 0.86–0.91 and unrelated issues around 0.56–0.63, which is where the 0.70 threshold comes from.
+
+**Short connect timeouts and `127.0.0.1`.** The PostgreSQL client library tries IPv6 first when given `localhost`; with Docker listening only on IPv4, a connection with no timeout waited indefinitely. Every connection now has a 10-second timeout, and local URLs use `127.0.0.1`.
 
 What a real model *does* with malicious input is measured by the evaluation suite (later phase), not asserted in unit tests.

ai-service/alembic.ini

@@ -0,0 +1,35 @@
+# Alembic configuration for the AI service's `ai` schema.
+# Run migrations with `uv run ai-db migrate` rather than calling alembic
+# directly: the command supplies the database URL from AI_DATABASE_URL.
+
+[alembic]
+script_location = migrations
+# The URL is set at runtime by ai_service.db.cli; never put credentials here.
+sqlalchemy.url =
+
+[loggers]
+keys = root,alembic
+
+[handlers]
+keys = console
+
+[formatters]
+keys = generic
+
+[logger_root]
+level = WARNING
+handlers = console
+
+[logger_alembic]
+level = INFO
+handlers =
+qualname = alembic
+
+[handler_console]
+class = StreamHandler
+args = (sys.stderr,)
+level = NOTSET
+formatter = generic
+
+[formatter_generic]
+format = %(levelname)-5.5s [%(name)s] %(message)s

ai-service/migrations/env.py

@@ -0,0 +1,26 @@
+"""Alembic environment. Migrations are plain SQL; no ORM models are involved."""
+
+from alembic import context
+from sqlalchemy import create_engine, pool
+
+config = context.config
+
+
+def run_migrations_online() -> None:
+    url = config.get_main_option("sqlalchemy.url")
+    if not url:
+        raise RuntimeError("No database URL. Run migrations with `uv run ai-db migrate`.")
+    engine = create_engine(url, poolclass=pool.NullPool, connect_args={"connect_timeout": 10})
+    with engine.connect() as connection:
+        context.configure(
+            connection=connection,
+            # Keep Alembic's bookkeeping table inside the ai schema too
+            version_table_schema="ai",
+        )
+        with context.begin_transaction():
+            context.run_migrations()
+
+
+if context.is_offline_mode():
+    raise RuntimeError("Offline (SQL script) mode is not supported")
+run_migrations_online()

ai-service/migrations/script.py.mako

@@ -0,0 +1,21 @@
+"""${message}
+
+Revision ID: ${up_revision}
+Revises: ${down_revision | comma,n}
+Create Date: ${create_date}
+"""
+
+from alembic import op
+
+revision = ${repr(up_revision)}
+down_revision = ${repr(down_revision)}
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    ${upgrades if upgrades else "pass"}
+
+
+def downgrade() -> None:
+    ${downgrades if downgrades else "pass"}

ai-service/migrations/versions/0001_issue_documents.py

@@ -0,0 +1,54 @@
+"""Resolved issues and their embeddings, for similarity search.
+
+Revision ID: 0001
+Revises:
+Create Date: 2026-10-01
+"""
+
+from alembic import op
+
+revision = "0001"
+down_revision = None
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    op.execute(
+        """
+        CREATE TABLE ai.issue_documents (
+            issue_id        BIGINT PRIMARY KEY,
+            company_id      BIGINT NOT NULL,
+            -- Engineers can only open issues on products they have access
+            -- to, so searches can be narrowed to the viewer's products
+            product_id      BIGINT NOT NULL,
+            ticket_number   VARCHAR(16) NOT NULL,
+            title           VARCHAR(200) NOT NULL,
+            -- The client's description: what gets embedded, with the title
+            problem         TEXT NOT NULL,
+            -- Staff comments explaining the fix: stored for suggestions, not embedded
+            resolution      TEXT NOT NULL,
+            resolved_at     TIMESTAMPTZ,
+            embedding       vector(384) NOT NULL,
+            embedding_model VARCHAR(64) NOT NULL,
+            content_hash    VARCHAR(64) NOT NULL,
+            indexed_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
+            CONSTRAINT issue_documents_company_positive CHECK (company_id > 0)
+        )
+        """
+    )
+    # Every search filters on company_id first
+    op.execute(
+        "CREATE INDEX idx_issue_documents_company ON ai.issue_documents (company_id, product_id)"
+    )
+    # Approximate nearest-neighbour index for when the corpus grows. At small
+    # sizes the planner prefers the company index plus an exact sort, which is
+    # both faster and exact; see docs/ai.md.
+    op.execute(
+        "CREATE INDEX idx_issue_documents_embedding ON ai.issue_documents "
+        "USING hnsw (embedding vector_cosine_ops)"
+    )
+
+
+def downgrade() -> None:
+    op.execute("DROP TABLE ai.issue_documents")

ai-service/pyproject.toml

@@ -5,13 +5,24 @@ description = "NewnopDesk AI service: triage, sentiment and retrieval for client
 readme = "README.md"
 requires-python = ">=3.12,<3.13"
 dependencies = [
+    "alembic>=1.20.0",
     "fastapi>=0.142.2",
+    "fastembed>=0.8.1",
     "google-genai>=2.26.0",
+    "pgvector>=0.5.0",
+    "psycopg-pool>=3.3.3",
+    "psycopg[binary]>=3.3.6",
     "pydantic>=2.13.5",
     "pydantic-settings>=2.15.0",
     "uvicorn[standard]>=0.54.0",
 ]
 
+[project.scripts]
+# Database setup: `uv run ai-db bootstrap` (once, as admin), then `uv run ai-db migrate`
+ai-db = "ai_service.db.cli:main"
+# Start the service: `uv run ai-serve` (always on a selector event loop; see serve.py)
+ai-serve = "ai_service.serve:main"
+
 [build-system]
 requires = ["uv_build>=0.12.21,<0.13.0"]
 build-backend = "uv_build"
@@ -23,6 +34,7 @@ dev = [
     "pytest>=9.1.1",
     "pytest-asyncio>=1.4.0",
     "ruff>=0.16.9",
+    "testcontainers[postgres]>=4.15.0",
 ]
 
 # ─── Tooling ──────────────────────────────────────────────────────────────────
@@ -61,7 +73,9 @@ files = ["src", "tests"]
 testpaths = ["tests"]
 asyncio_mode = "auto"
 # Tests marked `live` call the real Gemini API and are skipped by default
-addopts = "--strict-markers -m 'not live'"
+addopts = "--strict-markers -m 'not live and not embedding'"
 markers = [
     "live: calls the real Gemini API; needs GEMINI_API_KEY. Run with: uv run pytest -m live",
+    "db: needs PostgreSQL with pgvector (a throwaway container is started automatically)",
+    "embedding: loads the real fastembed model (downloads ~67 MB on first run)",
 ]