fix: reclaim SQLite space after removing documentation

docs/concepts/data-storage.md

@@ -419,9 +419,14 @@ Health monitoring capabilities:
 
 ### Maintenance Operations
 
-Regular maintenance tasks:
+SQLite does not shrink the database file when documents are deleted. Deleted rows become free pages, and WAL mode records those deletes in `documents.db-wal`, so the on-disk size can grow even after a remove.
+
+After bulk deletes (`removeVersion` and `removeAllDocuments`) the store runs `wal_checkpoint(PASSIVE)`. That never waits on readers or writers, so search stays available. It truncates WAL frames when nothing else is using the file. It does not shrink `documents.db`.
+
+`VACUUM` takes an exclusive lock and blocks readers. Run `docs-mcp-server compact` (or `compact` against a remote worker with `--server-url`) when the store is idle to rewrite the file and reclaim free pages. The compact command stores SQLite temporary data on disk instead of in process memory, so it keeps RSS low but needs enough free disk space for SQLite's temporary copy of the database. In-memory databases skip compaction. Single-page deletes during refresh do not checkpoint or vacuum.
+
+Other regular maintenance:
 
-- VACUUM operations for space recovery
 - Index rebuilding via REINDEX
 - Orphaned record cleanup via foreign key constraints
 - Performance analysis using EXPLAIN QUERY PLAN

docs/guides/basic-usage.md

@@ -81,14 +81,15 @@ npx @arabold/docs-mcp-server@latest fetch-url https://react.dev/reference/react/
 | `find-version <library>` | Resolve the best matching version for a library |
 | `refresh <library>` | Re-scrape an existing library, skipping unchanged pages |
 | `remove <library>` | Delete a library or version from the index |
+| `compact` | Reclaim unused SQLite pages and shrink the store. Takes an exclusive lock, may block searches, and needs temporary disk space while it runs. |
 
 Run `npx @arabold/docs-mcp-server@latest --help` for the full command reference.
 
 ### Output Behavior
 
 - Structured commands (`list`, `search`, `find-version`) default to **JSON** on stdout in non-interactive runs.
 - Use `--output json|yaml|toon` to pick a format.
-- Plain-text commands (`fetch-url`, `scrape`, `refresh`, `remove`) write their output directly to stdout.
+- Plain-text commands (`fetch-url`, `scrape`, `refresh`, `remove`, `compact`) write their output directly to stdout.
 - Use `--quiet` to suppress non-error diagnostics or `--verbose` for debug output.
 - In non-interactive runs, diagnostics stay off stdout so agents and scripts can parse results safely.
 

skills/docs-manage/SKILL.md

@@ -3,8 +3,9 @@ name: docs-manage
 description: >-
   Manage the Grounded Docs MCP Server documentation index. Covers scraping
   and indexing documentation from URLs or local files, refreshing existing
-  indexes with changed content, and removing libraries from the index.
-  Use when you need to add, update, or delete indexed documentation.
+  indexes with changed content, removing libraries from the index, and
+  compacting the SQLite store. Use when you need to add, update, or delete
+  indexed documentation, or reclaim disk space after removals.
 compatibility: Requires Node.js 22+ and npx
 metadata:
   author: grounded.tools
@@ -21,6 +22,7 @@ on stdout.
 - A library is not yet indexed and you need its docs available for search.
 - Documentation may be stale and you want to pull in updated pages.
 - You want to remove a library or version from the index to free space.
+- The store file is still large after removals and you want to compact it.
 
 ## Commands
 
@@ -123,6 +125,37 @@ npx @arabold/docs-mcp-server@latest remove react --version 18.3.1
 ```
 
 This is destructive and cannot be undone. Re-run `scrape` to re-index.
+Bulk deletes checkpoint the WAL without locking readers. Use `compact` when
+idle if the main store file is still large.
+
+### compact
+
+Reclaim unused SQLite pages and truncate the WAL file so the store shrinks
+on disk. This takes an exclusive lock and may block searches until it
+finishes. VACUUM uses temporary files to keep memory usage low, so make sure
+there is enough free disk space for SQLite's temporary copy of the database.
+
+```bash
+npx @arabold/docs-mcp-server@latest compact [options]
+```
+
+| Flag | Alias | Description |
+|------|-------|-------------|
+| `--force` | | Run VACUUM even when SQLite reports no free pages |
+| `--server-url <url>` | | Remote pipeline worker URL |
+| `--quiet` | | Suppress non-error diagnostics |
+| `--verbose` | | Enable debug logging |
+
+Example:
+
+```bash
+npx @arabold/docs-mcp-server@latest compact
+```
+
+Removing documentation does not shrink the main SQLite file. Bulk deletes
+only run a non-blocking WAL checkpoint. Run this command when the store is
+idle to VACUUM and reclaim free pages. Add `--force` only when you want to
+rewrite the store even if SQLite reports no free pages.
 
 ## Output behaviour
 
@@ -145,6 +178,9 @@ npx @arabold/docs-mcp-server@latest refresh react --version 19.0.0
 
 # 3. Clean up old versions
 npx @arabold/docs-mcp-server@latest remove react --version 18.3.1
+
+# 4. Reclaim unused store space if needed
+npx @arabold/docs-mcp-server@latest compact
 ```
 
 ## Important notes

src/cli/commands/compact.test.ts

@@ -0,0 +1,76 @@
+/** Unit test for compact command */
+
+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
+import yargs from "yargs";
+import { createCompactCommand } from "./compact";
+
+const stdoutWriteMock = vi.fn();
+
+const compactFn = vi.fn(async () => ({
+  skipped: false,
+  vacuumed: true,
+  beforeBytes: 2048,
+  afterBytes: 1024,
+  reclaimedBytes: 1024,
+}));
+vi.mock("../../store", () => ({
+  createDocumentManagement: vi.fn(async () => ({
+    shutdown: vi.fn(),
+    compact: compactFn,
+  })),
+}));
+vi.mock("../utils", () => ({
+  getGlobalOptions: vi.fn(() => ({ storePath: undefined })),
+  getEventBus: vi.fn(() => ({
+    on: vi.fn(),
+    emit: vi.fn(),
+  })),
+  CliContext: {},
+  setupLogging: vi.fn(),
+}));
+vi.mock("../../utils/config", async (importOriginal) => {
+  const actual = await importOriginal<typeof import("../../utils/config")>();
+  return {
+    ...actual,
+    loadConfig: vi.fn(() => ({
+      app: { storePath: "/mock/store" },
+    })),
+  };
+});
+
+describe("compact command", () => {
+  let stdoutWriteSpy: { mockRestore: () => void };
+
+  beforeEach(() => {
+    vi.clearAllMocks();
+    stdoutWriteMock.mockReset();
+    stdoutWriteSpy = vi
+      .spyOn(process.stdout, "write")
+      .mockImplementation(stdoutWriteMock as any);
+  });
+
+  afterEach(() => {
+    stdoutWriteSpy.mockRestore();
+  });
+
+  it("compacts when free pages exist and reports reclaimed space", async () => {
+    const parser = yargs().scriptName("test");
+    createCompactCommand(parser);
+
+    await parser.parse("compact");
+
+    expect(compactFn).toHaveBeenCalledWith({ force: false });
+    expect(stdoutWriteMock).toHaveBeenCalledWith(
+      "Compacted store from 2.0 KB to 1.0 KB (reclaimed 1.0 KB).\n",
+    );
+  });
+
+  it("forces vacuum when requested", async () => {
+    const parser = yargs().scriptName("test");
+    createCompactCommand(parser);
+
+    await parser.parse("compact --force");
+
+    expect(compactFn).toHaveBeenCalledWith({ force: true });
+  });
+});

src/cli/commands/compact.ts

@@ -0,0 +1,73 @@
+/**
+ * Compact command - Reclaims unused SQLite pages and truncates the WAL file.
+ */
+
+import type { Argv } from "yargs";
+import { createDocumentManagement } from "../../store";
+import { TelemetryEvent, telemetry } from "../../telemetry";
+import { loadConfig } from "../../utils/config";
+import { logger } from "../../utils/logger";
+import { formatBytes } from "../../utils/string";
+import { renderTextOutput } from "../output";
+import { type CliContext, getEventBus } from "../utils";
+
+export function createCompactCommand(cli: Argv) {
+  cli.command(
+    "compact",
+    "Reclaim unused space in the document store (exclusive lock; may block searches)",
+    (yargs) => {
+      return yargs
+        .option("force", {
+          type: "boolean",
+          description: "Run VACUUM even when SQLite reports no free pages",
+          default: false,
+        })
+        .option("server-url", {
+          type: "string",
+          description:
+            "URL of external pipeline worker RPC (e.g., http://localhost:8080/api)",
+          alias: "serverUrl",
+        });
+    },
+    async (argv) => {
+      await telemetry.track(TelemetryEvent.CLI_COMMAND, {
+        command: "compact",
+        useServerUrl: !!argv.serverUrl,
+      });
+
+      const serverUrl = argv.serverUrl as string | undefined;
+      const appConfig = loadConfig(argv, {
+        configPath: argv.config as string,
+        searchDir: argv.storePath as string,
+      });
+
+      const eventBus = getEventBus(argv as CliContext);
+
+      const docService = await createDocumentManagement({
+        serverUrl,
+        eventBus,
+        appConfig,
+      });
+      try {
+        const result = await docService.compact({ force: argv.force === true });
+
+        if (result.skipped) {
+          renderTextOutput("Skipped compaction for in-memory store.");
+        } else if (result.reclaimedBytes > 0) {
+          renderTextOutput(
+            `Compacted store from ${formatBytes(result.beforeBytes)} to ${formatBytes(result.afterBytes)} (reclaimed ${formatBytes(result.reclaimedBytes)}).`,
+          );
+        } else {
+          renderTextOutput("Store is already compact.");
+        }
+      } catch (error) {
+        logger.error(
+          `❌ Failed to compact store: ${error instanceof Error ? error.message : String(error)}`,
+        );
+        throw error;
+      } finally {
+        await docService.shutdown();
+      }
+    },
+  );
+}

src/cli/index.ts

@@ -14,6 +14,7 @@ import {
 import { loadConfig } from "../utils/config";
 import { resolveStorePath } from "../utils/paths";
 // Commands
+import { createCompactCommand } from "./commands/compact";
 import { createConfigCommand } from "./commands/config";
 import { createDefaultAction } from "./commands/default";
 import { createFetchUrlCommand } from "./commands/fetchUrl";
@@ -164,6 +165,7 @@ export function createCli(argv: string[]): Argv {
     .showHelpOnFail(true);
 
   // Register Commands
+  createCompactCommand(cli);
   createConfigCommand(cli);
   createDefaultAction(cli);
   createFetchUrlCommand(cli);

src/store/DocumentManagementClient.ts

@@ -11,6 +11,7 @@ import type { IDocumentManagement } from "./trpc/interfaces";
 import type { DataRouter } from "./trpc/router";
 import type {
   ActivityHistory,
+  CompactResult,
   DbVersionWithLibrary,
   EmbeddingConfigInfo,
   FindVersionResult,
@@ -92,6 +93,10 @@ export class DocumentManagementClient implements IDocumentManagement {
     await this.client.removeAllDocuments.mutate({ library, version: version ?? null });
   }
 
+  async compact(options?: { force?: boolean; vacuum?: boolean }): Promise<CompactResult> {
+    return this.client.compact.mutate(options ?? {});
+  }
+
   async getVersionsByStatus(statuses: VersionStatus[]): Promise<DbVersionWithLibrary[]> {
     return this.client.getVersionsByStatus.query({
       statuses: statuses as unknown as string[],

src/store/DocumentManagementService.test.ts

@@ -39,6 +39,18 @@ const mockStore = {
   queryLibraryVersions: vi.fn().mockResolvedValue(new Map<string, any[]>()),
   addDocuments: vi.fn(),
   deletePages: vi.fn(),
+  compact: vi.fn().mockResolvedValue({
+    skipped: false,
+    vacuumed: true,
+    beforeBytes: 100,
+    afterBytes: 50,
+    reclaimedBytes: 50,
+  }),
+  removeVersion: vi.fn().mockResolvedValue({
+    documentsDeleted: 1,
+    versionDeleted: true,
+    libraryDeleted: false,
+  }),
   // Status tracking methods
   updateVersionStatus: vi.fn(),
   updateVersionProgress: vi.fn(),
@@ -350,6 +362,7 @@ describe("DocumentManagementService", () => {
 
       await docService.removeAllDocuments(library, version);
       expect(mockStore.deletePages).toHaveBeenCalledWith(library, version); // Fix: Use mockStoreInstance
+      expect(mockStore.compact).toHaveBeenCalledWith({ force: false, vacuum: false });
     });
 
     it("should handle removing documents with null/undefined/empty version", async () => {
@@ -362,6 +375,29 @@ describe("DocumentManagementService", () => {
       expect(mockStore.deletePages).toHaveBeenCalledWith(library, ""); // Fix: Use mockStoreInstance
     });
 
+    it("should still remove documents when compaction fails", async () => {
+      mockStore.compact.mockRejectedValueOnce(new Error("checkpoint busy"));
+
+      await expect(
+        docService.removeAllDocuments("test-lib", "1.0.0"),
+      ).resolves.toBeUndefined();
+      expect(mockStore.deletePages).toHaveBeenCalledWith("test-lib", "1.0.0");
+    });
+
+    it("should compact the store", async () => {
+      const compactResult = {
+        skipped: false,
+        vacuumed: true,
+        beforeBytes: 200,
+        afterBytes: 80,
+        reclaimedBytes: 120,
+      };
+      mockStore.compact.mockResolvedValueOnce(compactResult);
+
+      await expect(docService.compact({ force: true })).resolves.toEqual(compactResult);
+      expect(mockStore.compact).toHaveBeenCalledWith({ force: true });
+    });
+
     describe("listVersions", () => {
       it("should return an empty array if the library has no documents", async () => {
         mockStore.queryUniqueVersions.mockResolvedValue([]); // Fix: Use mockStoreInstance