graph-sync

Keep a RAG knowledge base in sync with Microsoft 365. graph-sync is a Rust service that incrementally mirrors a OneDrive or SharePoint document library using Microsoft Graph delta queries, converts Word documents to Markdown, triggers re-indexing in agentic-rag-assistant, and logs every run to a Dataverse table so the Power Platform can alert on failures and chart throughput.

# illustrative output
$ graph-sync sync
run #1 succeeded (full resync): +214 added, ~0 updated, -0 deleted, >0 moved, 37 skipped, 0 failed (48.2 MB, 9312 ms)
$ graph-sync sync          # later: only what changed is fetched
run #2 succeeded: +1 added, ~2 updated, -1 deleted, >5 moved, 0 skipped, 0 failed (0.3 MB, 611 ms)

Why it exists

Company knowledge lives in SharePoint and OneDrive, but a RAG pipeline needs it as files on disk and needs to know when those files change. Re-downloading a whole library on a schedule is slow and hits Graph throttling limits. graph-sync asks Graph only for what changed since last time, reproduces those changes locally (including renames and moves, without downloading again), and tells the RAG service to re-index.

Architecture

flowchart LR
    subgraph M365["Microsoft 365"]
        OD[(OneDrive /<br/>SharePoint)]
        DV[(Dataverse<br/>Sync Runs table)]
    end
    subgraph GS["graph-sync (Rust)"]
        AUTH[auth<br/>device code · client creds<br/>single-flight token cache]
        GC[graph client<br/>delta paging · retry/backoff<br/>streaming download]
        ENG[sync engine<br/>reconcile · move · delete]
        QX[QuickXorHash<br/>integrity check]
        EX[docx → Markdown]
        ST[(SQLite state<br/>item tree · delta link · runs)]
    end
    subgraph RAG["agentic-rag-assistant"]
        API[POST /ingest]
        CH[(ChromaDB)]
    end
    PA[Power Automate<br/>failure alerts]
    PBI[Power BI<br/>run history]

    OD -- "GET /root/delta" --> GC
    OD -- "GET /items/{id}/content" --> GC
    AUTH --> GC
    GC --> ENG
    ENG <--> ST
    ENG --> QX --> EX --> CORPUS[/corpus/ mirror/]
    ENG -- "run changed files" --> API
    CORPUS --> API --> CH
    ENG -- "POST /api/data/v9.2/…" --> DV
    DV --> PA
    DV --> PBI

Loading

What happens in one run

1. Enumerate. Page through /root/delta, following @odata.nextLink until the final page returns an @odata.deltaLink. On the first run that lists the whole drive; afterwards the stored delta link returns only changes. If Graph answers 410 Gone (link expired), the engine drops it and re-enumerates, then reconciles away anything no longer on the server.
2. Record. Upsert every item's id, parentReference.id, name, cTag and hash into SQLite. Graph deliberately omits paths from delta responses, and renaming a folder reports only the folder itself, so local paths are rebuilt by walking parent ids (a recursive CTE finds a deleted folder's descendants).
3. Reconcile. For every known file, compare where it should be with where it is. A different cTag means the content changed, so download it. Same cTag at a new path means the file was renamed or moved (or its folder was), so rename it on disk with no download. If it's now outside SYNC_FOLDER, an unsupported type, or too large, remove it.
4. Download. Up to SYNC_CONCURRENCY downloads run in parallel. Each is streamed to a .part file, hashed incrementally with QuickXorHash, rejected on mismatch, converted if it's a .docx, then renamed into place atomically.
5. Commit. The new delta link is saved only after everything above, so a crash replays the same changes rather than losing them. Failed files are marked failed and retried on the next run.
6. Notify. Sinks run: the RAG sink calls POST /ingest if anything changed, and the Dataverse sink writes the run row. If a sink fails, the error is logged and the sync still succeeds.

Engineering highlights

Concern	How it's handled	Where
Throttling	Honours Retry-After (seconds or HTTP-date, capped); otherwise exponential backoff with full jitter; retries 429/5xx and connect/timeout errors	retry.rs, http.rs
Token storms	Single-flight token cache: 16 concurrent callers trigger 1 token request; refresh 2 min before expiry; one forced refresh on 401	auth/mod.rs
Integrity	Rust port of Microsoft's QuickXorHash, streaming, verified against rclone's independent test vectors	quickxor.rs
Renames without re-download	cTag (content) vs. path diff; paths rebuilt from the id tree	sync.rs, state.rs
Crash safety	.part + atomic rename; delta link committed last; failed items retried	sync.rs
Untrusted names	Component sanitising, ../absolute-path rejection, case-insensitive collision detection	paths.rs
Blocking work	.docx unzip + XML parse moved to spawn_blocking	sync.rs
Secrets at rest	State DB (holds the refresh token) created 0600; tokens never logged; Authorization stripped on the cross-host download redirect	state.rs, graph.rs
Schema evolution	PRAGMA user_version migrations; refuses to open a newer schema	state.rs

Quick start

Prerequisites: Rust 1.88+ (rustup.rs) and an Entra ID app registration (docs/AZURE_SETUP.md, ~5 minutes).

git clone https://github.com/PreetRaut/graph-sync && cd graph-sync
cp .env.example .env          # set GRAPH_CLIENT_ID
cargo run --release -- login  # device code sign-in, once
cargo run --release -- sync   # mirror into ./corpus
cargo run --release -- status # state + run history

Fastest possible try-out (no app registration): sign in to Graph Explorer, copy the access token from the "Access token" tab, then:

GRAPH_ACCESS_TOKEN=eyJ0eXAi... cargo run --release -- sync

(The token lasts about an hour, so this is for kicking the tyres, not for running on a schedule.)

Feed the RAG assistant

Start agentic-rag-assistant (uvicorn src.api.main:app), then set:

RAG_URL=http://localhost:8000
RAG_INGEST_DIR=/absolute/path/to/graph-sync/corpus

Each run that changes the mirror calls POST /ingest. Then ask questions about your SharePoint content with POST /query. With Docker, docker compose --profile rag up -d runs both services with a shared volume (see docker-compose.yml).

Log runs to Dataverse and alert with Power Automate

Create the Sync Runs table and flow described in docs/POWER_PLATFORM.md, then set:

DATAVERSE_URL=https://yourorg.crm4.dynamics.com
DATAVERSE_ENTITY_SET=cr123_syncruns
DATAVERSE_COLUMN_PREFIX=cr123_

Commands

Command	Purpose
graph-sync login	Device code sign-in; stores a refresh token in the state DB
graph-sync sync	One run. Exit code 0 = succeeded, 2 = partial (some files failed, will retry), 1 = failed
graph-sync sync --watch --interval 300	Keep syncing until Ctrl-C
graph-sync status	Delta-link state, file counts, recent runs
graph-sync reset [--logout]	Forget the delta link (forces full resync); optionally sign out

All settings are environment variables. .env.example documents each one. --json-logs / LOG_JSON=true switches to structured logs.

Testing

cargo test                                   # 51 tests, no network needed
cargo clippy --all-targets -- -D warnings

The integration tests in tests/ run the real engine against a mock Microsoft Graph, identity platform, RAG API and Dataverse (wiremock). They cover:

• a four-run scenario: full sync over two pages, then an edit plus a folder rename (the file moves with no re-download), then a delete, then a no-op
• 410 Gone → full resync that reconciles server-side deletions
• a corrupted download rejected by QuickXorHash and retried next run
• the /content 302 redirect followed without forwarding the bearer token
• 429 + Retry-After honoured; persistent 503 fails the run without losing the delta link
• hostile names (.., a?b) staying inside the mirror; a sanitised-name collision not evicting the existing file; SYNC_FOLDER scoping; moving a file out of scope
• the client credentials, device code (pending → approved / declined) and refresh token rotation flows
• the RAG sink firing only on change; Dataverse rows with prefixed columns; a failing sink not failing the sync

CI runs fmt, clippy, tests on Linux/macOS/Windows, an MSRV check, cargo audit and a Docker build.

Project status and limitations

• Verified end to end against mocked APIs. Before relying on it, check a first live run against your own tenant using the checklist in docs/AZURE_SETUP.md.
• RAG deletions: agentic-rag-assistant's /ingest only upserts, so chunks for deleted or renamed files stay in ChromaDB until it is rebuilt. The planned fix is a DELETE /documents/{doc_id} endpoint there, called from the RAG sink with the removed paths.
• Scale: reconciliation scans all known files each run (fast in SQLite up to tens of thousands of files). Beyond that, restrict the scan to items touched by the delta and their descendants.
• Change notifications: runs are polled. Graph webhooks (/subscriptions) could trigger runs instead, but they need a public HTTPS endpoint.
• PDFs, .txt and .md pass through unchanged (the RAG service parses PDFs itself). Other Office formats are skipped.

Docs

• docs/AZURE_SETUP.md: app registration, permissions, finding site and drive IDs
• docs/POWER_PLATFORM.md: Dataverse table, application user, Power Automate alert, Power BI
• docs/WALKTHROUGH.md: design decisions and trade-offs, in detail

License

MIT