A RAG-based Document Q&A Agent using Azure AI Foundry & Document Intelligence

This system reads business documents and answers questions about them. Its architecture is based on a system developed for a customer. Customer-specific features and extra components were removed to create a template that can be used for any project.

You can upload documents such as invoices, contracts, and reports. The system extracts structured fields using Azure Document Intelligence, indexes the document text for hybrid semantic search with Azure AI Search, and lets you have a multi-turn conversation about the documents through an Azure OpenAI agent orchestrated with LangGraph. The agent can retrieve relevant passages, look up extracted fields, and compare documents.

It is exposed as a small FastAPI service and deployed to Azure Container Apps with Bicep files and a GitHub Actions + OIDC CD pipeline.

There is no keys in code or CD. Every service-to-service call uses Managed Identity and RBAC no keys in code or CD.

Architecture

graph TD

User((User/Client))

App[FastAPI App<br/>Azure Container Apps]
Blob[(Blob Storage<br/>Raw + JSON)]
DocInt[Document Intel.<br/>Form Recognizer]
AOAI[Azure OpenAI<br/>GPT + Embed]
AISearch[Azure AI Search<br/>Vector + BM25]

Telemetry[App Insights / OpenTelemetry]
MLflow[MLflow<br/>Azure AI Foundry]

subgraph Azure[Azure]
    App --> Blob
    App --> DocInt
    App --> AOAI
    App --> AISearch
    App -.-> Telemetry
    App <-.-> MLflow

end

User -->|/ingest, /chat, /status| App

style Azure fill:none,stroke:#0078D4,stroke-width:2px,stroke-dasharray: 8 4

Loading

API

Endpoint	Purpose
POST /ingest	Stores file, extracts and chunks text and save the embedding vector in the index. Returns {doc_id, status, n_chunks, fields_summary, source_name}.
POST /chat	Multi-turn Q&A. Returns {answer, citations, trace}.
GET /status	Health + which dependency endpoints are configured.

The agent has three tools: search_index (hybrid retrieval), get_document_fields (exact structured fields), compare_documents (cross-document diff).

Provision infrastructure

One resource group, one region, all in Bicep:

make infra RG=rg-docagent-demo LOCATION=swedencentral

This provision all resources but the FastAPI app (the Container APP). This should be run once manually.

scripts/bootstrap-oidc.sh <subscription-id> <github-org/repo> rg-docagent-demo

Tear everything down:

make teardown RG=rg-docagent-demo

Deploy

Push to main. GitHub Actions does the rest: lint, test, OIDC login, Bicep deploy, az acr build, Trivy scan, az containerapp update, smoke-test GET /status. Set repo variables AZURE_CLIENT_ID, AZURE_TENANT_ID, AZURE_SUBSCRIPTION_ID. See .github/workflows/deploy.yml.

Rollback: re-point the app at a previous SHA:

az containerapp update -g rg-docagent-demo -n ca-docagent \
  --image crdocagentdemo.azurecr.io/docagent:<previous-sha>