Notion Query Translator

Notion Query Translator is an advanced, LLM-powered agent system designed to translate natural language user requests into executable, precise Python code to interact with the Notion API.

Built on top of LangGraph, it features a robust Retrieval-Augmented Generation (RAG) pipeline to consult Notion's API documentation, an isolated code-execution sandbox, and a self-reflection loop that autonomously debugs and corrects API mismatches before returning the final result.

---

🎯 What it does

Instead of manually writing complex JSON payloads, handling pagination, or managing database relation lookups, you can simply ask the agent to perform a task:

"Find tasks where the 'Last Reviewed' date is older than 30 days. Archive them and add a comment to each saying 'This task is stale. Archived.'"

The agent will:

1. Retrieve the correct Notion API documentation and your workspace's database schema.
2. Plan a step-by-step implementation.
3. Generate the Python code using the requests library.
4. Execute the code in a sandboxed environment.
5. Reflect & Repair if the Notion API returns an error (e.g., HTTP 400 Bad Request), adjusting schemas and re-executing until successful.

✨ Key Features

• Agentic Workflow: Uses a state machine (precheck → resolve → retrieve → plan → codegen → execute → reflect) to ensure high-reliability code generation.
• Interactive Entity Resolution: A human-in-the-loop mechanism that automatically queries Notion for page titles mentioned in the prompt and uses a rich terminal UI to ask the user to disambiguate if multiple pages share the same name.
• Interactive CLI Loop: Run in shell mode (no prompt argument) and use /config --think|--no-think, /clear, and /exit between turns.
• Layered Runtime Environment: Loads .env first, then .env.sandbox with override semantics for ephemeral sandbox IDs.
• Local RAG Pipeline: Ingests Notion documentation into a local Qdrant vector database, chunked optimally with chonkie and embedded using fastembed.
• Advanced Query Engineering: Implements Multi-Query, Chain-of-Thought (CoT) Decomposition, and Domain Decomposition to maximize retrieval accuracy.
• Evaluation Harness: Deep integration with LangSmith for running deterministic, synthetic API benchmarks (evals/) across different LLM models (e.g., Google Gemini, Gemma).
• Automated Error Analysis: Consolidates evaluation results and posts structured diagnostic reports directly back to your Notion workspace.

---

📚 Documentation Map

• System architecture and major decisions: docs/ARCHITECTURE.md
• Crucial component tradeoffs: docs/COMPONENTS.md
• Evaluation design and caveats: docs/EVALUATION.md

Default strategy (current)

• Uses a deterministic hardcoded context by default.
• Keeps planning disabled by default.
• Uses self-correction as the primary retry behavior.
• Executes in sandbox mode by default with egress controls.

These defaults were chosen from iterative evaluation results and remain configurable.

🚀 Getting Started

Prerequisites

• Python >= 3.13.5
• A Notion Integration Token (Create one at notion.so/my-integrations)
• Google Gemini API Key (or applicable OpenAI-compatible endpoints)
• uv (Recommended for fast dependency management)

Installation

1. Clone the repository:

   git clone https://github.com/your-username/notion-query-translator.git
   cd notion-query-translator

Install dependencies: This project uses pyproject.toml and supports uv.

# Create a virtual environment and install the package with dev dependencies

uv venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
uv pip install -e ".[dev]"

Configure Environment Variables: Copy the example environment file and fill in your keys:

cp .env.example .env

Make sure to fill in your NOTION_TOKEN, GOOGLE_API_KEY, and LANGSMITH_API_KEY (if running evaluations).

Optional sandbox override layer:

• .env stores your stable secrets and default IDs.
• .env.sandbox stores ephemeral IDs generated by sandbox provisioning.
• At runtime, .env.sandbox overrides overlapping IDs from .env.

Provision the Test Environment (Optional but recommended): If you want to run the evaluation suite safely without touching your real Notion data, run the setup script to provision a sandbox Notion database:

python -m src.evaluation.sandbox

This writes the generated IDs to .env.sandbox (it does not overwrite .env).

Build the RAG Database: Initialize the local Qdrant vector database with the Notion API corpora:

python scripts/build_rag.py

💻 Usage Command Line Interface (CLI)

The project includes a CLI powered by Typer with a rich terminal UI. When you run a command, you'll see:

• A splash screen introducing the application.
• A dynamic spinner that updates in real-time as the agent progresses through its pipeline stages (precheck → retrieve → codegen → execute → reflect).
• Beautiful page rendering — affected Notion pages are fetched and displayed with rich property tables and formatted markdown.
• All internal logs are silently routed to logs/logs.log.

# Start interactive shell mode

notion-agent run

# Inside shell mode, use slash commands such as:
# /config --think
# /config --no-think
# /clear
# /exit

# Run a simple request

notion-agent run "Create a new task called 'Review Architecture'"

# Run with self-reflection and planning enabled (Think Mode)

notion-agent run "Find all unstarted urgent tasks and mark them as Do Now" --think

Example output:

 ╔══════════════════════════════════════════════════════╗
 ║          Notion Query Translator  v0.1              ║
 ║  Translate natural language → Notion API actions    ║
 ╚══════════════════════════════════════════════════════╝

 Prompt: Create a new task called 'Review Architecture'

 ⠋ Writing Python script for Notion API...

 📄 1 page(s) affected
 ─────────── Architecture Review ───────────
  Page Properties
  Property  │ Value
  Status    │ In Progress
  Priority  │ High
  ...

Running the Evaluation Pipeline

To run the LangSmith evaluation suite against the YAML test cases defined in evals/:

# Execute the agent against the test cases

python -m notion_query.run_pipeline

# Evaluate the generated artifacts (Code, RAG context, Execution pass/fail) using LLM-as-a-judge

python evaluation/evaluate.py

Retrieving your Database Schemas

To ground the agent in your specific workspace structure, you can generate a schema report:

python scripts/schema_retriever.py

This fetches your workspace databases and generates a token-efficient Python TypedDict representation to be injected into the LLM context.

📁 Project Structure

dmsavkov-notion-query-translator/
├── data/
│ ├── .qdrant_storage/ # Local Qdrant vector DB
│ └── context/ # Hardcoded contexts and schema reports
├── evals/ # YAML-based evaluation benchmarks (simple & complex)
├── logs/ # Runtime logs (backend stdout captured here)
├── notion_query/ # CLI entry points and runtime env bootstrap
├── scripts/ # Utilities for schema retrieval and error analysis
├── src/ # Core application logic
│ ├── core/ # Lifecycle and execution orchestration
│ ├── models/ # Config/state models and prompt/context assets
│ ├── presentation/ # Rich CLI rendering & Notion data fetching
│ │ ├── cli_shell.py # Interactive loop prompt + slash command parsing
│ │ ├── notion_requesting.py # Notion API fetch (requests-based)
│ │ ├── sanitization.py # Markdown cleanup & property flattening
│ │ ├── ui_bridge.py # Pure-state singleton for streaming UI
│ │ └── viewer.py # Rich table/panel/markdown renderer
│ ├── utils/ # Qdrant/OpenAI/execution/telemetry helpers
│ ├── nodes.py # LangGraph node definitions
│ └── evaluator.py # LangSmith evaluation judges
└── tests/ # Pytest test suites (unit, integration, smoke)
    └── fixtures/ # Static Notion API payloads for testing

🤝 Contributing

Contributions are welcome! Please follow these steps:

• Fork the repository.
• Create a new branch (git checkout -b feature/amazing-feature).
• Run the test suite to ensure nothing is broken (pytest tests/).
• Commit your changes (git commit -m 'Add amazing feature').
• Push to the branch (git push origin feature/amazing-feature).
• Open a Pull Request.

Please ensure all new functionality is covered by unit tests in the tests/ directory.

🆘 Help and Support

• If you encounter a bug or have a feature request, please open an issue on GitHub.
• For questions about the Notion API itself, refer to the Official Notion API Documentation.
• For tracing and debugging LLM trajectories, check your LangSmith Dashboard.

Maintainer: dmsavkov