Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 18 additions & 4 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,24 @@

## [Unreleased]

- Backup manifests: each batch run writes `~/.mda/manifests/<run_id>.json` (override with `MDA_MANIFEST_DIR`); `mda restore --list` shows runs (apply not implemented yet).
- TUI: press **S** to cycle bundled skills (`markdown-document-architect`, `categorize-vault-notes`).
- Docs: `docs/LOCAL-DEV.md` for local branch workflow without GitHub CI.
- README: note that integration tests need `ANTHROPIC_API_KEY`.
## [0.3.1]

- **TUI pre-flight:** Modal before batch **P** with file count, output mode, size/token warnings, Continue/Cancel.
- **TUI undo:** **U** restores files from the latest batch manifest `backup_path` entries (in-place runs with backups).
- **TUI manifests:** Each **P** batch writes `~/.mda/manifests/<run_id>.json` for CLI/TUI restore.
- **Docs:** `docs/RECOMMENDATIONS-REPORT.md` with follow-up actions and issue status.

## [0.3.0]

- **Restore apply:** `mda restore <run_id>` dry-run; `mda restore <run_id> --apply --yes` copies from manifest `backup_path` entries.
- **Batch checkpoint:** `--checkpoint` copies sources to `~/.mda/checkpoints/<run_id>/` before processing (`MDA_CHECKPOINT_DIR` override).
- **Vault scan:** `mda vault-scan <vault>` alias for `mda script scan_vault --skill categorize-vault-notes`; TUI **Shift+V** runs scan on current folder with log output.
- **Cost warnings:** `warn_large_inputs` reports rough token budget when `--max-tokens` may be tight for the batch.
- **TUI preview:** extraction/read errors show a prominent `EXTRACTION ERROR` banner and red metadata line.
- **Security:** `docs/SECURITY.md` with API key rotation runbook (#24).
- Backup manifests: each batch run writes `~/.mda/manifests/<run_id>.json`; `mda restore --list` lists runs.
- TUI: press **S** to cycle bundled skills.
- Docs: `docs/LOCAL-DEV.md`; README notes optional integration CI (`continue-on-error` on `main`).

## [0.2.10]

Expand Down
16 changes: 13 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# mda-cli

**Roadmap:** [docs/ROADMAP.md](docs/ROADMAP.md) · **Gap analysis:** [docs/GAP-ANALYSIS.md](docs/GAP-ANALYSIS.md) · **Recommendations:** [docs/RECOMMENDATIONS.md](docs/RECOMMENDATIONS.md) · **Vault playbook:** [docs/VAULT-PLAYBOOK.md](docs/VAULT-PLAYBOOK.md) · **Troubleshooting:** [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md) ([tracked issues](https://github.com/B0LK13/mda-cli/issues))
**Roadmap:** [docs/ROADMAP.md](docs/ROADMAP.md) · **Gap analysis:** [docs/GAP-ANALYSIS.md](docs/GAP-ANALYSIS.md) · **Recommendations:** [docs/RECOMMENDATIONS.md](docs/RECOMMENDATIONS.md) · **Status report:** [docs/RECOMMENDATIONS-REPORT.md](docs/RECOMMENDATIONS-REPORT.md) · **Vault playbook:** [docs/VAULT-PLAYBOOK.md](docs/VAULT-PLAYBOOK.md) · **Troubleshooting:** [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md) · **Security:** [docs/SECURITY.md](docs/SECURITY.md) ([tracked issues](https://github.com/B0LK13/mda-cli/issues))

CLI for **Markdown Document Architect (MDA)** and related bundled skills: loads `SKILL.md` and companion standards (bundled by default), calls Anthropic or OpenRouter, and writes restructured output (strips the outer fence when present).

Expand Down Expand Up @@ -106,9 +106,16 @@ mda path\to\file.md
mda path\to\vault -r --max-files 10 -q
mda path\to\vault --dry-run
mda path\to\file.md --in-place --backup
mda path\to\vault -r --checkpoint --max-files 10
mda restore --list
mda restore 20260101T120000Z
mda restore 20260101T120000Z --apply --yes
mda vault-scan E:\ObsidianVault7 --recursive
```

Optional environment variables: `MDA_SKILL` (skill id), `MDA_SKILL_DIR` (explicit skill folder), `MDA_EXTENSIONS`, `MDA_MODEL`, `MDA_MAX_TOKENS`, `MDA_API_TIMEOUT`. See [.env.example](.env.example) for a full template.
Optional environment variables: `MDA_SKILL` (skill id), `MDA_SKILL_DIR` (explicit skill folder), `MDA_EXTENSIONS`, `MDA_MODEL`, `MDA_MAX_TOKENS`, `MDA_API_TIMEOUT`, `MDA_MANIFEST_DIR`, `MDA_CHECKPOINT_DIR`. See [.env.example](.env.example) for a full template.

**CI:** Unit tests and smoke install run on every push/PR. An optional integration job on `main` runs live API tests when `ANTHROPIC_API_KEY` is set; it uses `continue-on-error: true` so billing or missing secrets do not block releases.

### Cost guardrails

Expand All @@ -128,14 +135,16 @@ mda E:\ObsidianVault7 -r --max-files 5 -q
mda E:\ObsidianVault7 -r --max-files 20 --max-tokens 4096
```

TUI: select a small set of files manually before pressing **P**. See [docs/VAULT-PLAYBOOK.md](docs/VAULT-PLAYBOOK.md).
TUI: select a small set of files manually before pressing **P** (pre-flight confirmation with token/size warnings). After an in-place batch, press **U** to undo from the latest manifest backups. See [docs/VAULT-PLAYBOOK.md](docs/VAULT-PLAYBOOK.md).

**Output naming:** sibling `*.restructured.md` for the default MDA skill (including PDF/DOCX sources). The `categorize-vault-notes` skill writes `*.restructured.txt` for non-Markdown sources.

### Safe overwrite workflow

- `--in-place` overwrites the source file.
- `--backup` is available only with `--in-place` in batch mode and writes a sibling backup before replacing the file.
- `--checkpoint` copies each source file under `~/.mda/checkpoints/<run_id>/` before the batch starts (separate from per-file `.bak` backups).
- Each batch writes a manifest under `~/.mda/manifests/`; use `mda restore --list` and `mda restore <run_id> --apply --yes` to roll back from `backup_path` entries.
- Generated outputs are written atomically. If the process fails during write, the original destination file is left in place.
- `--json-lines` now includes `action`, `changed`, `bytes_in`, `bytes_out`, and `backup_path` fields.

Expand All @@ -158,6 +167,7 @@ Assign YAML frontmatter and optional folder hints for Obsidian vaults (flat or n
mda --skill categorize-vault-notes E:\ObsidianVault7\note.md
mda script --list --skill categorize-vault-notes
mda script scan_vault --skill categorize-vault-notes -- E:\ObsidianVault7 --limit 20
mda vault-scan E:\ObsidianVault7 --recursive --limit 20
mda script plan_moves --skill categorize-vault-notes -- E:\ObsidianVault7
mda script plan_moves --skill categorize-vault-notes -- E:\ObsidianVault7 --apply
```
Expand Down
79 changes: 79 additions & 0 deletions docs/RECOMMENDATIONS-REPORT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
# mda-cli Recommendations Report

**Date:** 2026-05-18
**Version:** 0.3.1 (branch `chore/close-p0-issues`)
**Related:** [RECOMMENDATIONS.md](RECOMMENDATIONS.md) · [IMPLEMENTATION-PLAN.md](IMPLEMENTATION-PLAN.md) · [GitHub Issues](https://github.com/B0LK13/mda-cli/issues)

---

## Executive summary

Milestone A and most of Milestone B on `chore/close-p0-issues` are implemented locally: CI/release scaffolding, restore apply, batch checkpoints, vault-scan, token warnings, and TUI polish. **v0.3.1** adds **batch pre-flight confirmation** before **P**, **one-level TUI undo** (**U**) from the latest manifest `backup_path` entries, and this report.

Remaining high-value work: merge/tag **0.3.x**, green or documented CI, full TUI skill picker (#27), vault workflow wizard (#32), and distribution docs (#17–#19). Integration tests (#4) stay optional on `main` until billing/credentials are stable.

---

## Completed since last report (v0.3.0 → v0.3.1)

| Area | Deliverable |
|------|-------------|
| **TUI** | Pre-flight modal before batch **P** — file count, output mode, size/token warnings, Continue/Cancel |
| **TUI** | **U** undo last batch via manifest `backup_path` (in-place runs with backups) |
| **TUI** | Batch writes `~/.mda/manifests/<run_id>.json` after each **P** run |
| **CLI** | `mda restore <run_id> --apply --yes` (v0.3.0) |
| **CLI** | `--checkpoint`, `mda vault-scan`, `warn_large_inputs` with `--max-tokens` |
| **Docs** | `docs/SECURITY.md`, `docs/LOCAL-DEV.md`, vault playbook & troubleshooting |
| **Quality** | Ruff + unit pytest in CI workflow (billing may block GitHub runs) |

---

## Open GitHub issues still relevant

| Issue | Priority | Topic | Notes |
|-------|----------|-------|-------|
| [#4](https://github.com/B0LK13/mda-cli/issues/4) | P1 | Integration CI | Optional job on `main`; needs API key + runner budget |
| [#14](https://github.com/B0LK13/mda-cli/issues/14) | P2 | Restore / rollback | **Largely done** — `restore --list` + `--apply`; TUI undo in 0.3.1 |
| [#17](https://github.com/B0LK13/mda-cli/issues/17)–[#21](https://github.com/B0LK13/mda-cli/issues/21) | P1–P2 | Distribution | Private wheel install, release checklist, pinning, PyPI/winget |
| [#22](https://github.com/B0LK13/mda-cli/issues/22)–[#23](https://github.com/B0LK13/mda-cli/issues/23) | P2 | CI quality | Coverage artifact, `ruff format --check` |
| [#25](https://github.com/B0LK13/mda-cli/issues/25)–[#26](https://github.com/B0LK13/mda-cli/issues/26) | P2–P1 | Security | Secret scanning; CI must not echo API keys |
| [#27](https://github.com/B0LK13/mda-cli/issues/27) | P1 | TUI skill picker | **S** cycles skills; full picker still open |
| [#28](https://github.com/B0LK13/mda-cli/issues/28)–[#29](https://github.com/B0LK13/mda-cli/issues/29) | P2 | TUI preview | Scroll/search; richer PDF preview |
| [#30](https://github.com/B0LK13/mda-cli/issues/30) | P2 | TUI undo | **Addressed in 0.3.1** for manifest-backed in-place batches |
| [#31](https://github.com/B0LK13/mda-cli/issues/31) | P1 | Pre-flight confirm | **Addressed in 0.3.1** |
| [#32](https://github.com/B0LK13/mda-cli/issues/32)–[#34](https://github.com/B0LK13/mda-cli/issues/34) | P1–P2 | Vault ops | Guided vault workflow; flatten wrapper; resume checkpoints |
| [#35](https://github.com/B0LK13/mda-cli/issues/35)–[#47](https://github.com/B0LK13/mda-cli/issues/47) | P2 | API cost, docs, providers | Token dashboards, docs site, multi-provider polish |

---

## Follow-up actions

| Action | Priority | Issue ref | Owner |
|--------|----------|-----------|-------|
| Merge `chore/close-p0-issues`, tag **v0.3.1**, publish release wheel | P0 | #8, #16 | Maintainer |
| Resolve or document CI billing; run smoke on Linux + Windows | P0 | #1, #2 | Maintainer |
| Close #30, #31 after verifying TUI preflight + undo in pilot | P1 | #30, #31 | Dev |
| Ship `docs/RELEASE-CHECKLIST.md` and private wheel install docs | P1 | #17, #18 | Dev |
| TUI skill picker (beyond **S** cycle) | P1 | #27 | Dev |
| Vault workflow wizard (flatten → scan → plan → MDA) | P1 | #32 | Dev |
| Enable integration job when `ANTHROPIC_API_KEY` available | P1 | #4 | DevOps |
| Coverage + format in CI | P2 | #22, #23 | Dev |
| Secret scanning in CI | P2 | #25 | DevOps |
| Evaluate PyPI vs private-only distribution | P2 | #20 | Product |

---

## IMPLEMENTATION-PLAN progress (WP snapshot)

| Work package | Status |
|--------------|--------|
| WP-A1 stabilization | ✅ Local |
| WP-B1 restore apply | ✅ |
| WP-B2 checkpoints | ✅ |
| WP-B3 vault-scan | ✅ |
| WP-B4 pre-flight (#31) | ✅ v0.3.1 |
| WP-B5 token warnings | ✅ |
| WP-B6 session undo (#30) | ✅ v0.3.1 (manifest / in-place) |
| WP-C distribution & docs | ⬜ |

See [IMPLEMENTATION-PLAN.md](IMPLEMENTATION-PLAN.md) for full sequencing and evidence tables.
33 changes: 33 additions & 0 deletions docs/SECURITY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# Security

mda-cli runs locally and sends document text to configured LLM providers (Anthropic, OpenRouter). This page covers secrets handling and key rotation — not WDAC or enterprise policy.

## Secrets

| Secret | Purpose |
|--------|---------|
| `ANTHROPIC_API_KEY` | Default provider API access |
| `OPENROUTER_API_KEY` | Fallback / alternate provider |
| `GITHUB_TOKEN` | Optional: install wheels from private GitHub Releases (see README) |

- Copy [.env.example](../.env.example) to `.env` for local development; **never commit** `.env` or keys.
- CI integration tests use `ANTHROPIC_API_KEY` from repository secrets only on `main` (optional job; failures do not block the pipeline).

## API key rotation

When a key may be exposed or you rotate credentials on a schedule:

1. **Create a new key** in the provider console (Anthropic or OpenRouter).
2. **Update local env**: edit `.env` or your shell profile; restart terminals and any running `mda` / TUI sessions.
3. **Update CI** (if used): GitHub → Settings → Secrets → update `ANTHROPIC_API_KEY`.
4. **Revoke the old key** in the provider console after confirming `mda --check` succeeds with the new key.
5. **Audit machines** that had the old key (shared PCs, backup scripts, password managers).

## Reporting issues

For security-sensitive bugs in this repository, contact the maintainer privately (do not open a public issue with keys or vault contents).

## Related docs

- [TROUBLESHOOTING.md](TROUBLESHOOTING.md) — install and runtime errors
- [README.md](../README.md) — cost guardrails and env vars
2 changes: 1 addition & 1 deletion mda_cli/__init__.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
"""Markdown Document Architect CLI package."""

__version__ = "0.2.10"
__version__ = "0.3.1"
171 changes: 164 additions & 7 deletions mda_cli/backup.py
Original file line number Diff line number Diff line change
@@ -1,20 +1,17 @@
"""Backup manifest tracking for batch runs (issue #14 groundwork).

Each batch run with ``--backup`` and/or in-place writes can append a JSON manifest
under ``~/.mda/manifests/`` (override with ``MDA_MANIFEST_DIR``). Full
``mda restore`` apply is not implemented yet; use ``mda restore --list`` to inspect.
"""
"""Backup manifests, restore apply, and optional batch checkpoints (issue #14, #34)."""

from __future__ import annotations

import json
import os
import shutil
from dataclasses import dataclass
from datetime import UTC, datetime
from pathlib import Path
from typing import TYPE_CHECKING

if TYPE_CHECKING:
from mda_cli.core import JobResult
from mda_cli.core import Job, JobResult

MANIFEST_VERSION = 1

Expand Down Expand Up @@ -105,3 +102,163 @@ def summarize_manifest(data: dict[str, object]) -> str:
if isinstance(e, dict) and e.get("backup_path")
)
return f"{run_id} {created} skill={skill} jobs={jobs} backups={backups}"


def default_checkpoint_dir() -> Path:
"""Directory where optional pre-batch source copies are stored."""
raw = os.environ.get("MDA_CHECKPOINT_DIR", "").strip()
if raw:
return Path(raw).expanduser()
return Path.home() / ".mda" / "checkpoints"


def find_manifest_path(run_id: str, manifest_dir: Path | None = None) -> Path | None:
"""Resolve a manifest file by run id (with or without ``.json`` suffix)."""
rid = run_id.removesuffix(".json")
root = (manifest_dir or default_manifest_dir()).expanduser()
direct = root / f"{rid}.json"
if direct.is_file():
return direct
for path in list_manifest_files(root):
try:
data = load_manifest(path)
except (OSError, ValueError):
continue
if data.get("run_id") == rid:
return path
return None


@dataclass(frozen=True)
class RestoreOperation:
"""One file to restore from a manifest backup_path."""

backup_path: Path
target_path: Path
src: Path | None
ok: bool
skip_reason: str | None = None


def plan_restore_operations(data: dict[str, object]) -> list[RestoreOperation]:
"""Build restore operations from manifest entries (only rows with backup_path)."""
ops: list[RestoreOperation] = []
entries = data.get("entries", [])
if not isinstance(entries, list):
return ops
for entry in entries:
if not isinstance(entry, dict):
continue
raw_backup = entry.get("backup_path")
if not raw_backup:
continue
backup_path = Path(str(raw_backup))
dst_raw = entry.get("dst") or entry.get("src")
if not dst_raw:
ops.append(
RestoreOperation(
backup_path=backup_path,
target_path=backup_path,
src=None,
ok=bool(entry.get("ok")),
skip_reason="missing dst/src in manifest entry",
)
)
continue
target = Path(str(dst_raw))
src = Path(str(entry["src"])) if entry.get("src") else None
ok = bool(entry.get("ok"))
skip: str | None = None
if not ok:
skip = "job did not succeed"
elif not backup_path.is_file():
skip = f"backup missing: {backup_path}"
ops.append(
RestoreOperation(
backup_path=backup_path,
target_path=target,
src=src,
ok=ok,
skip_reason=skip,
)
)
return ops


def restore_from_manifest(
manifest_path: Path,
*,
dry_run: bool = False,
) -> tuple[int, int, list[str], str | None]:
"""Restore all ``backup_path`` entries in one manifest.

Returns ``(applied, skipped, messages, error)`` where ``error`` is set when
the manifest cannot be used (missing file, no backup entries).
"""
if not manifest_path.is_file():
return 0, 0, [], f"manifest not found: {manifest_path}"
try:
data = load_manifest(manifest_path)
except (OSError, ValueError) as e:
return 0, 0, [], f"invalid manifest: {e}"
ops = plan_restore_operations(data)
if not ops:
return 0, 0, [], "no backup_path entries in manifest"
applied, skipped, messages = apply_restore_operations(ops, dry_run=dry_run)
return applied, skipped, messages, None


def apply_restore_operations(
ops: list[RestoreOperation],
*,
dry_run: bool = True,
) -> tuple[int, int, list[str]]:
"""Copy backup_path → target_path. Returns (applied, skipped, messages)."""
applied = 0
skipped = 0
messages: list[str] = []
for op in ops:
if op.skip_reason:
skipped += 1
messages.append(f"SKIP {op.target_path}: {op.skip_reason}")
continue
if dry_run:
messages.append(f"WOULD RESTORE {op.backup_path} -> {op.target_path}")
applied += 1
continue
try:
op.target_path.parent.mkdir(parents=True, exist_ok=True)
shutil.copy2(op.backup_path, op.target_path)
except OSError as e:
skipped += 1
messages.append(f"FAILED {op.target_path}: {e}")
continue
applied += 1
messages.append(f"RESTORED {op.target_path} <- {op.backup_path}")
return applied, skipped, messages


def write_batch_checkpoint(
*,
run_id: str,
jobs: list[Job],
base: Path | None = None,
checkpoint_dir: Path | None = None,
) -> Path:
"""Copy source files to ``~/.mda/checkpoints/<run_id>/`` before a batch run."""
root = (checkpoint_dir or default_checkpoint_dir()).expanduser() / run_id
root.mkdir(parents=True, exist_ok=True)
base_resolved = base.resolve() if base is not None else None
for job in jobs:
src = job.src.resolve()
if base_resolved is not None:
try:
rel = src.relative_to(base_resolved)
except ValueError:
rel = Path(src.name)
else:
rel = Path(src.name)
dest = root / rel
dest.parent.mkdir(parents=True, exist_ok=True)
shutil.copy2(src, dest)
return root
Loading