Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,14 @@ adheres to [Semantic Versioning](https://semver.org/).

### Added

- **Public MCP `cvm_fund` quotes/structure parity (CVM only).** `dataset=daily`
accepts `start`/`end` or `months` up to 12, stitches a single-class RCVM 175
continuation onto legacy 555 INF_DIARIO when the files share sibling CNPJs,
and returns `served[].nicename` / class / subclass for the series actually
returned. Multiple subclasses stay grouped (`pick_required`) — the tool does
not pick a FIDC série. `dataset=catalog` `q` also matches classe and
subclasse names. Worker tool description maps Mais Retorno data tools onto
`cvm_fund` datasets. Calc/analytics tools stay out. Rate limits unchanged.
- **Public MCP `cvm_fund` (Worker) + RCVM 175 cadastro.** The public
Worker at `/mcp` now resolves an open-ended fund by CNPJ (or name) against
official CVM `registro_fundo_classe.zip` and returns INF_DIARIO cota/PL/cotistas
Expand Down
18 changes: 15 additions & 3 deletions docs/DEPLOY_WORKERS_MCP.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,15 +20,27 @@ registry FTS5, code mode): `pip install openfindata` ou FastAPI interno.
`cvm_fund` no Worker:

- `dataset=catalog` + `cnpj` ou `q` — cadastro oficial RCVM 175
(`registro_fundo_classe.zip`: fundo + classe + subclasse). `cad_fi.csv`
não lista fundos já adaptados à Resolução 175.
(`registro_fundo_classe.zip`: fundo + classe + subclasse). `q` casa
nome de fundo, classe e subclasse. `cad_fi.csv` não lista fundos já
adaptados à Resolução 175.
- `dataset=daily` + `cnpj` — série de cotas INF_DIARIO. Sem `year`/`month`
usa o mês mais recente no diretório CVM; `months` (1–3) olha para trás.
usa o mês mais recente; `months` (1–12) ou `start`/`end` (YYYY-MM-DD,
no máximo 12 meses; janelas maiores pedem nova chamada). Classe única
adaptada da 555 costura o CNPJ legado quando o arquivo CVM permite.
`served[]` traz `nicename` / classe / subclasse efetivamente devolvidos;
várias séries no mesmo CNPJ vêm agrupadas (`pick_required`) — não
escolher FIDC sênior vs subordinada pelo usuário. ZIP anual HIST da CVM
passa de 32 MB e não entra no isolate.
- `dataset=periods` — stamps YYYYMM publicados (`product=CDA` ou `INF_DIARIO`).
- `dataset=holdings` + `cnpj` — CDA (carteira). Sem `year`/`month` usa o
CDA mais recente. Scan em stream por CNPJ; linhas `CONFID` são sigilo,
não carteira aberta completa. Não usa Mais Retorno.

Mapeamento das tools de **dados** da Mais Retorno (cálculo/analytics fora):
`search_assets` / `get_asset_info` / `list_fund_structure` /
`get_fund_class_subclass` → `catalog`; `get_quotes` → `daily`;
`get_available_wallets` → `periods`; `get_wallet_detail` → `holdings`.

## Deploy

```bash
Expand Down
18 changes: 14 additions & 4 deletions docs/MCP_SURFACE.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
> and [`docs/DEPLOY_WORKERS_MCP.md`](DEPLOY_WORKERS_MCP.md). Public `/mcp` is
> 60 req/60s per IP with a 20/10s burst; overflow is 429 + Retry-After (no queue,
> no code mode, no API key). Worker tools: the 9 JSON macro sources plus
> `cvm_fund` (RCVM 175 cadastro + INF_DIARIO + CDA holdings/periods).
> `cvm_fund` (RCVM 175 cadastro + INF_DIARIO history ≤12 months + CDA holdings/periods).
> Lâmina/perfil stay on the internal FastAPI catalog below.

## Problem
Expand Down Expand Up @@ -86,7 +86,7 @@ findata_run_code (code mode, opt-in)
| `bcb_ptax` | `/ptax/usd`, `/ptax/usd/period`, `/ptax/{currency}` | `start`+`end` → period |
| `bcb_focus` | `/focus/{indicators,annual,monthly,selic,top5}` | `horizon`, `panel`, `indicator` |
| `cvm_company` | companies search/list, `fca/*`, `ipe` | `dataset=search\|list\|fca_*\|filings` |
| `cvm_fund` | `funds`, `funds/cadastro`, `funds/{daily,holdings,lamina,profile,periods}`, returns | `dataset`; `cnpj`/`q` → RCVM 175; omit year/month → latest CDA/INF_DIARIO |
| `cvm_fund` | `funds`, `funds/cadastro`, `funds/{daily,holdings,lamina,profile,periods}`, returns | `dataset`; `cnpj`/`q` → RCVM 175 (classe/subclasse names too); `daily` `start`/`end` or `months`≤12; omit year/month → latest CDA/INF_DIARIO |
| `cvm_structured_fund` | `funds/{fii,fidc,fip}/*` | `kind` + `dataset` |
| `b3_index` | index portfolio + monthly + list | `dataset`, omit `symbol` to list |
| `tesouro_bonds` | bonds list/search/history | `dataset` |
Expand Down Expand Up @@ -120,8 +120,18 @@ local/agent use. A production deployment should run it in a real sandbox

- `registry_lookup(q="PETR4")` → PETROBRAS, CNPJ `33.000.167/0001-01`, `[PETR3, PETR4]` (offline).
- `bcb_ptax(start=2024-01-02, end=2024-01-05)` → daily PTAX USD series (the handoff's headline flow).
- `cvm_fund(dataset=catalog, cnpj="38.729.027/0001-92")` → cadastro RCVM 175 (classe, condomínio, PL).
- `cvm_fund(dataset=daily, cnpj="38729027000192", year=2026, month=8)` → INF_DIARIO (cota/PL/cotistas).
- `cvm_fund(dataset=catalog, cnpj="38.729.027/0001-92")` → cadastro RCVM 175 (classe, condomínio, PL, subclasses).
- `cvm_fund(dataset=daily, cnpj="38729027000192", start="2025-09-01", end="2026-08-31")` → INF_DIARIO history (`served[].nicename`).
- `cvm_fund(dataset=periods, product="CDA")` → YYYYMM stamps + `latest`.
- `cvm_fund(dataset=holdings, cnpj="38729027000192")` → latest CDA carteira (CONFID = sigilo).

Mais Retorno **data** tools (CVM/official only; calc tools are out of scope):

| Mais Retorno | `cvm_fund` |
|---|---|
| `search_assets` / `get_asset_info` / `get_fund_class_subclass` | `dataset=catalog` (`q` or `cnpj`) |
| `list_fund_structure` | `catalog.classes[].subclasses[]` — do not pick a FIDC série |
| `get_quotes` | `dataset=daily` (`start`/`end` or `months`≤12; report `served[]`) |
| `get_available_wallets` | `dataset=periods` `product=CDA` |
| `get_wallet_detail` | `dataset=holdings` |
- `findata_run_code("import findata; ...")` → runs in the sandbox, returns captured stdout.
2 changes: 1 addition & 1 deletion docs/agents/orientation.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ mortos, fronteira atual, openfindata, findata.
| Rede nos unit tests? | Proibido. `respx` nos unitários; live só `@pytest.mark.integration` | `AGENTS.md`, CI nightly |
| Credenciais no repo? | Nunca. Fontes públicas preferidas; BdD usa billing project do operador via env | `AGENTS.md`, `docs/SOURCES_WITH_AUTH.md` |
| MCP: 1:1 com REST ou curado? | Catálogo curado em `mcp_app` (~25 tools); REST intacto | `docs/MCP_SURFACE.md` |
| MCP público vs interno? | Worker `openfindata.com.br/mcp` (macro JSON + `cvm_fund` catalog/daily/holdings/periods); FastAPI/Tailscale tem o catálogo completo | `docs/DEPLOY_WORKERS_MCP.md` |
| MCP público vs interno? | Worker `openfindata.com.br/mcp` (macro JSON + `cvm_fund` catalog/daily≤12m/holdings/periods); FastAPI/Tailscale tem o catálogo completo | `docs/DEPLOY_WORKERS_MCP.md` |
| Code mode no MCP? | Opt-in via `FINDATA_MCP_CODE_MODE=1`; off por default | `docs/MCP_SURFACE.md`, `mcp_app.py` |
| Charts: quais deps de plot? | Não adicionar matplotlib/pandas/plotly etc. só para gráfico | `AGENTS.md`, `docs/CHART_STANDARDS.md` |
| Publicar no PyPI? | Só com aprovação humana explícita | `AGENTS.md` |
Expand Down
211 changes: 195 additions & 16 deletions src/findata/api/mcp_app.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@
from datetime import date
from typing import Any, Literal

import httpx
from fastapi import APIRouter, FastAPI, HTTPException, Query, Request
from fastapi.responses import JSONResponse
from pydantic import BaseModel, Field
Expand All @@ -49,6 +50,7 @@
from findata.sources.bcb import focus, ptax, sgs
from findata.sources.cvm import (
companies,
continuation_class_cnpj,
fca,
fidc,
fii,
Expand All @@ -62,7 +64,10 @@
latest_period,
list_periods,
profile,
quote_served_label,
related_quote_cnpjs,
)
from findata.sources.cvm.parser import cnpj_digits
from findata.sources.ibge import indicators
from findata.sources.ipea import series as ipea_series
from findata.sources.openfinance import directory as of_dir
Expand All @@ -76,6 +81,7 @@
_RGF_MAX_PERIOD = 3 # RGF quadrimestre runs 1..3
_DAILY_MONTHS_MAX = 12
_YYYYMM_LEN = 6
_STITCH_SIBLING_CNPJS = 2 # requested + one fundo/classe pair


def _stamp_to_year_month(stamp: str) -> tuple[int, int]:
Expand Down Expand Up @@ -106,6 +112,170 @@ async def _resolve_cvm_month(year: int | None, month: int | None, product: str)
return _stamp_to_year_month(latest)


def _parse_iso_month(value: str) -> tuple[int, int]:
try:
parsed = date.fromisoformat(value)
except ValueError as exc:
raise HTTPException(400, "start/end must be YYYY-MM-DD") from exc
return parsed.year, parsed.month


def _stamps_inclusive(start: tuple[int, int], end: tuple[int, int]) -> list[tuple[int, int]]:
if start > end:
raise HTTPException(400, "`start` must be on or before `end`")
stamps: list[tuple[int, int]] = []
year, month = start
while (year, month) <= end:
stamps.append((year, month))
if len(stamps) > _DAILY_MONTHS_MAX:
raise HTTPException(
400,
f"daily window exceeds {_DAILY_MONTHS_MAX} months; page `start`/`end`",
)
year, month = _add_months(year, month, 1)
return stamps


async def _daily_month_stamps(
year: int | None,
month: int | None,
months: int,
start: str | None,
end: str | None,
) -> list[tuple[int, int]]:
if start or end:
if not start or not end:
raise HTTPException(400, "pass both `start` and `end`, or omit both")
return _stamps_inclusive(_parse_iso_month(start), _parse_iso_month(end))
resolved_year, resolved_month = await _resolve_cvm_month(year, month, "INF_DIARIO")
return _lookback_months(resolved_year, resolved_month, months)


def _row_field(row: Any, name: str, default: Any = "") -> Any:
return getattr(row, name, default) if not isinstance(row, dict) else row.get(name, default)


def _dedupe_daily_rows(series: list[Any], prefer_cnpj: str) -> list[Any]:
chosen: dict[tuple[str, str], Any] = {}
for row in series:
key = (str(_row_field(row, "dt_comptc")), str(_row_field(row, "id_subclasse")))
prev = chosen.get(key)
if prev is None or cnpj_digits(str(_row_field(row, "cnpj"))) == prefer_cnpj:
chosen[key] = row
return [chosen[key] for key in sorted(chosen)]


def _in_date_range(dt_comptc: str, start: str | None, end: str | None) -> bool:
if start and dt_comptc < start:
return False
return not (end and dt_comptc > end)


def _group_daily_series(
series: list[Any],
cadastro: list[Any],
compact: bool,
requested: str,
needles: list[str],
) -> dict[str, Any]:
target = continuation_class_cnpj(cadastro, requested) or ""
if not target and cadastro and len(needles) == _STITCH_SIBLING_CNPJS:
target = requested
groups: dict[tuple[str, str], list[Any]] = {}
for row in series:
row_digits = cnpj_digits(str(_row_field(row, "cnpj")))
sub = str(_row_field(row, "id_subclasse"))
group_cnpj = target if target and row_digits in needles else row_digits
groups.setdefault((group_cnpj, sub), []).append(row)
served: list[dict[str, Any]] = []
for (cnpj, sub), rows in groups.items():
label = quote_served_label(cadastro, cnpj, sub)
item: dict[str, Any] = {
"cnpj": cnpj,
"id_subclasse": sub,
**label,
"points": len(rows),
}
if compact:
item["dates"] = [str(_row_field(row, "dt_comptc")) for row in rows]
item["vl_quota"] = [_row_field(row, "vl_quota", 0) for row in rows]
served.append(item)
pick_required = len(served) > 1
note = (
"Multiple INF_DIARIO series for this CNPJ (classes/subclasses). "
"They are different investments — pass `id_subclasse` or the class CNPJ; do not pick."
if pick_required
else "Report served[].nicename / class / subclass actually returned."
)
payload: dict[str, Any] = {
"source": "cvm_inf_diario",
"pick_required": pick_required,
"served": served,
"note": note,
}
if not compact:
payload["series"] = series
return payload


async def _cvm_fund_quotes(
cnpj: str,
year: int | None,
month: int | None,
months: int,
start: str | None,
end: str | None,
id_subclasse: str | None,
compact: bool,
limit: int,
) -> dict[str, Any]:
stamps = await _daily_month_stamps(year, month, months, start, end)
try:
cadastro = await get_fund_cadastro(cnpj=cnpj, limit=20)
except httpx.HTTPError:
cadastro = []
needles = related_quote_cnpjs(cadastro, cnpj) if cadastro else [cnpj]
prefer = continuation_class_cnpj(cadastro, cnpj) or cnpj_digits(cnpj)
series: list[Any] = []
missing: list[str] = []
truncated = False
for stamp_year, stamp_month in stamps:
try:
chunk = await funds.get_fund_daily(stamp_year, stamp_month, ",".join(needles))
except httpx.HTTPStatusError as exc:
if exc.response.status_code == httpx.codes.NOT_FOUND:
missing.append(f"{stamp_year}{stamp_month:02d}")
continue
raise
kept = [
row
for row in chunk
if (not id_subclasse or str(_row_field(row, "id_subclasse")) == id_subclasse)
and _in_date_range(str(_row_field(row, "dt_comptc")), start, end)
]
series.extend(kept)
if len(series) >= limit:
series = series[:limit]
truncated = True
break
series = _dedupe_daily_rows(series, prefer)
payload = _group_daily_series(series, cadastro, compact, cnpj_digits(cnpj), needles)
payload.update(
{
"cnpj": cnpj_digits(cnpj),
"from": f"{stamps[0][0]}{stamps[0][1]:02d}" if stamps else None,
"to": f"{stamps[-1][0]}{stamps[-1][1]:02d}" if stamps else None,
"start": start,
"end": end,
"months": len(stamps),
"needles": needles,
"missing": missing,
"truncated": truncated,
}
)
return payload


# ── Registry: the entry point ─────────────────────────────────────


Expand Down Expand Up @@ -354,6 +524,18 @@ async def cvm_fund(
months: int = Query(
1, ge=1, le=_DAILY_MONTHS_MAX, description="daily: lookback months including the end month"
),
start: str | None = Query(
None, description="daily: YYYY-MM-DD inclusive start (use with end; max 12 months)"
),
end: str | None = Query(
None, description="daily: YYYY-MM-DD inclusive end (use with start; max 12 months)"
),
id_subclasse: str | None = Query(
None, description="daily: keep only this CVM subclass/série; do not guess for the user"
),
compact: bool = Query(
False, description="daily: served[].dates + served[].vl_quota instead of series[]"
),
horizon: Literal["monthly", "yearly"] = Query(
"monthly", description="returns granularity (dataset=returns)"
),
Expand All @@ -372,13 +554,14 @@ async def cvm_fund(
] = Query("INF_DIARIO", description="periods: which CVM document set to list stamps for"),
limit: int = Query(500, ge=1, le=5000),
) -> Any:
"""Open funds in one tool. ``catalog`` with ``cnpj`` or ``q`` reads the official
RCVM 175 registro (fundo+classe+subclasse). Bare ``catalog`` still pages the
legacy ``cad_fi.csv`` (non-adapted funds only). ``periods`` lists YYYYMM
stamps. ``daily`` is INF_DIARIO (cota/PL/cotistas); omit ``year``/``month``
for the latest published month, or pass ``months`` to look back. CDA
``holdings`` is a separate monthly delayed feed — omit ``year``/``month``
for the latest CDA. CONFID rows are sigilo, not a complete open book.
"""Open funds in one tool. Mais Retorno data mapping (CVM/official only; no
calc tools): ``catalog`` = search_assets + get_asset_info +
list_fund_structure + get_fund_class_subclass (``q`` matches fundo,
classe and subclasse names); ``daily`` = get_quotes (``start``/``end`` or
``months``≤12; ``served[].nicename`` is what was returned; never pick a
FIDC série); ``periods``+``holdings`` = get_available_wallets +
get_wallet_detail. Bare ``catalog`` still pages legacy ``cad_fi.csv``.
CDA ``holdings`` CONFID is sigilo, not a complete open book.
"""
if dataset == "catalog":
if cnpj or q:
Expand All @@ -393,15 +576,11 @@ async def cvm_fund(
block_list = [b.strip() for b in blocks.split(",") if b.strip()] if blocks else None
return await holdings.get_fund_holdings(cnpj, year, month, block_list)
if dataset == "daily":
year, month = await _resolve_cvm_month(year, month, "INF_DIARIO")
if months == 1:
return (await funds.get_fund_daily(year, month, cnpj))[:limit]
series: list[Any] = []
for stamp_year, stamp_month in _lookback_months(year, month, months):
series.extend(await funds.get_fund_daily(stamp_year, stamp_month, cnpj))
if len(series) >= limit:
break
return series[:limit]
if not cnpj:
raise HTTPException(400, "dataset=daily requires `cnpj`")
return await _cvm_fund_quotes(
cnpj, year, month, months, start, end, id_subclasse, compact, limit
)
if year is None:
raise HTTPException(400, f"dataset={dataset} requires `year`")
if month is None:
Expand Down
6 changes: 6 additions & 0 deletions src/findata/sources/cvm/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,10 @@
FundCadastro,
FundClasse,
FundSubclass,
continuation_class_cnpj,
get_fund_cadastro,
quote_served_label,
related_quote_cnpjs,
)
from findata.sources.cvm.companies import Company, get_companies, search_company
from findata.sources.cvm.fca import (
Expand Down Expand Up @@ -104,6 +107,7 @@
"FundSubclass",
"IPEDocument",
"StatementType",
"continuation_class_cnpj",
"get_companies",
"get_dfp",
"get_fca_dri",
Expand All @@ -128,5 +132,7 @@
"latest_period",
"list_files",
"list_periods",
"quote_served_label",
"related_quote_cnpjs",
"search_company",
]
Loading
Loading