Skip to content

fix(engine): stop fabricating acceptance and funnel on degraded reads - #289

Merged
cpenaforte-clickbus merged 22 commits into
fix/pr-read-degradation-org-reportfrom
fix/pr-read-degradation-accuracy
Oct 8, 2026
Merged

cpenaforte-clickbus merged 22 commits into
fix/pr-read-degradation-org-reportfrom
fix/pr-read-degradation-accuracy

Conversation

@cpenaforte-clickbus

@cpenaforte-clickbus cpenaforte-clickbus commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Contexto e Motivação

Uma varredura antes do merge da série (#280 a #285), com o engine rodando sobre dados reais do GitHub e falhas simuladas em cada passo da leitura de PRs, mostrou que ainda saíam números inventados quando o enriquecimento falhava. Com a passada de GraphQL derrubada, a acceptance do repositório afirmava 0 commits em PR e 0% de PR rate, contra 30 commits e 33% no run limpo. E, uma vez sem acceptance, o funil de origem caía no valor padrão e mostrava 100% "em PR". Valores nulos no GraphQL ainda derrubavam a leitura inteira para fetch, e o gh api -F convertia tipos e lia arquivo a partir de valores das variáveis.

Resolver agora porque são exatamente o tipo de número que a #236 existe para não publicar, e a série ainda não entrou na main.

Refs #236

Mudanças

  • Com o enriquecimento degradado, acceptance_by_origin e acceptance_by_tool são omitidos (casar commit com PR depende das commit refs), e também os campos de PR aberto parado, cuja última atividade usa os pushes de commit.
  • O funil de origem sai inteiro sempre que a leitura degradou e a acceptance está ausente, em vez de supor que todo commit entrou por PR. Ausência por desenho — sem gh ou sem remote — mantém o comportamento de antes.
  • O parser do enriquecimento trata repository, pullRequests ou nodes nulos como página incompleta e ignora nós e commits nulos, em vez de levantar exceção.
  • Todas as variáveis de texto do GraphQL vão com gh api -f: com -F, um repositório de nome só numérico virava número e quebrava o enriquecimento em todo run, e um valor começando com @ era lido como caminho de arquivo.
  • A linha de console diz quando a leitura veio incompleta, distingue falha da listagem de falha de passo secundário e conta PRs de todos os estados.
  • Um helper único monta os grupos de acceptance, no lugar de dois blocos duplicados (commit de refatoração separado, sem mudança de comportamento).

Plano de Teste

A verificação roda fora da máquina de quem construiu: o job cli do CI (RFC 0029, PM-4).

Impacto e Risco

  • Áreas afetadas: agregador, funil de origem, leitor de PRs do GitHub, console e docs/METRICS.md.
  • Compatibilidade: um run sem degradação gera o mesmo metrics.json e o mesmo relatório, byte a byte. Runs com o enriquecimento degradado deixam de trazer acceptance, staleness e funil.
  • Performance: nenhuma mudança.
  • Segurança: endurece a chamada ao gh: nenhum valor vindo da resposta do GitHub pode mais ser interpretado como caminho de arquivo.
  • Acessibilidade e custo AWS: N/A — sem UI e sem infraestrutura.

Sobre o tamanho (RFC 0028 §2.4): 482 linhas, faixa 251–500. São correções da mesma varredura, cada uma num commit próprio; a parte de degradação por estado foi separada no PR seguinte justamente para caber.

Plano de Rollback

git revert do merge. Volta o comportamento da #285; nada a migrar.

Referências

Autoria Assistida

  • Agente: Claude Code — Claude Sonnet 5.5 e Claude Opus 5.5 nos commits; Claude Sonnet 5.5 nas revisões por task; Claude Opus 5.5 na varredura, na orquestração e nas revisões finais
  • Escopo da assistência: varredura de bugs e validação com dados reais, implementação guiada por testes; a decisão de omitir em vez de estimar foi do autor
  • Revisão humana: pendente — o autor revisa antes do merge

🤖 Generated with Claude Code

cpenaforte-clickbus and others added 20 commits October 7, 2026 20:01
Acceptance matches commits to PRs through commit_refs, and open-PR
staleness reads the last commit push from them. Both come from the
enrichment pass. With it failed a real run reported 0 commits in PRs
against 30 clean, and underestimated last activity, inflating stalled
PRs. Omit acceptance_by_origin/by_tool whole and the four staleness
fields; open-PR count and ages stay. Documented in METRICS.md.

Refs #236

Assisted-by: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A null data.repository, or a node with commits: null, raised
AttributeError, which dropped the whole PR read to the fetch step. A
non-dict at those points is now a missing page (read incomplete, what
was already read is kept) or a node with no commits.

Refs #236

Assisted-by: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
gh api -F reads a file when the value starts with "@", so a cursor
beginning with that character would be misread. -f sends plain text.

Refs #236

Assisted-by: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The console said "failed ... continuing without PR data" when only a
secondary pass failed on a window with no PRs, and said nothing when
PRs were found but the read had degraded. Now PRs plus degradation
print the found count with the incomplete steps; no PRs with only
secondary steps degraded say the read was incomplete; basic or fetch
degraded keep the failure message. Clean runs print as before.

Refs #236

Assisted-by: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The by-origin and by-tool blocks repeated the same comprehension plus
an update loop for the review-derived keys. One small helper now
builds both. No behavior change: the key order in each group is
unchanged, so a clean run stays byte-identical.

Refs #236

Assisted-by: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The funnel's "In PR" stage reads acceptance_by_origin and, when it is
absent, assumes every commit is in a PR. With the enrichment failed
that field is now omitted, so the funnel reported 100% In PR and
inflated every later conversion. Return no funnel then (the stages
chain, so dropping one would break its meaning). Repos with no PR data
keep the old default. Also fix two METRICS.md rows that claimed the
review-round and single-pass fields drop on enrichment; only reviews
gates them.

Refs #236

Assisted-by: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The funnel returned nothing only when enrichment failed. When every
state's list failed (basic) or the whole read raised (fetch), there are
no PRs, acceptance is absent, and the "In PR" stage fell back to every
commit being in a PR: 100% In PR in metrics.json and the report. Now
the funnel is omitted whenever the read was degraded and acceptance is
absent. PRs absent by design (no gh, no GitHub remote) mark nothing as
degraded and keep the old default.

Refs #236

Assisted-by: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The enrichment query sent owner and name with `-F`, which lets gh
coerce the value: a repo named 2048 went out as the number 2048 (and
`true`/`null` as a boolean/null), so the `String!` variable failed on
every page and the enrichment always degraded. Every variable of the
query is a string, so all of them now go with `-f`, as the cursor and
states already did.

Refs #236

Assisted-by: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A page with `"nodes": null` raised a TypeError, and a null PR node or
a null commit entry raised an AttributeError; any of them crashed the
whole PR read into `fetch`. The enrichment pass now reads a null list
as empty and skips nodes and commit entries that are not objects,
keeping the rest of the page.

Refs #236

Assisted-by: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The console printed "{count} merged PRs found", but the count is every
PR read in the window: merged, closed and open. Both the clean and the
degraded line now say "{count} PRs found" ("{count} PRs encontrados"
in pt-BR).

Refs #236

Assisted-by: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A page whose `nodes` was null, or not a list at all, was read as an
empty page: the enrichment pass ended "complete" with none of that
page's PRs. It is now an incomplete page, like a null `pullRequests`:
the pass keeps what earlier pages collected and reports enrichment as
failed. Null PR nodes and commit entries inside the list are still
skipped.

Refs #236

Assisted-by: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The enrichment retry waited 2 s on any gh failure, deterministic ones
included: a 401, a 404 or a GraphQL error fails the same way on the
second try, so every run paid the pause per page and per PR state for
nothing. The retry now reads gh's stderr and only tries again on a
transient error (HTTP 5xx, a timeout, a dropped connection).

The reviews pass, which the code documents as prone to 504s, had no
retry at all, while one failure of it now drops review metrics, the
flow block and staleness from the report. It gets the same single
transient-only retry. The basic list stays as it was.

The pause constant becomes _RETRY_DELAY_S, since both passes share it.

Refs #236

Assisted-by: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The PR read fetches merged, closed and open PRs separately, but the
aggregator gated on the union of the three: a reviews or enrichment
failure only for open PRs wiped review, flow, acceptance and merge
strategy, which read merged PRs alone. That threw away good data.

PullRequestFetch now keeps the failed steps per state
(degraded_by_state), the window-cache slice inherits them, and every
aggregate() call in the CLI and the org runner passes them. Review,
flow, acceptance and merge strategy follow the merged state; open-PR
staleness follows the open state; a closed-state failure omits
nothing, since closed PRs only feed counts. Without the per-state map
(a fetch that never reached the states, or no map at all) every state
takes the union, as before.

The origin funnel cannot tell the states apart from the union alone,
so it now skips when enrichment failed and acceptance is absent. The
aggregator always omits acceptance when the merged state's enrichment
failed, so the funnel's "In PR" stage still never falls back to a
made-up "every commit is in a PR" then, and the funnel survives a
failure that hit only open or closed PRs.

pr_enrichment_degraded keeps listing the union, so the JSON contract
does not change, and a clean run's outputs stay identical. The report
caveat and METRICS.md now say the omissions follow the PR state.

Refs #236

Assisted-by: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When `gh pr list` failed for one state while the others were read, the
PR list was not empty, so everything the missing state feeds read as a
real zero: with the merged list gone, acceptance showed 0% for every
group, the activity timeline 0 PRs merged a week, and merge strategy
"unknown" with reliable commit metrics; with any list gone, Flow Load
undercounted the WIP.

A missing merged list now omits what merged PRs feed (PR lifecycle,
review, Flow Efficiency, human coverage, acceptance, merge strategy)
and leaves the timeline's prs_merged unknown; a missing open list
omits every open-PR field; a missing list in any state omits Flow
Load. Without the per-state map the union applies, and `fetch` counts
as every list failed. Clean runs are unchanged.

Refs #236

Assisted-by: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The transient-error check matched anywhere in gh's stderr: "EOF"
matched "thereof", and "timeout" matched any error about a repo named
like timeout-svc, so a deterministic failure there was retried with
the pause on every page. The repo's owner/name, name and owner are
now blanked out of stderr first (only where they stand alone, so a
repo named "out" leaves "timeout" whole), and the patterns match whole
words: HTTP 5xx, timeout/timed out, deadline exceeded, connection
reset/refused, EOF. GitHub's GraphQL "may be the result of a timeout"
error is still retried. Also refresh the enrichment docstring, which
still said a page failed "even after one retry".

Refs #236

Assisted-by: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The nullable-when cells said "`reviews` degraded" or "`enrichment`
degraded" while the prose already gated each field on the PR state
that feeds it; the cells now carry that state, and the ones a failed
list omits say so (acceptance, funnel, timeline, PR lifecycle, Flow
Load, Flow Efficiency, coverage, open PR aging, merge strategy). §15
documents the failed-list rule, defines a stopped enrichment pass as
failed even when earlier pages came back, and says a 4xx or other
GraphQL error is not retried (GitHub's "may be the result of a
timeout" error is).

Refs #236

Assisted-by: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The report caveat listed flow, review and in-PR metrics as what a
degraded read may leave missing, but a failed open list, or a failed
enrichment or reviews pass for open PRs, omits open-PR fields too. The
caveat now names them ("open-PR metrics" / "de PRs abertos"); its
opening sentence is unchanged.

Refs #236

Assisted-by: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
§24 listed a failed list only in the nullable-when cell. It now says
the whole series goes when `gh pr list` failed for any state (or
`fetch`), author_concurrency included, and why: the missing list would
undercount the WIP in every bucket it overlaps, and keeping only the
commit-based author count would emit buckets in a shape the platform
does not read.

Refs #236

Assisted-by: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Several notes still described the older PR read. The aggregate()
docstring called `prs` merged PRs, but they come in all three states.
METRICS.md §10 and the funnel module said Iris only fetches merged PRs;
it reads merged, closed and open PRs, and the limitation is that
acceptance counts only commits in merged PRs. §10 now says a failure
confined to closed or open PRs leaves acceptance and the funnel where a
clean run would have them, without implying they are always present.
§15 now says that with merged enrichment failed, Flow Efficiency is
computed over the PRs whose commit refs came back, so it is skewed
rather than missing.

Refs #236

Assisted-by: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A failed merged list omits the PR lifecycle fields (counts, time to
merge, cycle time), and a partial read skews them, but the report
caveat did not list them among what may be missing or skewed. It now
does, in both languages; its opening sentence is unchanged.

Refs #236

Assisted-by: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cpenaforte-clickbus cpenaforte-clickbus added type: tech-debt Código sub-ótimo conhecido, workaround, ou cleanup pendente ai-assisted PR gerado total ou parcialmente com auxílio de agente de IA (RFC 0028 §2.2.7) labels Oct 8, 2026
@vercel

vercel Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
clickbus-iris Ready Ready Preview Oct 8, 2026 1:44pm UTC

Request Review

…-list

fix(engine): omit what a failed PR list feeds, per state
fix(engine): gate degradation per PR state and retry transient errors
@cpenaforte-clickbus
cpenaforte-clickbus merged commit 0f5d438 into fix/pr-read-degradation-org-report Oct 8, 2026
4 of 5 checks passed
@cpenaforte-clickbus
cpenaforte-clickbus deleted the fix/pr-read-degradation-accuracy branch October 8, 2026 14:37

This branch was successfully deployed

1 active deployment
Preview — 3a9cdfde Deployed Oct 8, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-assisted PR gerado total ou parcialmente com auxílio de agente de IA (RFC 0028 §2.2.7) type: tech-debt Código sub-ótimo conhecido, workaround, ou cleanup pendente

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant