Repository navigation
chore: implement pre-commit hooks with security focus - #63
Conversation
Adds .gitattributes file to enforce LF line endings across all text files. This must be done first to prevent conflicts with pre-commit's mixed-line-ending hook and Git's autocrlf behavior. Part of pre-commit hooks implementation (based on data-platform PR #197).
Implements comprehensive pre-commit hooks for the scraper repository with emphasis on security for web scraping operations. Hooks added (5 phases): - File hygiene: whitespace, line endings, YAML/JSON validation - Security: detect-secrets, bandit (SSRF, injection risks) - Custom: site_urls.yaml sync, SSRF allowlist validation - Python: Ruff linter + formatter - Type checking: mypy (permissive mode, core modules only) Changes: - Add .pre-commit-config.yaml with 15+ hooks - Add .bandit config (skip legitimate subprocess/SQL patterns) - Add .secrets.baseline (reviewed for false positives) - Add scripts/validate_site_urls.py (schema validation) - Add scripts/validate_ssrf_allowlist.py (SSRF coverage) - Wire tests/unit/test_config_sync.py as pre-commit hook - Add pre-commit, mypy, bandit to dev dependencies - Add mypy config (permissive, models + config only) - Add per-file-ignores for legacy code (tests, scrapers, api) - Format all Python files with Ruff (automated changes) - Add # nosec comments to legitimate code patterns - Update README.md with Development Setup section Performance: ~7-9s for typical commits Based on successful data-platform implementation (PR #197). Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Fixes critical and high-priority issues discovered in code review: ## Critical Fixes - Fix Python 3.14 breaking change: elt.s → elt.value in AST parsing - Add support for ast.List and ast.Set in _ALLOWED_URL_PREFIXES extraction - Document setup prerequisites in README (poetry install before pre-commit) ## High Priority Fixes - Move Bandit B105 skip from global to per-file-ignores (via inline # nosec) - Add # nosec B105 inline comments for password fallbacks - Document inverted prefix logic in SSRF allowlist validation - Clarify B608 skip justification (SQLAlchemy parameterized queries) ## Medium Priority Fixes - Regenerate .secrets.baseline to sync with current codebase - Document www.gov.br subdomain limitation in validate_site_urls.py ## Details **validate_ssrf_allowlist.py:** - Line 22: isinstance now accepts (ast.Tuple, ast.List, ast.Set) - Line 23: elt.s → elt.value (forward-compatible with Python 3.14) - Lines 74-79: Added comment explaining inverted prefix validation logic **.bandit:** - Removed B105 skip (now handled via inline # nosec comments) - Enhanced B608 skip comment with detailed justification **postgres_manager.py:** - Lines 100, 102, 106: Added # nosec B105 inline comments **README.md:** - Added warning about setup order (poetry install before pre-commit) - Enhanced documentation with numbered steps - Added note about "poetry: command not found" error **validate_site_urls.py:** - Added comment documenting www.gov.br subdomain limitation **.secrets.baseline:** - Regenerated to sync with current codebase ## Testing - ✅ poetry run python scripts/validate_ssrf_allowlist.py - ✅ poetry run python scripts/validate_site_urls.py - ✅ poetry run pre-commit run validate-ssrf-allowlist --all-files - ✅ poetry run pre-commit run validate-site-urls --all-files
Correções Implementadas após Análise Crítica da RevisãoImplementei correções para os problemas validados na revisão de código. A análise detalhada está documentada em:
🔍 MetodologiaInvestiguei o código real com 3 agentes Explore para validar cada ponto da revisão. Descobri:
✅ Correções Implementadas (Commit b90029f)🔴 CRÍTICAS (bloqueavam merge)
🟡 ALTAS (devem ser corrigidas)
🟢 MÉDIAS (recomendadas)
❌ Falsos Positivos Refutados1. Bandit B608 SQL injection skip perigosoRevisão afirmou: Skip global pode mascarar injeções reais. Análise real:
2.
|
| Categoria | Quantidade |
|---|---|
| Problemas reais confirmados | 5 |
| Falsos positivos refutados | 5 |
| Novos problemas descobertos | 2 |
| Arquivos modificados | 6 |
| Linhas adicionadas | 25 |
| Linhas removidas | 17 |
Impacto:
- ✅ Previne breaking change em Python 3.14
- ✅ Melhora experiência de primeiro uso (onboarding)
- ✅ Scope de segurança mais preciso (inline # nosec)
- ✅ Documentação de lógica não-óbvia
- ✅ Nenhum impacto negativo - mudanças são defensivas
Status: ✅ Pronto para merge após estas correções.
Objetivo
Implementar pre-commit hooks abrangentes no repositório scraper, com foco em segurança para operações de web scraping.
Hooks Implementados (5 fases)
Fase 1: File Hygiene (< 1s)
trailing-whitespace,end-of-file-fixer,mixed-line-endingcheck-yaml,check-json,check-tomlcheck-merge-conflict,check-added-large-filesFase 2: Security (2-4s) — CRÍTICO
detect-secrets: Previne vazamento de credenciais (DATABASE_URL, Pub/Sub tokens, Telegram bot tokens)bandit: Detecta SSRF, injection, insecure deserialization (web scraping = alto risco)Fase 3: Python Quality (3-5s)
ruff-check: Linter (substitui Black/isort/flake8)ruff-format: Formatter (line-length 100)Fase 4: Type Checking (1-2s)
mypy: Modo permissivo, habilitado apenas paramodels/,config.pyFase 5: Custom Validators (< 2s) — EXCLUSIVOS DO SCRAPER
validate-site-urls: Valida schema YAML (scraper_type, URL format)validate-ssrf-allowlist: Valida cobertura de domínios no allowlist SSRFtest-config-sync: Valida sync entre src/ e dags/ site_urls.yaml (155 DAGs dependem disso)Arquivos Criados
.pre-commit-config.yaml: 15+ hooks em 5 fases.bandit: Config focada em riscos de web scraping (skip subprocess/SQL legítimos).secrets.baseline: Baseline de false positives (8 arquivos, revisado manualmente).gitattributes: Enforce LF line endings (commitado separadamente)scripts/validate_site_urls.py: Validador schema YAMLscripts/validate_ssrf_allowlist.py: Validador cobertura SSRFArquivos Modificados
pyproject.toml: Adiciona pre-commit, mypy, bandit + configsREADME.md: Seção "Development Setup" com instruçõesPerformance
Teste Local
Baseado em
Implementação bem-sucedida em data-platform (PR #197).
Aprendizados aplicados:
Impacto
✅ Security: Múltiplas camadas de proteção (secrets, SSRF, allowlist)
✅ Quality: Formatação e linting automáticos
✅ Reliability: Validações que previnem quebra de 155 DAGs Airflow
✅ DX: Feedback imediato (10s local vs 5min no CI)
🎯 Trade-off consciente: 7-9s/commit vs potencial de incident (secret leak, SSRF exploit, DAGs quebradas)