feat(brokers): verified registry-sourced list, custom list config, audit auto-prune - #55
Merged
Conversation
…dit auto-prune Addresses #54 (outdated / unverified broker email addresses) in three parts: - data/brokers-verified.yaml: a smaller second embedded list (48 entries) of companies confirmed to be data brokers (US: state-registry-checkable; EU: credit / data-intelligence bureaus) with a working removal contact. Selected with `send --list verified` or options.broker_list: verified. The full list stays the default. broker.LoadList resolves: --brokers flag -> options.broker_file -> verified -> ~/.eraser/brokers.yaml -> embedded full. - options.broker_file: point the send-family commands at your own list from config (the CLI already had --brokers). - audit-brokers --fix: clears the email of every broker with a dead mail domain (MX + host lookup both fail), keeping the row and adding a dated note. The weekly broker-audit workflow now runs --fix and opens a PR when the list changed, instead of just failing red for a maintainer to prune by hand. Out of scope: one-by-one review of individual questionable entries in the full list - that stays manual via docs/broker-replies.md.
…ied list Ran a per-broker web-search sweep over all 762 entries in data/brokers.yaml (legitimacy, operating status, contact currency). Applied only low-risk fixes directly; everything else is a flagged note for manual review, nothing deleted: - 19 entries: on-file contact was a stale/personal address or missing an opt-out URL - updated to the company's own published privacy/DSR contact. - 46 entries: flagged in notes as possibly not a data broker (21), looks defunct (3), a possible duplicate of another entry (2), or uncertain (20) - each with the evidence found, for a human to confirm before any deletion. data/brokers-verified.yaml: removed datajoy-eu (self-admits "not even a data broker", corroborating an existing note) and oracle-america (its ad/data business - BlueKai/Oracle Data Cloud - shut down in 2024); added ussearch's missing opt-out URL. EU coverage in the verified list is down to 4 entries and still needs the registry-sourced expansion this was meant to get. data/brokers.yaml: also fixed background-information-services by hand from a forwarded broker reply - FCRA-exempt consumer reporting agency, no record on file, miscategorized as marketing instead of background-check. Full findings: see the sweep summary shared alongside this commit.
…unt floor Per follow-up review of the sweep's uncertain/duplicate findings: - Merge brightcheck into brigthswipe (same company - PRNewswire confirms BrightSwipe Inc. as the entity behind the BrightCheck product). - Clean up the zenleads note - its contact already routes to Apollo.io directly, no separate Apollo.io entry exists to merge into, so it's kept rather than flagged as a possible duplicate. - Remove the 20 entries the sweep could only mark "uncertain": no corroborating web presence at all, or a real company whose own privacy policy states it doesn't sell personal data (SaaS/consulting vendors processing a client's own data, not brokering third-party consumer data). List count: 762 -> 741. Update the "750+" claim to "700+" in README.md, docs/architecture.md, docs/broker-replies.md, and EU-NOTES.md - the count floor's own note had flagged that a dedup pass below 750 would make it false.
Resolved the sweep's remaining not_a_broker/looks_defunct flags: - Deleted 20 not_a_broker entries: SaaS/consulting/insurance vendors whose own privacy policies explicitly disclaim selling personal data (HubSpot, Ipsos, Greenhouse, Qwoted, Accenture, Deloitte Consulting, PwC Product Sales, UPS Capital, and others) - they process a client's own data rather than brokering third-party consumer data, so an erasure request has nothing to act on regardless of address. - Kept birchwood-credit-services instead of deleting it: it's an FCRA-covered consumer reporting agency, same exemption pattern already established for goodhire/saferent-solutions/background-information-services in this list - email cleared, category corrected marketing -> background-check, note cites the exemption. - Deleted the 3 looks_defunct entries: near-north-america (Near Intelligence filed bankruptcy 2023, already-bounced email), oracle-america (its ad-data business, BlueKai/Oracle Data Cloud, shut down in 2024 - matches the earlier removal from the verified list), veromi (already-bounced email, domain now parked). List count: 741 -> 718 (still comfortably above the "700+" floor). EU-NOTES.md's EU/ad-tech entry list updated to drop datajoy-eu and scope3.
Added 10 real EU data brokers/data-intelligence companies, each verified via their own current GDPR/privacy-request contact (not imported blind): Credit/B2B-intelligence bureaus (financial-b2b): - schufa (SCHUFA, Germany - major consumer credit bureau) - crif (CRIF, Italy - pan-EU credit/business-information bureau) - dun-bradstreet-eu (D&B EU, incl. former Bisnode entities) Address/marketing data brokers: - deutsche-post-direkt (Germany) - az-direct (Germany/Austria, Bertelsmann group - note cites a 2026 noyb complaint about its responsiveness) Ad-tech (marketing): - adform (Denmark), criteo (France), rtb-house (Poland), equativ/Smart AdServer (France), teads (France) data/brokers-verified.yaml: added these 10 plus two already-present EU entries with solid contacts (smartclip, etarget-sk) that hadn't been copied over yet - verified EU count 4 -> 16 (out of 19 EU-region entries in the full list; seawave-media has no email, adikteev is a processor-not-controller dead end, genius-sports is region:global so already covered regardless of filter). List counts: full 718 -> 728, verified 46 -> 58. EU-NOTES.md's EU entry list and the count narrative updated accordingly.
Manual validation pass over the full list (723 entries) and verified list (61 entries): - Removed stale duplicates: infogroup, towerdata (superseded by atdata), gravy-analytics (superseded by unacast), tapad, zabasearch (dup id), us, freshaddress, seawave-media, whitepages-2, intelius-2, epsilon-data-management, free-data-services, environics-analytics. - Fixed trailing-dot-broken opt_out_url values: monevo, skyhook-holding, truth-now. - Refreshed privacy/removal contacts from current site content: equativ, foursquare, peoplesmart, intelius, instantcheckmate, truthfinder, equifax-marketing. - Added region: global consumer-reporting/credit bureaus not previously covered: nosis (Argentina), circulo-de-credito and buro-de-credito-mx (Mexico), datacredito-experian-co (Colombia), cial-dun-bradstreet (LatAm), centrix-nz (New Zealand), metropol-crb (Kenya). - category: requires-id on adform and nosis (both require ID verification before acting on a request). Docs updated to match: EU-NOTES.md, README.md, docs/architecture.md, docs/auditing.md, docs/commands.md.
broker.Select(ids, regions, categories, excluded, excludedCategories) generalizes Filter into arbitrary broker selection; Filter now delegates to it. UnknownIDs reports any --broker/broker_ids values that don't match a real entry, so a typo fails loudly instead of silently sending to nobody. CLI: `send` gains --broker, --region, --category, --exclude (repeatable/CSV) and --status (eligible|never|failed|all, replacing the old resend-cooldown-only skip logic) via the new filterBrokersByStatus. --resend now means --status all. Web: getBrokersWithStatus takes includeIDs/excludeIDs and now also honors options.regions (previously only enforced by the CLI's Filter path - a configured region restriction had no effect on the web brokers list or bulk send). handleAPISendAll exposes broker_ids/exclude_ids form fields and an "eligible" status option; both handleAPISendOne and handleAPISendAll now refuse to send while options.dry_run is set (previously only the CLI path respected dry_run). brokers.html gets an advanced-filters panel wired to the new fields. Tests: cmd/eraser/cmd_send_test.go, internal/web/handlers_send_filters_test.go.
… rolling daily limit
Three related bugs found in a code review of the send-job path, all in the
same area: JobManager/JobPersistence and processSendJob.
1. handleAPISendAll's GetActive-then-Create had a TOCTOU window: two
concurrent requests for the same profile could both pass the "no active
job" check before either created one, each spawning its own
processSendJob and double-sending every broker in the batch.
JobManager.CreateIfNoActive checks and inserts under one lock so only one
request can win; the early GetActive check stays as a fast-fail (skip
building the send list and validating the sender for a request that's
going to be rejected anyway), it's just no longer the only guard.
2. JobPersistence wrote every profile's pending-job state to the same
pending_job.json. GetActive already lets two profiles send concurrently
("each against its own daily limit and history" per its own doc
comment), so a second profile's Save silently overwrote the first's, and
a restart could only ever resume (or forget) whichever wrote last.
Persistence is now keyed by profile (pending_job-<id>.json; the default
profile keeps the legacy bare filename so an in-flight job survives this
upgrade), and checkPendingJob resumes every configured profile's pending
job, not just one.
3. processSendJob's daily-limit check compared against a local `sent`
counter that always starts at 0 - so daily_send_limit only ever capped a
single invocation. Resuming a paused job after a restart, or clicking
"Send all" again later the same day, could each push another full batch
past the configured limit. It's now anchored to CountSentSince's actual
rolling-24h total, the same source the CLI's `send` already uses.
Tests: TestCreateIfNoActiveRejectsSecondJobForSameProfile,
TestCreateIfNoActiveIsRaceSafe (run with -race),
TestJobPersistencePerProfileFiles,
TestJobPersistenceDefaultProfileUsesLegacyFilename,
TestProcessSendJobRespectsRollingDailyLimit.
SMTPSender.Send takes a context.Context - both handleAPISendOne and processSendJob wrap it in a 30s timeout, and a job's Cancel button cancels it - but the actual network code (tls.Dial, smtp.NewClient, Auth/Mail/Rcpt/ Data/Write/Close/Quit) never looked at ctx at all. net/smtp has no context support of its own, so a broker's SMTP server that accepted the TCP connection and then never responded (a hang, a firewall black-holing the session) blocked the send indefinitely: the 30s timeout was a no-op, and clicking Cancel on a stuck job did nothing - the goroutine just kept blocking on the read. send() now dials with DialContext, sets a deadline from ctx when one is present, and runs a watcher goroutine that closes the connection on ctx.Done() - net/smtp's blocking Read/Write calls return an error as soon as the underlying conn closes, which is what actually makes both the timeout and the cancel button work. TLS handshakes use HandshakeContext directly rather than tls.Dial. Also collapsed sendWithTLS and the smtp.SendMail plaintext path into one send() that both callers share, rather than plaintext going through a second, unfixed code path. Tests: TestSendRespectsContextDeadline, TestSendRespectsContextCancellation (both against a listener that accepts a connection and never responds).
FetchBounceEmails classifies a message as a bounce/NDR purely from its From/Subject text - neither is authenticated, and every broker's email is public in this open-source repo's data/brokers.yaml. Anyone who can land a message in the monitored inbox (a spoofed From, or any sender not SPF/DKIM-checked) could shape a fake NDR naming any of the ~700 public broker addresses and have `cleanup-bounces --remove` silently clear that broker's real contact. cleanup-bounces now only acts on a "bounce" whose extracted address belongs to a broker at least one configured profile has an actual successful-send history record for. This doesn't authenticate the bounce itself, but it shrinks the attack surface from any public broker address down to ones this user has genuinely emailed - a much narrower, much less guessable set. Unmatched candidates are printed as skipped rather than silently dropped, so a real bounce that lands here (e.g. from a broker emailed via a since-removed profile) is still visible for the user to act on by hand. everSentBrokerIDs factors the set-building out for testing (real IMAP scanning isn't mocked in this test suite, so it's the part that can actually be unit tested). Tests: TestEverSentBrokerIDs, TestEverSentBrokerIDsSingleProfileConfig.
The server binds 127.0.0.1 only, and csrf.Protect enforces same-origin via the Sec-Fetch-Site header rather than a token - but Sec-Fetch-Site doesn't cover DNS rebinding. A browser computes it from the requesting page's origin STRING (scheme+host+port), not the IP that hostname currently resolves to, so a page served from an attacker-controlled hostname that's been DNS-rebound to 127.0.0.1 still reads as same-origin to the browser - Sec-Fetch-Site: same-origin, cookies included - even though the request is actually reaching this local server. The only place that mismatch is still visible is the Host header, which still names the attacker's hostname. requireLoopbackHost rejects any request whose Host isn't 127.0.0.1, ::1, or localhost, registered before securityHeaders/csrf.Protect in the middleware chain. Since nothing outside the machine can reach this port under any hostname at all, a non-loopback Host only ever shows up via this rebinding trick, never a legitimate request. Existing router-level tests (routes_smoke_test.go) built requests with httptest.NewRequest, which defaults Host to "example.com" - now rejected by design, so they're updated to a small loopbackRequest helper that sets a realistic Host, matching what a real browser actually sends. Tests: TestHostAllowed, TestRequireLoopbackHostBlocksNonLoopback.
The verified list's original 46 US non-EU entries all carried the same generic note - "cross-checkable against the California CPPA data-broker registry" - asserted when the list was first built, never actually checked per-entry. 24 of those notes survived the later sweep/dedup passes; this runs the check that was owed. Findings (oag.ca.gov's historical registry, since cppa.ca.gov itself wasn't reachable to query directly): - 16 confirmed: alliant-cooperative-data-solutions, beenverified, cuebiq-group, foursquare-labs, instantcheckmate, kbm-group (registration independently corroborates its existing note), kochava, merkle, mylife, neighborwho (covered under BeenVerified's registration), nuwber, peoplefindersdaas, safegraph, spokeo, truthfinder, versium-analytics. Notes now cite the specific oag.ca.gov registration. - 7 not found under this name in a targeted search: advancedbackgroundchecks, checkr (plausibly FCRA-exempt, which would explain rather than concern), corelogic (likely registered under a subsidiary name), fastpeoplesearch, peoplesmart, thatsthem, usphonebook. Kept in the list - the verified bar is a confirmed broker with a current working contact, which none of this changes - but the notes now say plainly that registry provenance wasn't confirmed, instead of asserting it. - idtrue: turned up defunct mid-check (HireRight shut it down in March 2025 after acquiring backgroundchecks.com; www.idtrue.com no longer resolves). Removed from both lists rather than just noted, matching how near-north-america/oracle-america/veromi were already handled this session. All edits preserve each entry's pre-existing notes (a prior full-replace attempt at this lost real content - e.g. checkr's bounce history, Merkle's dentsu-merge note - and was reverted before landing).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #54.
The full broker list is broad and inherited from upstream, so it carries noise — dead domains, non-brokers, stale contacts. This addresses that in three parts rather than rewriting 762 entries.
1. Verified list (
data/brokers-verified.yaml)A second embedded list (48 entries): companies we can confidently call data brokers — US names cross-checkable against state registries, EU credit / data-intelligence bureaus — each with a working removal contact. The people-search / background-check set plus the big registry-listed marketing/adtech brokers (Acxiom, Epsilon, Oracle, LiveRamp, Experian/Equifax/TransUnion marketing arms, Kochava, Gravy Analytics, SafeGraph, …) and EU bureaus (Creditreform, Regis24, Creditsafe).
eraser send --list verified(per run) oroptions.broker_list: verified(permanent)broker.LoadListresolution for the send-family commands (send,draft,mark-sent,serve):--brokersflag →options.broker_file→ verified →~/.eraser/brokers.yaml→ embedded full. Everything else (audit-brokers,guides, reply processing) stays on the full list.Grow it from registry CSVs with the existing
scripts/import-registries— seedocs/auditing.md.2.
options.broker_fileConfig equivalent of the global
--brokersflag, for web / non-CLI users.--brokersstill wins.3.
audit-brokers --fix+ weekly auto-prune--fixclears the email of every broker whose mail domain is dead (MX and host lookup both fail — the reliable signal; website verdicts are never acted on), keeping the row and appending a dated note..github/workflows/broker-audit.ymlnow runs--fixweekly and opens a PR whendata/brokers.yamlchanged, instead of failing red for a maintainer to prune by hand. A quiet week opens no PR.Not in scope
One-by-one review of individual questionable entries in the full list (the "law firms" point from #54) stays manual via
docs/broker-replies.md. The verified list is the structural answer for users who want signal now.Notes
broker.Validategained aminCountparam — verified list floor isMinVerifiedBrokerCount(20), full list keepsMinSaneBrokerCount(200).validate-brokerswith no arg now checks both embedded lists.broker.LoadListprecedence, verified embed parses + validates,applyAuditFixonly touches email-dead entries.go build/go vet/go test -race/ golangci-lint / staticcheck / deadcode all clean.Manual verification
send --list verified --dry-run→ 48 brokers; full → 590options.broker_file→ only that list;--brokersflag still overridesaudit-brokers --fixon a fixture → cleared only the email-dead entry,email: ""+ dated note, row otherwise intact,.bakwritten, exit 0, still passesvalidate-brokers