fix(data): add source citations for 7086 website records - #345
Merged
Merged
Conversation
Upgrade website records from Wikidata-only (verified:false) to source-verified by adding an independent, checkable citation: the English Wikipedia article linked from each record's Wikidata entity (enwiki sitelink). Only records whose Wikidata entity has an enwiki article were verified; the Wikipedia URL was appended to source_urls and verified set to true. No stored values were fabricated or altered. Records with no enwiki article were left untouched (still verified:false, Wikidata-only). 17 records whose launch_date disagreed with Wikidata inception (P571) were left untouched for manual review, since a site's launch date and the subject's inception are distinct facts. Refs #296
…bsite records Regenerated site/public/v1/websites from TechEngine (python -m app.dump against a fresh scratch DATABASE_URL with TECHAPI_DATA_DIR pointing at this worktree). Reflects the added Wikipedia citations and verified:true flags. Refs #296
github-actions
Bot
added
bug
Member
🔎 Data verification — Tier 0 (offline existence/trust)Scored by 12/12 categories ? 191,819 records ? diff base develop@e9fc443 Changed records in this PR7086 record(s) assessed. %%{init: {"theme":"base","themeVariables":{"pie1":"#3fb950","pie2":"#d29922","pie3":"#f85149","pieStrokeWidth":"0px","pieOpacity":"1"}}}%%
pie showData
title Verification bands — all records
"Green" : 5747
"Yellow" : 1339
"Red" : 0
Full-dataset baseline191819 record(s) assessed. %%{init: {"theme":"base","themeVariables":{"pie1":"#3fb950","pie2":"#d29922","pie3":"#f85149","pieStrokeWidth":"0px","pieOpacity":"1"}}}%%
pie showData
title Verification bands — all records
"Green" : 38519
"Yellow" : 152245
"Red" : 1055
Hard violations (forced red):
green = authoritative source + complete + consistent · yellow = plausible, needs confirmation · red = sparse/weak source or a hard contradiction. Promotion to |
Member
TechEngine change review: PASS12/12 categories ? 191,819 records ? diff base develop@e9fc443
Changed data
Changed record exampleswebsite modified
Heuristic review
|
Member
TechEngine validation stats: PASS12/12 categories ? 191,819 records ? diff base develop@e9fc443
Data summary
Warning Tracked verified coverage is below 50% for monitor 0.0% (0/882), software 0.8% (358/42493), laptop 5.3% (105/1995), tablet 8.3% (288/3459), soc 13.4% (284/2117), smartphone 18.8% (17581/93407), all 20.5% (39343/191819), watch 21.0% (94/447), and 1 more. Validation notes
Key output:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Upgrades 7,086
data/website/records from uncited (Wikidata-only,verified: false) to source-verified by adding an independent, checkableauthoritative citation and flipping
verified: true. No stored values werefabricated, guessed, or altered.
Addresses the uncited-website backlog. Closes #296.
Methodology
Every website record previously carried exactly one
source_urlsentry — itsWikidata item (
wikidata.org/wiki/Q…) — and ~32k wereverified: false. AWikidata item alone is a database cross-reference, not an independent
confirmation, so those records were effectively uncited.
For each
verified: false, Wikidata-only record:(
wbgetentities,props=sitelinks|claims|labels).enwikisitelink), thatarticle is a real, human-checkable authoritative source describing the same
entity. The exact article URL returned by the API was appended to
source_urlsandverifiedset totrue.launch_dateagainst Wikidata inception (P571).(still
verified: false, Wikidata-only) — no citation was invented.Every added URL is the genuine Wikipedia sitelink Wikidata reports for that
exact entity;
verified: trueis only ever set together with a real articleURL. This is fully reproducible against the live Wikidata API.
Counts
verified: true)launch_datevs WikidataP571year mismatch, flagged for manual reviewverified: truebefore this pass (untouched)On the 0 value corrections
17 records had a
launch_datewhose year disagreed with the Wikidata inceptiondate (
P571). These were not auto-corrected: a website's launch date and theinception of its subject/parent organisation are distinct facts (e.g. Netflix the
company, 1997, vs. the streaming service, 2007). Overwriting one with the other
would introduce wrong data, which the task forbids. They are left exactly as-is
and flagged for manual review instead.
Verification
python -m app.validate→ passed.python integrity_check.py <worktree>/data --strict→ exit 0, nohard anomalies (empty
structuralsection; only pre-existing advisoryCPU/GPU benchmark ratio outliers printed, unrelated to this change).
(
python -m app.dump, fresh scratchDATABASE_URL,TECHAPI_DATA_DIRpointing at this worktree) — regenerated
site/public/v1/websitesfor theaffected records (last commit). Unrelated pre-existing dump drift in other
categories was reverted to keep this PR scoped to
data/website/only.Scope
data/website/only. 7,086 data files + 7,086 regenerated dump files.This is one batch; the remaining uncited records (those without an English
Wikipedia article) are intentionally left for future batches per the task's
"batch into multiple PRs" guidance.