Skip to content

Commit 1223a96

Browse files
committed
fix(data): merge 153 confirmed website near-duplicate records
Scanned all ~40k data/website records for near-duplicates keyed on normalized homepage_url and display name. Merged 153 groups that denote the same real website imported more than once - QID-suffixed clones, bare-domain stubs vs proper records, protocol/www/trailing-slash/punctuation variants, and same-entity name variants (e.g. Amazon.com/Amazon, ANNO/AustriaN Newspapers Online, translated open-data portal names). Each merge keeps the fuller/verified/better-sourced side, unions source_urls, ORs the verified flag, and fills only missing optional fields. Redundant files deleted (154 files; one 3-way group). Websites have no inbound cross-category references, so deletions are safe. Kept as distinct: different exhibitions/services sharing a museum or portal landing URL, board pages, registration services differing by URL fragment, and borderline company-vs-website / subsite / regional-version pairs. validate.py passes; integrity_check.py --strict reports no hard anomalies. Refs #296
1 parent 01d6565 commit 1223a96

307 files changed

Lines changed: 396 additions & 1782 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎data/website/3d/3dm.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,8 @@
33
"name": "3DM",
44
"verified": false,
55
"source_urls": [
6-
"https://www.wikidata.org/wiki/Q22906088"
6+
"https://www.wikidata.org/wiki/Q22906088",
7+
"https://www.wikidata.org/wiki/Q89139345"
78
],
89
"homepage_url": "http://www.3dmgame.com/",
910
"launch_date": "2001-01-01"

‎data/website/3d/3dmgame-com.json‎

Lines changed: 0 additions & 9 deletions
This file was deleted.

‎data/website/ab/abs-cbn-news.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,8 @@
33
"name": "ABS-CBN News",
44
"verified": false,
55
"source_urls": [
6-
"https://www.wikidata.org/wiki/Q110278688"
6+
"https://www.wikidata.org/wiki/Q110278688",
7+
"https://www.wikidata.org/wiki/Q20311853"
78
],
89
"homepage_url": "https://news.abs-cbn.com/",
910
"owners": [

‎data/website/ab/abs-cbnnews-com.json‎

Lines changed: 0 additions & 12 deletions
This file was deleted.

‎data/website/ac/actu-mag-fr.json‎

Lines changed: 0 additions & 12 deletions
This file was deleted.

‎data/website/ac/actu-mag.json‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,11 @@
33
"name": "Actu-Mag",
44
"verified": false,
55
"source_urls": [
6-
"https://www.wikidata.org/wiki/Q124397674"
6+
"https://www.wikidata.org/wiki/Q124397674",
7+
"https://www.wikidata.org/wiki/Q117352796"
78
],
8-
"homepage_url": "https://www.actu-mag.fr/"
9+
"homepage_url": "https://www.actu-mag.fr/",
10+
"languages": [
11+
"French"
12+
]
913
}

‎data/website/af/aftermath-q130051799.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,8 @@
33
"name": "Aftermath",
44
"verified": false,
55
"source_urls": [
6-
"https://www.wikidata.org/wiki/Q130051799"
6+
"https://www.wikidata.org/wiki/Q130051799",
7+
"https://www.wikidata.org/wiki/Q136342168"
78
],
89
"homepage_url": "https://aftermath.site",
910
"launch_date": "2023-11-01"

‎data/website/af/aftermath-q136342168.json‎

Lines changed: 0 additions & 9 deletions
This file was deleted.

‎data/website/ag/agi-repository-q68169300.json‎

Lines changed: 0 additions & 9 deletions
This file was deleted.

‎data/website/ag/agi-repository.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,8 @@
33
"name": "AGI Repository",
44
"verified": false,
55
"source_urls": [
6-
"https://www.wikidata.org/wiki/Q111645441"
6+
"https://www.wikidata.org/wiki/Q111645441",
7+
"https://www.wikidata.org/wiki/Q68169300"
78
],
89
"homepage_url": "https://agi.repo.nii.ac.jp/"
910
}

0 commit comments

Comments
 (0)