Commit 1223a96
committed
fix(data): merge 153 confirmed website near-duplicate records
Scanned all ~40k data/website records for near-duplicates keyed on normalized homepage_url and display name. Merged 153 groups that denote the same real website imported more than once - QID-suffixed clones, bare-domain stubs vs proper records, protocol/www/trailing-slash/punctuation variants, and same-entity name variants (e.g. Amazon.com/Amazon, ANNO/AustriaN Newspapers Online, translated open-data portal names).
Each merge keeps the fuller/verified/better-sourced side, unions source_urls, ORs the verified flag, and fills only missing optional fields. Redundant files deleted (154 files; one 3-way group). Websites have no inbound cross-category references, so deletions are safe.
Kept as distinct: different exhibitions/services sharing a museum or portal landing URL, board pages, registration services differing by URL fragment, and borderline company-vs-website / subsite / regional-version pairs. validate.py passes; integrity_check.py --strict reports no hard anomalies.
Refs #2961 parent 01d6565 commit 1223a96
307 files changed
Lines changed: 396 additions & 1782 deletions
File tree
- data/website
- 3d
- ab
- ac
- af
- ag
- ai
- ak
- am
- an
- ap
- ar
- au
- ba
- bd
- be
- bi
- bo
- br
- bu
- ca
- ce
- ci
- cn
- co
- cr
- d-
- da
- dc
- de
- di
- do
- dr
- dw
- ec
- eh
- el
- en
- eo
- ex
- f-
- fa
- fi
- fl
- fr
- fu
- ge
- gi
- go
- gr
- gu
- ha
- hc
- he
- hi
- ho
- hu
- id
- in
- ir
- ja
- ko
- ku
- la
- le
- li
- lr
- ly
- ma
- me
- mi
- mo
- my
- na
- ne
- ni
- nj
- no
- nu
- of
- oh
- on
- op
- pe
- pi
- pl
- po
- pr
- ra
- ru
- sa
- si
- sk
- sl
- sp
- st
- su
- te
- tf
- th
- ti
- tn
- to
- ud
- uk
- un
- va
- vg
- vr
- wa
- wi
- ya
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
6 | | - | |
| 6 | + | |
| 7 | + | |
7 | 8 | | |
8 | 9 | | |
9 | 10 | | |
| |||
This file was deleted.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
6 | | - | |
| 6 | + | |
| 7 | + | |
7 | 8 | | |
8 | 9 | | |
9 | 10 | | |
| |||
This file was deleted.
This file was deleted.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
6 | | - | |
| 6 | + | |
| 7 | + | |
7 | 8 | | |
8 | | - | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
9 | 13 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
6 | | - | |
| 6 | + | |
| 7 | + | |
7 | 8 | | |
8 | 9 | | |
9 | 10 | | |
| |||
This file was deleted.
This file was deleted.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
6 | | - | |
| 6 | + | |
| 7 | + | |
7 | 8 | | |
8 | 9 | | |
9 | 10 | | |
0 commit comments