Commit f6c4599
committed
fix(data): merge 50 confirmed smartphone near-duplicate records
The strict integrity gate (integrity_check.py --strict) lists "near-duplicates in
other categories not yet scanned" as open on #296. Scanned all 93,407 smartphone
records for the three duplicate patterns the tablet/watch pass (PR#324) found, using
the same method: normalize slugs/names (ignore punctuation and brand-prefix variance),
then cross-check core specs (release_date, soc, ram_gb, battery_mah, weight_g, os)
to separate genuine duplicates from similar-sounding but distinct SKUs.
Found 767 brand-prefix sibling pairs (a bare slug like "nord-n100" and a
brand-prefixed twin "oneplus-nord-n100" for the same brand+model). Of these, exactly
50 had byte-identical core specs and are the same device imported twice; the merges
here cover only those 50. The remaining 717 pairs are NOT clean duplicates and are
left untouched and documented in the PR (see below).
Merge rules (keeps the diff reviewable, never invents a value):
- Keeper = the shorter, non-brand-prefixed slug (same convention as the Intel Atom
CPU dedup), kept at its existing path.
- Two sub-patterns:
* phonedb.net x sady36-kaggle pairs (allview, blackview, blu, caterpillar, doogee,
lava, qmobile, ulefone, unihertz, zte): the bare side is the richer scrape
(cameras, connectivity) but was verified=false; the prefixed side is the
phonedb.net record (verified=true). Kept the richer bare record, unioned
source_urls (so phonedb.net is retained as the human-review citation), set
verified=true.
* oneplus / xiaomi pairs: the prefixed side carried the fuller fields
(connectivity, display resolution/refresh, wired-charging). Carried those over
into the bare keeper, unioned source_urls.
- One real spec conflict found inside an apparent duplicate: OnePlus 5T Star Wars
Edition had main-camera 20MP on one side and 16MP on the other. GSMArena and
OnePlus list the 5T main sensor as 16MP (Sony IMX398); the 20MP is the secondary.
Corrected to 16MP and added the GSMArena review as a source.
- Every merged record keeps the union of both sides' source_urls; the 50 redundant
brand-prefixed files are deleted; base_model_slug realigned to the kept slug.
Also surfaced (reported, not "fixed" — repairs would require inventing values):
a systemic placeholder-value bug. 487 distinct device names across 624 files share
the exact triple battery_mah=1500, weight_g=150.0, ram_gb=1 — spanning 1994-2020 and
even watches/tablets, mostly from the gsmarena-mobile-devices Kaggle import. This is
the same fabricated-defaults pattern (1500/150) PR#324 caught in tablet/watch. It is
also what makes 717 of the 767 prefix pairs look like "conflicts": the bare side
often carries these 1500/150 defaults plus a generic YYYY-01-01 date while the twin
has real specs. Flagged for a dedicated repair pass rather than mass-editing here.
Both gates pass on the merged tree: `python -m app.validate` and
`integrity_check.py <data> --strict` (0 hard anomalies).
Closes #2961 parent 2bd6bc4 commit f6c4599
100 files changed
Lines changed: 270 additions & 1694 deletions
File tree
- data/smartphone
- allview/2018
- blackview
- 2018
- 2019
- blu/2019
- caterpillar/2020
- doogee
- 2018
- 2019
- lava/2018
- oneplus
- 2017
- 2018
- 2019
- 2020
- 2021
- 2023
- 2024
- qmobile/2019
- ulefone/2019
- unihertz/2017
- xiaomi/2023
- zte
- 2019
- 2020
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 0 additions & 27 deletions
This file was deleted.
Lines changed: 4 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
4 | | - | |
| 4 | + | |
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
12 | | - | |
| 12 | + | |
13 | 13 | | |
14 | | - | |
| 14 | + | |
| 15 | + | |
15 | 16 | | |
16 | 17 | | |
17 | 18 | | |
| |||
Lines changed: 0 additions & 27 deletions
This file was deleted.
Lines changed: 0 additions & 27 deletions
This file was deleted.
Lines changed: 0 additions & 27 deletions
This file was deleted.
Lines changed: 0 additions & 27 deletions
This file was deleted.
Lines changed: 4 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
4 | | - | |
| 4 | + | |
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
12 | | - | |
| 12 | + | |
13 | 13 | | |
14 | | - | |
| 14 | + | |
| 15 | + | |
15 | 16 | | |
16 | 17 | | |
17 | 18 | | |
| |||
Lines changed: 4 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
4 | | - | |
| 4 | + | |
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
12 | | - | |
| 12 | + | |
13 | 13 | | |
14 | | - | |
| 14 | + | |
| 15 | + | |
15 | 16 | | |
16 | 17 | | |
17 | 18 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
4 | | - | |
| 4 | + | |
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
12 | | - | |
| 12 | + | |
13 | 13 | | |
14 | | - | |
| 14 | + | |
| 15 | + | |
15 | 16 | | |
16 | 17 | | |
17 | 18 | | |
| |||
Lines changed: 4 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
4 | | - | |
| 4 | + | |
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
12 | | - | |
| 12 | + | |
13 | 13 | | |
14 | | - | |
| 14 | + | |
| 15 | + | |
15 | 16 | | |
16 | 17 | | |
17 | 18 | | |
| |||
0 commit comments