Skip to content

Commit f6c4599

Browse files
committed
fix(data): merge 50 confirmed smartphone near-duplicate records
The strict integrity gate (integrity_check.py --strict) lists "near-duplicates in other categories not yet scanned" as open on #296. Scanned all 93,407 smartphone records for the three duplicate patterns the tablet/watch pass (PR#324) found, using the same method: normalize slugs/names (ignore punctuation and brand-prefix variance), then cross-check core specs (release_date, soc, ram_gb, battery_mah, weight_g, os) to separate genuine duplicates from similar-sounding but distinct SKUs. Found 767 brand-prefix sibling pairs (a bare slug like "nord-n100" and a brand-prefixed twin "oneplus-nord-n100" for the same brand+model). Of these, exactly 50 had byte-identical core specs and are the same device imported twice; the merges here cover only those 50. The remaining 717 pairs are NOT clean duplicates and are left untouched and documented in the PR (see below). Merge rules (keeps the diff reviewable, never invents a value): - Keeper = the shorter, non-brand-prefixed slug (same convention as the Intel Atom CPU dedup), kept at its existing path. - Two sub-patterns: * phonedb.net x sady36-kaggle pairs (allview, blackview, blu, caterpillar, doogee, lava, qmobile, ulefone, unihertz, zte): the bare side is the richer scrape (cameras, connectivity) but was verified=false; the prefixed side is the phonedb.net record (verified=true). Kept the richer bare record, unioned source_urls (so phonedb.net is retained as the human-review citation), set verified=true. * oneplus / xiaomi pairs: the prefixed side carried the fuller fields (connectivity, display resolution/refresh, wired-charging). Carried those over into the bare keeper, unioned source_urls. - One real spec conflict found inside an apparent duplicate: OnePlus 5T Star Wars Edition had main-camera 20MP on one side and 16MP on the other. GSMArena and OnePlus list the 5T main sensor as 16MP (Sony IMX398); the 20MP is the secondary. Corrected to 16MP and added the GSMArena review as a source. - Every merged record keeps the union of both sides' source_urls; the 50 redundant brand-prefixed files are deleted; base_model_slug realigned to the kept slug. Also surfaced (reported, not "fixed" — repairs would require inventing values): a systemic placeholder-value bug. 487 distinct device names across 624 files share the exact triple battery_mah=1500, weight_g=150.0, ram_gb=1 — spanning 1994-2020 and even watches/tablets, mostly from the gsmarena-mobile-devices Kaggle import. This is the same fabricated-defaults pattern (1500/150) PR#324 caught in tablet/watch. It is also what makes 717 of the 767 prefix pairs look like "conflicts": the bare side often carries these 1500/150 defaults plus a generic YYYY-01-01 date while the twin has real specs. Flagged for a dedicated repair pass rather than mass-editing here. Both gates pass on the merged tree: `python -m app.validate` and `integrity_check.py <data> --strict` (0 hard anomalies). Closes #296
1 parent 2bd6bc4 commit f6c4599

100 files changed

Lines changed: 270 additions & 1694 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎data/smartphone/allview/2018/allview-x5-soul-global-dual-sim-td-lte.json‎

Lines changed: 0 additions & 27 deletions
This file was deleted.

‎data/smartphone/allview/2018/x5-soul-global-dual-sim-td-lte.json‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,18 @@
11
{
22
"slug": "x5-soul-global-dual-sim-td-lte",
33
"base_model_slug": "x5-soul-global-dual-sim-td-lte",
4-
"name": "X5 Soul Global Dual SIM TD-LTE",
4+
"name": "Allview X5 Soul Global Dual SIM TD-LTE",
55
"brand": "allview",
66
"soc": "mediatek-helio-p22-mt6762",
77
"release_date": "2018-09-01",
88
"ram_gb": 4,
99
"battery_mah": 3000,
1010
"weight_g": 185.0,
1111
"os": "Android",
12-
"verified": false,
12+
"verified": true,
1313
"source_urls": [
14-
"https://www.kaggle.com/datasets/sady36/mobile-phones-specs"
14+
"https://www.kaggle.com/datasets/sady36/mobile-phones-specs",
15+
"https://phonedb.net/index.php?m=device&id=16505"
1516
],
1617
"storage_options_gb": [
1718
32

‎data/smartphone/blackview/2018/blackview-bv9500-global-dual-sim-td-lte.json‎

Lines changed: 0 additions & 27 deletions
This file was deleted.

‎data/smartphone/blackview/2018/blackview-bv9500-pro-global-dual-sim-td-lte.json‎

Lines changed: 0 additions & 27 deletions
This file was deleted.

‎data/smartphone/blackview/2018/blackview-max-1-dual-sim-td-lte.json‎

Lines changed: 0 additions & 27 deletions
This file was deleted.

‎data/smartphone/blackview/2018/blackview-p10000-pro-global-dual-sim-td-lte.json‎

Lines changed: 0 additions & 27 deletions
This file was deleted.

‎data/smartphone/blackview/2018/bv9500-global-dual-sim-td-lte.json‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,18 @@
11
{
22
"slug": "bv9500-global-dual-sim-td-lte",
33
"base_model_slug": "bv9500-global-dual-sim-td-lte",
4-
"name": "BV9500 Global Dual SIM TD-LTE",
4+
"name": "Blackview BV9500 Global Dual SIM TD-LTE",
55
"brand": "blackview",
66
"soc": "mediatek-helio-p23-mt6763t",
77
"release_date": "2018-08-01",
88
"ram_gb": 4,
99
"battery_mah": 10000,
1010
"weight_g": 365.0,
1111
"os": "Android",
12-
"verified": false,
12+
"verified": true,
1313
"source_urls": [
14-
"https://www.kaggle.com/datasets/sady36/mobile-phones-specs"
14+
"https://www.kaggle.com/datasets/sady36/mobile-phones-specs",
15+
"https://phonedb.net/index.php?m=device&id=16487"
1516
],
1617
"storage_options_gb": [
1718
64

‎data/smartphone/blackview/2018/bv9500-pro-global-dual-sim-td-lte.json‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,18 @@
11
{
22
"slug": "bv9500-pro-global-dual-sim-td-lte",
33
"base_model_slug": "bv9500-pro-global-dual-sim-td-lte",
4-
"name": "BV9500 Pro Global Dual SIM TD-LTE",
4+
"name": "Blackview BV9500 Pro Global Dual SIM TD-LTE",
55
"brand": "blackview",
66
"soc": "mediatek-helio-p23-mt6763t",
77
"release_date": "2018-06-01",
88
"ram_gb": 6,
99
"battery_mah": 10000,
1010
"weight_g": 365.0,
1111
"os": "Android",
12-
"verified": false,
12+
"verified": true,
1313
"source_urls": [
14-
"https://www.kaggle.com/datasets/sady36/mobile-phones-specs"
14+
"https://www.kaggle.com/datasets/sady36/mobile-phones-specs",
15+
"https://phonedb.net/index.php?m=device&id=16477"
1516
],
1617
"storage_options_gb": [
1718
128

‎data/smartphone/blackview/2018/max-1-dual-sim-td-lte.json‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,18 @@
11
{
22
"slug": "max-1-dual-sim-td-lte",
33
"base_model_slug": "max-1-dual-sim-td-lte",
4-
"name": "Max 1 Dual SIM TD-LTE",
4+
"name": "Blackview Max 1 Dual SIM TD-LTE",
55
"brand": "blackview",
66
"soc": "mediatek-helio-p23-mt6763t",
77
"release_date": "2018-12-01",
88
"ram_gb": 6,
99
"battery_mah": 4680,
1010
"weight_g": 216.0,
1111
"os": "Android",
12-
"verified": false,
12+
"verified": true,
1313
"source_urls": [
14-
"https://www.kaggle.com/datasets/sady36/mobile-phones-specs"
14+
"https://www.kaggle.com/datasets/sady36/mobile-phones-specs",
15+
"https://phonedb.net/index.php?m=device&id=16467"
1516
],
1617
"storage_options_gb": [
1718
64

‎data/smartphone/blackview/2018/p10000-pro-global-dual-sim-td-lte.json‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,18 @@
11
{
22
"slug": "p10000-pro-global-dual-sim-td-lte",
33
"base_model_slug": "p10000-pro-global-dual-sim-td-lte",
4-
"name": "P10000 Pro Global Dual SIM TD-LTE",
4+
"name": "Blackview P10000 Pro Global Dual SIM TD-LTE",
55
"brand": "blackview",
66
"soc": "mediatek-helio-p23-mt6763",
77
"release_date": "2018-05-01",
88
"ram_gb": 4,
99
"battery_mah": 11000,
1010
"weight_g": 300.0,
1111
"os": "Android",
12-
"verified": false,
12+
"verified": true,
1313
"source_urls": [
14-
"https://www.kaggle.com/datasets/sady36/mobile-phones-specs"
14+
"https://www.kaggle.com/datasets/sady36/mobile-phones-specs",
15+
"https://phonedb.net/index.php?m=device&id=16488"
1516
],
1617
"storage_options_gb": [
1718
64

0 commit comments

Comments
 (0)