Skip to content

data: import GSMArena cigarplug batch — 2,498 SoC-linked phones + tablets/watches/pdas - #98

Merged
Seungpyo1007 merged 8 commits into
mainfrom
data/import-staging
Jul 1, 2026
Merged

Seungpyo1007 merged 8 commits into
mainfrom
data/import-staging

Conversation

@Seungpyo1007

Copy link
Copy Markdown
Member

What & why

Second GSMArena batch from the cigarplug/scrape-gsma dataset (10,679-row GSMArena scrape). PR #97 imported the tablet/watch/pda subset; this brings in 2,498 new records — mostly smartphones — with each phone linked to a real SoC in the catalog.

Category Added
smartphone 2,379
tablet 101
watch 9
pda 9
total 2,498

All records land verified: false. Smartphones satisfy the stricter phone schema: integer ram_gb, ranges (battery 500–12000 mAh, weight 50–500 g), and a soc FK resolved to an existing SoC via a precise die-number + marketing-name matcher (unmatched chipsets are skipped, not guessed — avoids the fuzzy-match trap).

Commits (each Refs #1): 5 smartphone brand-group commits (Samsung/Huawei/Honor · LG/Xiaomi/Oppo/Vivo · Motorola/HTC/ZTE/Sony · Asus/Lenovo/Nokia/Meizu/Realme · remaining), tablet, watch+pda, and a surgical dump refresh.

Closes #1

Source

GSMArena via cigarplug/scrape-gsma (https://github.com/cigarplug/scrape-gsma), *-scrapegsma-<row> slugs. raw_source preserves the original chipset/CPU/GPU/memory strings; variant.source_row cites the CSV row.

Dump

Regenerated via TechEngine app.dump at pinned submodule c0928f9 (scoring v2.0.0). Only the 2,498 new records' detail+score files, the four category indexes and the manifest counts are committed (smartphones 90118→92497, tablets 3354→3455, watches 424→433, pdas 131→140) — no churn on unchanged records.

Checklist

  • python -m app.validate — ✅ passed (full dataset)
  • python TechEngine/integrity_check.py data --strict — ✅ no hard anomalies
  • git diff --check — ✅ no whitespace errors
  • Correct data/<category>/<brand>/<year>/<base>/<slug>.json paths, kebab-case unique slugs
  • source_urls present on every record

GSMArena device dump (cigarplug/scrape-gsma) for Samsung Galaxy,
Huawei and Honor lines, with SoC linked to the known SoC catalog.
Bulk import, verified:false.

Refs #1
GSMArena device dump (cigarplug/scrape-gsma) for LG, Xiaomi/Redmi/Poco,
Oppo and Vivo lines. SoC-linked, verified:false.

Refs #1
GSMArena device dump (cigarplug/scrape-gsma) for Motorola, HTC, ZTE
and Sony Xperia lines. SoC-linked, verified:false.

Refs #1
…, Realme

GSMArena device dump (cigarplug/scrape-gsma) for Asus, Lenovo, Nokia,
Meizu and Realme lines. SoC-linked, verified:false.

Refs #1
GSMArena device dump (cigarplug/scrape-gsma) for the remaining brands
(Infinix, BLU, Alcatel, Gionee, Allview, Wiko, Tecno, Microsoft,
Micromax, Panasonic, BlackBerry, Archos, BQ and others). SoC-linked,
verified:false.

Refs #1
Additional GSMArena tablet records from the cigarplug/scrape-gsma
dump not covered by the earlier batch. verified:false.

Refs #1
…g dump)

Remaining GSMArena smartwatch and PDA records from the
cigarplug/scrape-gsma dump. verified:false.

Refs #1
Add the 2,498 new GSMArena (cigarplug/scrape-gsma) records to the
static v1 dump — smartphone/tablet/watch/pda detail + smartphone
score entries — and bump the category index counts and manifest.
Regenerated via TechEngine app.dump at pinned submodule c0928f9
(scoring v2.0.0); only the new records and affected indexes are
committed.

Refs #1
@Seungpyo1007

Copy link
Copy Markdown
Member Author

🤖 Review — PASS

Check Command Result
Schema/self-check python -m app.validate ✅ passed (full dataset)
Integrity gate python TechEngine/integrity_check.py data --strict ✅ no hard anomalies
Whitespace git diff --check origin/main..HEAD ✅ none
Engine dump app.dump @ c0928f9 ✅ 104,825 records, no serializer errors

Data-change summary

  • Added: 2,498 — smartphone 2,379, tablet 101, watch 9, pda 9
  • Modified/Deleted: 0
  • All verified: false, each with source_urls.
  • Smartphones carry an integer ram_gb and a soc FK resolved to the existing SoC catalog (die-number + marketing-name exact match; unmatched chipsets skipped rather than guessed).

Example records

  • data/smartphone/google/2019/pixel-4-xl/google-pixel-4-xl-scrapegsma-5.json (soc snapdragon-855)
  • data/smartphone/samsung/2015/galaxy-a3-2016/…-scrapegsma-7959.json
  • data/tablet/acer/2011/iconia-smart/acer-iconia-smart-scrapegsma-1771.json

Heuristic warnings

  • A handful of voice-calling phablets (e.g. HP Slate voicetab) land under smartphone — genuinely ambiguous GSMArena devices; acceptable for a verified:false import.

Site build

  • No site/src changes. site/public/v1 dump regenerated (scoring v2.0.0); only new records + affected indexes/manifest committed.

@Seungpyo1007

Copy link
Copy Markdown
Member Author

📊 Dataset stats (post-merge projection)

Metric Count
Total records 104,825
verified: true 10,028
verified: false 94,791
missing verified 6
Verified ratio 9.57%

Warning

Verified ratio is 9.57% (down slightly from 9.80% as 2,498 new verified: false records land). Expected for the bulk-import campaign — promotion to verified happens later via the TechEngine verify-network reality check. The 6 records missing a verified field pre-date this PR.

Campaign progress toward +50k: cigarplug/scrape-gsma is now largely exhausted (~2,871 total imported across PR #97 + #98). Remaining headroom needs additional datasets (Kaggle phone dumps, PhoneDB, AllPhones SKU variants) — tracked in #1.

@Seungpyo1007
Seungpyo1007 merged commit 887d12a into main Jul 1, 2026
5 checks passed
@github-project-automation github-project-automation Bot moved this from In Progress to Done in TechAPI-Project Jul 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

data Dataset changes enhancement New feature or request

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

TechAPI dataset roadmap and status: all categories (1989-2026)

1 participant