Skip to content

data: recover 846 GSMArena budget phones via SoC-matcher fix - #101

Merged
Seungpyo1007 merged 5 commits into
mainfrom
data/import-staging
Jul 7, 2026
Merged

Seungpyo1007 merged 5 commits into
mainfrom
data/import-staging

Conversation

@Seungpyo1007

Copy link
Copy Markdown
Member

What & why

Follow-up to PR #98. While importing cigarplug/scrape-gsma I found a bug in my SoC-matcher's die-number extraction (it stripped hyphens, breaking the \b word boundary so mediatek-mt6582 never matched the chipset Mediatek MT6582). Fixing it recovers 846 budget/mid-range phones — mostly MediaTek MT6xxx devices — that link to already-existing SoCs (no new SoC records needed).

  • Added: 846 smartphones (BLU, Allview, Verykool, Lenovo, Micromax, Alcatel, ZTE, Panasonic, Gionee, and other budget brands)
  • Modified/Deleted: 0
  • All verified: false, integer ram_gb, valid soc FK to the existing catalog.

Commits (each Refs #1): 4 brand-group smartphone commits + surgical dump refresh.

Closes #1

Source

GSMArena via cigarplug/scrape-gsma, *-scrapegsma-<row> slugs; rows previously skipped only because the die-token matcher missed their SoC.

Dump

Regenerated via TechEngine app.dump at pinned submodule c0928f9 (scoring v2.0.0). Only the 846 new records' detail+score files, the smartphones index (92,497→93,343) and the manifest are committed — no churn on unchanged records.

Checklist

  • python -m app.validate — ✅ passed
  • python TechEngine/integrity_check.py data --strict — ✅ no hard anomalies
  • git diff --check — ✅ clean
  • Correct paths, kebab-case unique slugs, source_urls present

… budget brands

Budget/emerging-market phones recovered from cigarplug/scrape-gsma
after fixing SoC die-number matching (MediaTek MT6xxx family now links
to the existing SoC catalog). verified:false.

Refs #1
… ZTE, Panasonic

More budget/mid phones recovered from cigarplug/scrape-gsma via the
SoC-matcher fix. SoC-linked, verified:false.

Refs #1
…ko, Lava and others

Recovered cigarplug/scrape-gsma phones across Gionee, Acer, Oppo, LG,
Wiko, Lava, Vivo, Tecno. SoC-linked, verified:false.

Refs #1
Remaining cigarplug/scrape-gsma phones unlocked by the SoC die-match
fix (Karbonn, Intex, Spice, Celkon, Maxwest and other budget brands).
SoC-linked, verified:false.

Refs #1
Add the 846 recovered GSMArena (cigarplug/scrape-gsma) smartphone
records to the static v1 dump — detail + score — and bump the
smartphones index count (92497->93343) and manifest. Regenerated via
TechEngine app.dump at pinned submodule c0928f9 (scoring v2.0.0);
only the new records and affected index are committed.

Refs #1
@Seungpyo1007

Copy link
Copy Markdown
Member Author

🤖 Review — PASS

Check Command Result
Schema/self-check python -m app.validate ✅ passed (full dataset)
Integrity gate python TechEngine/integrity_check.py data --strict ✅ no hard anomalies
Whitespace git diff --check origin/main..HEAD ✅ none
Engine dump app.dump @ c0928f9 ✅ 105,671 records, no serializer errors

Data-change summary

  • Added: 846 smartphones · Modified/Deleted: 0
  • All verified:false, integer ram_gb, valid soc FK (all 846 verified against the SoC catalog: 0 invalid).
  • Root cause fixed: SoC die-token matcher stripped hyphens → \b boundary broke → mediatek-mt6582 never matched Mediatek MT6582. Now die numbers (MT6xxx, SM8xxx, MSM8xxx, SDM6xx) extract correctly.

Example records

  • data/smartphone/acer/2012/liquid-gallant-duo/acer-liquid-gallant-duo-scrapegsma-1759.json (soc mediatek-mt6575)
  • BLU/Allview/Verykool budget devices on MediaTek MT6580/MT6582/MT6737.

Site build

  • No site/src changes. Surgical dump refresh (846 new records + smartphones index + manifest only).

@Seungpyo1007

Copy link
Copy Markdown
Member Author

📊 Dataset stats (post-merge projection)

Metric Count
Total records 105,671
verified: true 10,028
verified: false 95,637
missing verified 6
Verified ratio 9.49%

Warning

Verified ratio 9.49% — expected for the bulk-import campaign; promotion happens later via TechEngine verify-network. The 6 missing-verified records pre-date this PR.

Session running total (cigarplug/scrape-gsma fully mined): PR #97 (373) + PR #98 (2,498) + this (846) = 3,717 net-new records. cigarplug is now exhausted; further phone growth needs new datasets or a new category (see #1).

@Seungpyo1007
Seungpyo1007 merged commit 2ab111b into main Jul 7, 2026
5 checks passed
@Seungpyo1007 Seungpyo1007 moved this from In Progress to Done in TechAPI-Project Jul 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

data Dataset changes enhancement New feature or request

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

TechAPI dataset roadmap and status: all categories (1989-2026)

1 participant