fix(data): merge 28 confirmed PDA near-duplicate records - #335
Merged
Merged
Conversation
Scanned the pda category (140 records, reviewed every match by hand) for the three duplicate patterns the tablet/watch (#324) and smartphone (#329/#330) passes turned up. Merged 28 redundant files into 21 keepers (140 -> 112), keeping the better-sourced side field by field and unioning source_urls. Pattern 1 - identical scrape imported twice under different row-id slugs (100% identical specs, differ only by source_row): - motorola/2002/accompli-388: kept ...-scrapegsma-344, deleted ...-1227. - mitac/2005/mio-a701: kept ...-scrapegsma-278, deleted ...-1161. Pattern 2 - same device under a brand-prefixed slug and a bare slug in sibling dirs. Same convention as the #330 GSMArena/phonedb merge: kept the bare GSMArena/Kaggle record (real specs + raw_gsmarena, sometimes Wikipedia), upgraded name to the brand-inclusive form, unioned source_urls to retain the phonedb reference, kept the keeper's soc/release_date, left verified as-is, deleted the phonedb twin: - treo-680 <- palm-treo-680-palm-lowrider ("Palm Treo 680") - treo-750v <- palm-treo-750v-htc-cheetah ("Palm Treo 750v") - treo-750 <- palm-treo-750-htc-cheetah ("Palm Treo 750") - treo-500v <- palm-treo-500v-palm-otto ("Palm Treo 500v") Treo 750 vs 750v and Treo 500 vs 500v were kept as distinct SKUs (only the matching-model twins were merged). palm-treo-500-palm-otto (2008 Treo 500) is a different device and was left alone. Pattern 3 - cross-source duplicate: the same PDA imported as a clean short-slug record and as a cigarplug/scrape-gsma row in the same base dir. Kept the clean record (correct soc FK, canonical slug), deleted the scrape twin, and repaired the keeper's ram_gb from the twin's cited memory_internal (the clean side carried the fabricated ram_gb=1 placeholder; the scrape side had the real value). The github.com/cigarplug/scrape-gsma URL was unioned so the repaired RAM is sourced: - 16 HP iPAQ pairs: h6310/h6315/h6320/h6325/hw6510/hw6515 -> 0.0625 GB (64 MB); hw6910/hw6915/rw6815/rw6818/rw6828/514 -> 0.0625; 610c/910c/data-messenger/ voice-messenger -> 0.125 GB (128 MB). - i-mate pda2 (two identical scrape twins, -1004 and -121) -> 0.125 GB. - i-mate pda2k -> 0.125 GB; i-mate pdal -> 0.062 GB. - palm treo-pro (3-way: bare + scrape-gsma-7066 + phonedb palm-treo-pro): kept the bare Wikipedia-cited record, ram 1 -> 0.125 GB (128 MB), name -> "Palm Treo Pro", unioned all four source_urls. Left unmerged and documented (ambiguous, per "leave ambiguous unmerged"): the GSM webOS palm-pre-scrapegsma-7065 and palm-pre-plus-scrapegsma-7063 overlap a region-split family (palm-pre[-plus]-palm-castle-c CDMA and palm-pre[-plus]-gsm-*-palm-castle-g EU/NA) but do not map 1:1, so no merge. The EU/NA regional variant pairs (pixi-gsm, pixi-plus-gsm, pre-2-gsm, pre[-plus]-gsm-castle-g) are legitimate separate SKUs and were left alone. No value was invented: every ram_gb repair took the real figure already present in the deleted twin's cited memory_internal. verified flags unchanged. Both gates pass: `python -m app.validate` and `integrity_check.py <data> --strict` (0 hard anomalies). Refs #296
Re-dumped the pdas collection from the deduped data tree (fresh scratch DB, TECHAPI_DATA_DIR on this worktree). pdas 140 -> 112: 28 merged-away detail pages removed, 21 survivor pages updated (repaired ram_gb, brand-inclusive names, unioned source_urls), and the top-level manifest pda count updated. No other collection touched.
github-actions
Bot
added
bug
Member
🔎 Data verification — Tier 0 (offline existence/trust)Scored by 12/12 categories ? 191,791 records ? diff base develop@ea3f31e Changed records in this PR24 record(s) assessed. %%{init: {"theme":"base","themeVariables":{"pie1":"#3fb950","pie2":"#d29922","pie3":"#f85149","pieStrokeWidth":"0px","pieOpacity":"1"}}}%%
pie showData
title Verification bands — all records
"Green" : 5
"Yellow" : 19
"Red" : 0
Full-dataset baseline191791 record(s) assessed. %%{init: {"theme":"base","themeVariables":{"pie1":"#3fb950","pie2":"#d29922","pie3":"#f85149","pieStrokeWidth":"0px","pieOpacity":"1"}}}%%
pie showData
title Verification bands — all records
"Green" : 32769
"Yellow" : 157967
"Red" : 1055
Hard violations (forced red):
green = authoritative source + complete + consistent · yellow = plausible, needs confirmation · red = sparse/weak source or a hard contradiction. Promotion to |
Member
TechEngine change review: PASS12/12 categories ? 191,791 records ? diff base develop@ea3f31e
Changed data
Changed record examplespda modified
pda deleted
Heuristic review
|
Member
TechEngine validation stats: PASS12/12 categories ? 191,791 records ? diff base develop@ea3f31e
Data summary
Warning Tracked verified coverage is below 50% for monitor 0.0% (0/882), software 0.8% (358/42493), laptop 5.3% (105/1995), tablet 8.3% (288/3459), soc 13.4% (284/2117), all 16.8% (32251/191791), smartphone 18.8% (17581/93407), website 19.0% (7635/40084), and 1 more. Validation notes
Key output:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Near-duplicate scan of the pda category (140 records — small enough to review
every match by hand). Merged 28 confirmed duplicate records into 21 keepers
(140 → 112 files), keeping the better-sourced side field by field and unioning
source_urls. Anything ambiguous was left unmerged and is documented below.Both gates pass on the merged tree:
python -m app.validate→ passedintegrity_check.py <data> --strict→ 0 hard anomalies (structural section empty)Per-pattern breakdown
The same three patterns the tablet/watch (#324) and smartphone (#329/#330) passes
turned up all appear here.
Pattern 1 — identical scrape imported twice under different row-id slugs
Two
cigarplug/scrape-gsmarows for the same device, byte-identical exceptsource_row. Kept the lower row-id, deleted the other (no field change needed):motorola/2002/accompli-388— kept…-scrapegsma-344, deleted…-1227mitac/2005/mio-a701— kept…-scrapegsma-278, deleted…-1161Pattern 2 — same device under a brand-prefixed slug and a bare slug (sibling dirs)
Same convention as the #330 GSMArena/phonedb merge: kept the bare GSMArena/Kaggle
record (real specs +
raw_gsmarena, sometimes Wikipedia), upgradednameto thebrand-inclusive form, unioned
source_urlsto retain the phonedb reference, keptthe keeper's
soc/release_date, leftverifiedas-is, deleted the phonedb twin:treo-680palm-treo-680-palm-lowridertreo-750vpalm-treo-750v-htc-cheetahtreo-750palm-treo-750-htc-cheetahtreo-500vpalm-treo-500v-palm-ottoTreo 750 vs 750v and Treo 500 vs 500v are distinct SKUs — only the
matching-model twins were merged.
palm-treo-500-palm-otto(2008 Treo 500) is adifferent device and was left untouched.
Pattern 3 — cross-source duplicate (clean short-slug record + scrape-gsma row, same base dir)
Kept the clean record (correct
socFK, canonical slug), deleted the scrape twin,and repaired the keeper's
ram_gbfrom the twin's citedmemory_internal— theclean side carried the fabricated
ram_gb = 1placeholder while the scrape side hadthe real figure.
github.com/cigarplug/scrape-gsmawas unioned so the repaired RAMis sourced. No value invented: in every pair the deleted twin already held the real
value.
rw6818/rw6828/514 →
0.0625GB (64 MB); 610c/910c/data-messenger/voice-messenger→
0.125GB (128 MB).pda2(two identical scrape twins-1004/-121) → 0.125 GB;pda2k→ 0.125 GB;pdal→ 0.062 GB.treo-pro(3-way: bare Wikipedia-cited +…-scrapegsma-7066+ phonedbpalm-treo-pro) — kept the bare record,ram_gb1 → 0.125 GB (128 MB), name →"Palm Treo Pro", unioned all four
source_urls.Left unmerged (ambiguous — documented, not guessed)
palm-pre-scrapegsma-7065(GSM webOS Pre, 2009-10) andpalm-pre-plus-scrapegsma-7063(GSM Pre Plus, 2010-03) overlap a region-split family —
palm-pre[-plus]-palm-castle-c(CDMA) and
palm-pre[-plus]-gsm-{eu,na}-palm-castle-g— but do not map 1:1, sono merge was made.
palm-pixi-gsm,palm-pixi-plus-gsm,palm-pre-2-gsm,palm-pre[-plus]-gsm-…-castle-g) are legitimate separate SKUs(different release dates + regions) and were left alone.
Systemic finding (reported, not mass-edited)
The same fabricated-defaults signature the earlier passes flagged is present here:
nearly every clean pda record carries
ram_gb = 1(real values are 16–256 MB),and many Palm records carry
os = "Feature phone"for what are Palm OS / webOSdevices. I only repaired
ram_gbinside the confirmed merges (where the deletedtwin supplied a cited real value); the remaining placeholder records were left
untouched rather than mass-edited, since fixing them needs a per-record source and
is out of scope for a dedup pass (per the "never invent values" rule).
Ownership / dump
data/pda/only, plus the mandatory dump refresh as the last commit (re-dumped thepdascollection from a fresh scratch DB withTECHAPI_DATA_DIRon this worktree;pdas 140 → 112, no other collection touched).
Closes #296 — this clears the pda slice of the "near-duplicates in other categories
not yet scanned" item, but that item stays open (smartphone still has 322
documented-unresolved pairs per #330, and other categories are not yet scanned), so
this is
Refs, notCloses.