Skip to content

fix(data): correct storage, battery and display fields from each record's own text - #301

Merged
Seungpyo1007 merged 1 commit into
developfrom
fix/e4-self-consistency
Sep 26, 2026
Merged

Seungpyo1007 merged 1 commit into
developfrom
fix/e4-self-consistency

Conversation

@Seungpyo1007

@Seungpyo1007 Seungpyo1007 commented Sep 26, 2026 •

Copy link
Copy Markdown
Member

What & why

Self-consistency fixes: every corrected value is derived from text already in the same record, with no outside lookups. Closes #296.

  • Tablet/watch storage (259 records): storage_options_gb held the RAM size mixed into the storage list (e.g. [1, 16] for "8/16 GB, 1 GB RAM"). Re-parsed from memory_internal. Records whose raw text lists several editions ("16 GB, 2 GB RAM - Standard edition 32 GB, 3 GB RAM - Premium") were left untouched, because a single-list parse would drop an edition.
  • iPad Pro 12.9 (2015, 2017): battery_mah had lost the thousands digit (307 / 891); restored to 10,307 / 10,891 from the raw battery text.
  • 3 laptops: storage_gb now matches the configuration in the model name (e.g. i7-7500U/8GB/128GB).
  • Monitors: size_inch (3) and resolution (2) now match the listing name. aspect_ratio is nulled where it contradicts the pixel resolution (24, e.g. 78:1 for 1920x1080).

Not changed: 43 devices whose release date is more than 2 years before their chip, and 15 records whose slug RAM disagrees with ram_gb. The correct value isn't derivable from the record, so they need sourcing.

Source

Each record's own memory_internal, raw battery text and product name.

Checklist

  • The PR targets develop
  • python -m app.validate passes locally
  • Files live at the correct path
  • No slugs changed
  • Sources unchanged (values derived from existing cited records)

…itor size/resolution from each record's own text

- tablet/watch storage_options_gb re-parsed from memory_internal where the
  import mixed the RAM size into the storage list (259 records)
- iPad Pro 12.9 (2015, 2017) battery_mah restored from the raw battery text
- 3 laptops: storage_gb taken from the configuration in the model name
- monitors: size_inch and resolution taken from the listing name (5), and
  aspect_ratio nulled where it contradicts the pixel resolution (24)

Refs #296
@github-actions github-actions Bot added bug Something isn't working data Dataset changes enhancement New feature or request labels Sep 26, 2026
@TechEngineBot

Copy link
Copy Markdown
Member

🔎 Data verification — Tier 0 (offline existence/trust)

Scored by app.verify; posted by TechEngineBot. Informational only — the structural gate (app.validate) is separate and authoritative for merge.

12/12 categories ? 191,814 records ? diff base develop@6f4b273

Changed records in this PR

291 record(s) assessed.

Software and website assess required fields and sources only; domain consistency rules are unavailable and these categories cannot earn green.

%%{init: {"theme":"base","themeVariables":{"pie1":"#3fb950","pie2":"#d29922","pie3":"#f85149","pieStrokeWidth":"0px","pieOpacity":"1"}}}%%
pie showData
    title Verification bands — all records
    "Green" : 8
    "Yellow" : 283
    "Red" : 0
Loading
Category Distribution Total 🟢 🟡 🔴 🟢 %
tablet 🟩🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨 255 8 247 0 3.1%
watch 🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨 6 0 6 0 0.0%
laptop 🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨 3 0 3 0 0.0%
monitor 🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨 27 0 27 0 0.0%
All 🟩🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨 291 8 283 0 2.7%

Full-dataset baseline

191814 record(s) assessed.

Software and website assess required fields and sources only; domain consistency rules are unavailable and these categories cannot earn green.

%%{init: {"theme":"base","themeVariables":{"pie1":"#3fb950","pie2":"#d29922","pie3":"#f85149","pieStrokeWidth":"0px","pieOpacity":"1"}}}%%
pie showData
    title Verification bands — all records
    "Green" : 24725
    "Yellow" : 166034
    "Red" : 1055
Loading
Category Distribution Total 🟢 🟡 🔴 🟢 %
brand 🟩🟩🟩🟩🟩🟩🟩🟨🟨🟨🟨🟨 207 122 85 0 58.9%
soc 🟩🟩🟨🟨🟨🟨🟨🟥🟥🟥🟥🟥 2117 284 837 996 13.4%
smartphone 🟩🟩🟨🟨🟨🟨🟨🟨🟨🟨🟨🟥 93407 18003 75350 54 19.3%
tablet 🟩🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟥 3459 292 3166 1 8.4%
watch 🟩🟩🟩🟨🟨🟨🟨🟨🟨🟨🟨🟥 447 94 350 3 21.0%
pda 🟩🟩🟩🟩🟩🟩🟩🟨🟨🟨🟨🟨 140 83 57 0 59.3%
gpu 🟩🟩🟩🟩🟩🟩🟩🟩🟩🟨🟨🟨 2076 1529 547 0 73.7%
cpu 🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟨🟥 4507 4211 295 1 93.4%
laptop 🟩🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨 1995 107 1888 0 5.4%
monitor 🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨 882 0 882 0 0.0%
software 🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨 42493 0 42493 0 0.0%
website 🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨 40084 0 40084 0 0.0%
All 🟩🟩🟨🟨🟨🟨🟨🟨🟨🟨🟨🟥 191814 24725 166034 1055 12.9%

Hard violations (forced red):

Count Check
1 !boost_ge_base

green = authoritative source + complete + consistent · yellow = plausible, needs confirmation · red = sparse/weak source or a hard contradiction. Promotion to verified runs in the scheduled verify-network workflow.

@Seungpyo1007
Seungpyo1007 merged commit a658758 into develop Sep 26, 2026
5 checks passed
@github-project-automation github-project-automation Bot moved this from In Progress to Done in TechAPI-Project Sep 26, 2026
@TechEngineBot

Copy link
Copy Markdown
Member

TechEngine change review: PASS

12/12 categories ? 191,814 records ? diff base develop@6f4b273

Check Result
python -m app.validate PASS
Changed dump JSON and manifest/index counts PASS
New hard integrity anomalies vs PR base PASS

Changed data

Category Added Modified Deleted Added verified Added unverified Added Kaggle-sourced
brand 0 0 0 0 0 0
soc 0 0 0 0 0 0
smartphone 0 0 0 0 0 0
tablet 0 255 0 0 0 0
watch 0 6 0 0 0 0
pda 0 0 0 0 0 0
gpu 0 0 0 0 0 0
cpu 0 0 0 0 0 0
laptop 0 3 0 0 0 0
monitor 0 27 0 0 0 0
software 0 0 0 0 0 0
website 0 0 0 0 0 0

Changed record examples

tablet modified

  • tablet/acer/2011/iconia-tab-a100/acer-iconia-tab-a100-scrapegsma-1774.json - Acer Iconia Tab A100
  • tablet/acer/2011/iconia-tab-a101/acer-iconia-tab-a101-scrapegsma-1773.json - Acer Iconia Tab A101
  • tablet/acer/2011/iconia-tab-a500/acer-iconia-tab-a500-scrapegsma-1772.json - Acer Iconia Tab A500
  • tablet/acer/2011/iconia-tab-a501/acer-iconia-tab-a501-scrapegsma-1770.json - Acer Iconia Tab A501
  • tablet/acer/2012/iconia-tab-a110/acer-iconia-tab-a110-scrapegsma-1756.json - Acer Iconia Tab A110
  • tablet/acer/2012/iconia-tab-a200/acer-iconia-tab-a200-scrapegsma-1763.json - Acer Iconia Tab A200
  • tablet/acer/2012/iconia-tab-a210/acer-iconia-tab-a210-scrapegsma-1762.json - Acer Iconia Tab A210
  • tablet/acer/2012/iconia-tab-a510/acer-iconia-tab-a510-scrapegsma-1767.json - Acer Iconia Tab A510
  • tablet/acer/2012/iconia-tab-a511/acer-iconia-tab-a511-scrapegsma-1766.json - Acer Iconia Tab A511
  • tablet/acer/2012/iconia-tab-a700/acer-iconia-tab-a700-scrapegsma-1765.json - Acer Iconia Tab A700
  • tablet/acer/2012/iconia-tab-a701/acer-iconia-tab-a701-scrapegsma-1764.json - Acer Iconia Tab A701
  • tablet/acer/2013/iconia-tab-a1-810/acer-iconia-tab-a1-810-scrapegsma-1749.json - Acer Iconia Tab A1-810
  • tablet/acer/2013/iconia-tab-a1-811/acer-iconia-tab-a1-811-scrapegsma-1748.json - Acer Iconia Tab A1-811
  • tablet/acer/2013/iconia-tab-a3/acer-iconia-tab-a3-scrapegsma-1747.json - Acer Iconia Tab A3
  • tablet/acer/2013/iconia-tab-b1-710/acer-iconia-tab-b1-710-scrapegsma-1754.json - Acer Iconia Tab B1-710
  • ... 240 more

watch modified

  • watch/apple/2017/watch-series-3-aluminum/apple-watch-series-3-aluminum-scrapegsma-2359.json - Apple Watch Series 3 Aluminum
  • watch/lg/2015/watch-urbane-lte/lg-watch-urbane-lte-scrapegsma-1082.json - LG Watch Urbane LTE
  • watch/lg/2015/watch-urbane-lte/lg-watch-urbane-lte-scrapegsma-199.json - LG Watch Urbane LTE
  • watch/samsung/2018/galaxy-watch/samsung-galaxy-watch-scrapegsma-7867.json - Samsung Galaxy Watch
  • watch/samsung/2019/galaxy-watch-active2/samsung-galaxy-watch-active2-scrapegsma-1501.json - Samsung Galaxy Watch Active2
  • watch/xiaomi/2019/mi-watch/xiaomi-mi-watch-scrapegsma-832.json - Xiaomi Mi Watch

laptop modified

  • laptop/asus/2017/asus-f756ux-t4201d-i7-7500u-8gb-128gb/asus-f756ux-t4201d-i7-7500u-8gb-128gb-ammok-243.json - Asus F756UX-T4201D (i7-7500U/8GB/128GB
  • laptop/asus/2017/asus-k756ux-t4340t-i5-7200u-8gb-500gb/asus-k756ux-t4340t-i5-7200u-8gb-500gb-ammok-229.json - Asus K756UX-T4340T (i5-7200U/8GB/500GB
  • laptop/asus/2017/asus-ux510ux-cn269t-i7-7500u-8gb-256gb/asus-ux510ux-cn269t-i7-7500u-8gb-256gb-ammok-333.json - Asus UX510UX-CN269T (i7-7500U/8GB/256GB

monitor modified

  • monitor/acer/2023/acer-eda343cur/acer-eda343cur-amazonmon-177.json - acer Nitro 34" UWQHD 3440 x 1440 1000R Curved PC Gaming Monitor | AMD FreeSync Premium | Up to 180Hz Refresh | Up to 0.5
  • monitor/asus/2023/asus-ve278h/asus-ve278h-amazonmon-844.json - ASUS VE278H 27" Full HD 1920x1080 2ms HDMI VGA Back-lit LED Monitor
  • monitor/asus/2023/asus-vg328h1b/asus-vg328h1b-amazonmon-653.json - ASUS TUF Gaming VG328H1B 32” Curved Monitor, 1080P Full HD, 165Hz (Supports 144Hz), Extreme Low Motion Blur, Adaptive-sy
  • monitor/dell/2023/dell-dell-inspiron-touchscreen/dell-dell-inspiron-touchscreen-amazonmon-206.json - Dell Inspiron 24 5400 23.8" FHD Touchscreen All-in-One Computer - 11th Gen Intel
  • monitor/dell/2023/dell-p3421w/dell-p3421w-amazonmon-384.json - Dell P3421W 34.1" QHD Wide 1440p WQHD Curved Screen WLED LCD Monitor - 21:9 - Black
  • monitor/dell/2023/dell-s2240m/dell-s2240m-amazonmon-802.json - Dell S2240M 21.5-Inch Screen LED-lit Monitor
  • monitor/gigabyte/2023/gigabyte-gs34wqc/gigabyte-gs34wqc-amazonmon-779.json - Gigabyte GS34WQC 34" 120Hz 1440P Curved Gaming Monitor, 3440x1440 VA 1500R Display, 1ms (MPRT) Response Time, HDR Ready,
  • monitor/innocn/2023/innocn-hdr400/innocn-hdr400-amazonmon-218.json - INNOCN 44C1G Ultrawide Monitor 43.8" WFHD 3840 x 1080p 120Hz Monitor 32:9 IPS Computer Split Screen Monitor 96% DCI-P3 H
  • monitor/innoview/2023/innoview-innoview-portable-screen/innoview-innoview-portable-screen-amazonmon-669.json - InnoView 2Pack 4k Portable Screen Nontouch & 10 Point Touch Screen Monitor 3840x2160P 350Nits USB C Monitor
  • monitor/lenovo/2023/lenovo-lenovo-thinkvision-full/lenovo-lenovo-thinkvision-full-amazonmon-465.json - Lenovo ThinkVision T22i-30 21.5 Full HD WLED LCD Monitor - 16:9 - Raven Black
  • monitor/lenovo/2023/lenovo-p24h-30/lenovo-p24h-30-amazonmon-434.json - Lenovo 23.8IN IPS 2560X1440 1000:1 THINKVISION P24H-30 HDMI DP USB BLK
  • monitor/lg/2023/lg-34bq77qb/lg-34bq77qb-amazonmon-637.json - LG 34BQ77QB-B-CR (Renewed)
  • monitor/lg/2023/lg-34gp63a/lg-34gp63a-amazonmon-109.json - LG UltraGear QHD 34-Inch Curved Gaming Monitor 34GP63A-B, VA with HDR 10 Compatibility and AMD FreeSync Premium, 160Hz,
  • monitor/lg/2023/lg-34wl500/lg-34wl500-amazonmon-578.json - LG 34WL500-B 34" 21:9 UltraWide Full HD HDR10 IPS LED Monitor, Black
  • monitor/lg/2023/lg-38bq88c/lg-38bq88c-amazonmon-648.json - LG 38BQ88C-W 38" UltraWide QHD+ HDR Curved Monitor with Ergo Stand 3840X1600 USB-C HDMI DP Monitor
  • ... 12 more

Heuristic review

  • Heuristic warnings: none found.

@TechEngineBot

Copy link
Copy Markdown
Member

TechEngine validation stats: PASS

12/12 categories ? 191,814 records ? diff base develop@6f4b273

Data summary

Category Total Verified Unverified Missing verified Tracked Verified % of tracked
brand 207 122 85 0 207 58.9%
soc 2117 193 1924 0 2117 9.1%
smartphone 93407 17501 75906 0 93407 18.7%
tablet 3459 273 3186 0 3459 7.9%
watch 447 39 408 0 447 8.7%
pda 140 74 66 0 140 52.9%
gpu 2076 1026 1050 0 2076 49.4%
cpu 4507 2869 1638 0 4507 63.7%
laptop 1995 0 1995 0 1995 0.0%
monitor 882 0 882 0 882 0.0%
software 42493 0 42493 0 42493 0.0%
website 40084 0 40084 0 40084 0.0%
all 191814 22097 169717 0 191814 11.5%

Warning

Tracked verified coverage is below 50% for laptop 0.0% (0/1995), monitor 0.0% (0/882), software 0.0% (0/42493), website 0.0% (0/40084), tablet 7.9% (273/3459), watch 8.7% (39/447), soc 9.1% (193/2117), all 11.5% (22097/191814), and 2 more.
Tracked coverage excludes records missing the verified field; see the Missing verified column for those records.
This does not fail validation. Keep imported records verified: false until manual audit, but treat this as follow-up verification work before relying on the affected categories as curated data.

Validation notes

  • The integrity gate blocks only hard anomalies introduced by this PR relative to its merge base.
  • Full advisory outlier listings are suppressed on successful runs because they are dataset-wide and mostly stable between PRs.
  • Failure runs still include a detailed log excerpt for debugging.

Key output:

## app.validate
## integrity_check.py (PR head compared with PR base)
loaded CPU=4507 GPU=2076
Integrity section Flagged lines
structural 18
CPU name/tier consistency (desktop mainstream only) 0
CPU single>multi (cinebench/geekbench — should be multi>=single) 0
CPU era-vs-score outliers 8
CPU cross-source ratio outliers (possible wrong-variant) 131
GPU cross-source ratio outliers + sanity 51

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working data Dataset changes enhancement New feature or request

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

Data accuracy: corrections, duplicates and dates (workstream)

2 participants