fix(data): merge 651 confirmed monitor near-duplicate records - #333
Merged
Merged
Conversation
Collapse identical re-imports from the amazon-monitor Kaggle dataset where the same physical monitor was ingested many times under different amazonmon-<row> slugs with byte-identical specs. For each (base_model_slug + full spec fingerprint) cluster, keep the lowest source_row as canonical, union source_urls, and delete the redundant files. Also merge one cross-source duplicate (asus-va24ehe rows 60 & 404: same VA24EHE, two title strings) keeping the cleaner title and adopting the more complete adaptive_sync=FreeSync attribute. 882 -> 231 monitor records. Genuine SKU/config variants left unmerged. Refs #296
Regenerate site/public/v1/monitors from the deduplicated data (python -m app.dump): drop the 651 orphaned detail pages for the merged-away records, rebuild the monitors list (882 -> 231), refresh the asus-va24ehe detail page with the merged adaptive_sync attribute, and update the monitor count in the v1 manifest. Scoped to monitor-attributable dump output only. Refs #296
github-actions
Bot
added
bug
Member
🔎 Data verification — Tier 0 (offline existence/trust)Scored by 12/12 categories ? 191,168 records ? diff base develop@ea3f31e Changed records in this PR1 record(s) assessed. %%{init: {"theme":"base","themeVariables":{"pie1":"#3fb950","pie2":"#d29922","pie3":"#f85149","pieStrokeWidth":"0px","pieOpacity":"1"}}}%%
pie showData
title Verification bands — all records
"Green" : 0
"Yellow" : 1
"Red" : 0
Full-dataset baseline191168 record(s) assessed. %%{init: {"theme":"base","themeVariables":{"pie1":"#3fb950","pie2":"#d29922","pie3":"#f85149","pieStrokeWidth":"0px","pieOpacity":"1"}}}%%
pie showData
title Verification bands — all records
"Green" : 32772
"Yellow" : 157341
"Red" : 1055
Hard violations (forced red):
green = authoritative source + complete + consistent · yellow = plausible, needs confirmation · red = sparse/weak source or a hard contradiction. Promotion to |
Member
TechEngine change review: PASS12/12 categories ? 191,168 records ? diff base develop@ea3f31e
Changed data
Changed record examplesmonitor modified
monitor deleted
Heuristic review
|
Member
TechEngine validation stats: PASS12/12 categories ? 191,168 records ? diff base develop@ea3f31e
Data summary
Warning Tracked verified coverage is below 50% for monitor 0.0% (0/231), software 0.8% (358/42493), laptop 5.3% (105/1995), tablet 8.3% (288/3459), soc 13.4% (284/2117), all 16.9% (32257/191168), smartphone 18.8% (17581/93407), website 19.0% (7635/40084), and 1 more. Validation notes
Key output:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Continuing the "near-duplicates in other categories not yet scanned" item after the tablet/watch (#324), smartphone (#329) and laptop (#332) passes, I scanned the monitor category. Scope: 882
data/monitor/<brand>/<year>/<base_model_slug>/<slug>.jsonrecords, all sourced from one Kaggle Amazon monitor dataset (amazon-monitor-durjoychandrapaul).Result: 651 confirmed near-duplicate records merged.
882 → 231.Method
Same approach as the prior passes: normalize slugs/names (punctuation- and brand-prefix-agnostic) and cross-check the specs that must be identical for the same physical panel —
release_date,size_inch,resolution,aspect_ratio,refresh_hz,panel_type,ppi, plus the model number embedded inname. The deterministic merge rule: group bybase_model_slug+ a full spec fingerprint (every field exceptvariant,slug,source_urls); for each cluster keep the lowestsource_rowas canonical, unionsource_urls, and delete the rest.Per-pattern results
slugandvariant.source_row. Kept the lowestsource_row, deleted the redundant files;source_urlswere identical so the union was a no-op and no values changed. Breakdown:asus-va329he84→1 (−83)aopen-aopen-by-acer(32SA2QK 4K) 69→1 (−68)acer-cb27263→1 (−62)acer-sh242y60→1 (−59)acer-xb273k60→1 (−59)acer-xz342cu47→1 (−46)acer-kb252q45→1 (−44)acer-sb240y(G0bi) 44→1 (−43)acer-sb272(EBI) 39→1 (−38)acer-xz320qu31→1 (−30)acer-hdr10(31.5" Nitro 240Hz, two byte-identical price clusters) 74→2 (−72)acer-sb27027→1 (−26)acer-kc242y14→1 (−13)acer-xb273u7→1 (−6)dell-u2723qevsu2723qe): none. Every record already lives under a consistent brand-prefixed base dir.asus-va24eherows 60 and 404 are the same ASUS VA24EHE (23.8", 1920×1080, 75 Hz, IPS, $99) imported with two different Amazon title strings. Kept row 60 (cleaner, untruncated title) and adopted the more completefeatures.adaptive_sync = "FreeSync"from row 404 — a real, documented VA24EHE spec (ASUS product page). No other field differed.Left unmerged (genuine distinct SKUs / ambiguous — documented, not guessed)
These share a coarse
base_model_slugbut are different real products, so they were intentionally kept:acer-eda343cur— 100 Hz/$200 vs 180 Hz/$280 (different refresh SKUs)acer-sb240y— G0bi (black, $95) vs G0wi (white, $110)acer-sb242y— EBI (IPS) vs Hbi ($128)acer-sb272— EBI (new) vs "E … (Renewed)" listingaopen-aopen-by-acer— 32SA2QK (31.5" 4K) vs 27E1 (27" FHD)gigabyte-m27q— M27Q vs M27Q PROhp-hp-series-monitor— Series 5 24" vs 524sa bundlelenovo-lenovo-thinkvision-full— ThinkVision M15 (15.6") vs T22i-30 (21.5")lg-lg-digital-signage— 49UH5F-H (49") vs 86UH5E-B (86")lg-lg-ultragear-gaming— 27" QHD 240 Hz vs 27" 5120×1440 165 Hzlilliput-fa1014-np— 1280×800 non-touch vs 1920×1080 touchsamsung-hdr2000— Odyssey Neo G7 vs Neo G8acer-hdr10residual — thehdr10slug is an importer fallback shared by a few genuinely different Nitros (27" WQHD 170 Hz, 27" FHD 240 Hz, and the 31.5" 240 Hz at two prices); left as-is rather than guess.Placeholder / fabricated-defaults check: clean (no bug)
No equivalent of the phone-batch
battery_mah/weight_gstamping. Spec value distributions across distinct base models are organic (27" popular; 1920×1080 common; 100/165/75 Hz common), each spread over many unrelated models — not one value stamped onto unrelated devices. The one systematic artifact isrelease_date = "2023-01-01"on all 882 records, but that is a uniform dataset-wide import placeholder that affects every record equally, so it is not a dedup signal and is out of scope for this merge. Noting it here for the "placeholder dates" tracker item.Verification
python -m app.validate→ Data validation passed (monitor required-field / slug-convention / filename-match / base-dir checks included).python integrity_check.py <data> --strict→ no hard anomalies (structural section clean: no duplicate monitor slugs, no slug/filename mismatches, no verified-without-source); only the pre-existing advisory CPU/GPU ratio outliers print, which the gate treats as review-only.python -m app.dump), scoped to the monitor collection: the 651 orphaned detail pages were pruned,v1/monitors/index.jsonrebuilt (231), theasus-va24ehedetail page picked up the mergedadaptive_sync, and thev1manifest monitor count updated882 → 231. Unrelated pre-existing serialization drift in other collections was deliberately kept out to keep this diff monitor-scoped and reviewable.Closes #296