fix(data): merge 183 confirmed tablet/watch near-duplicate records - #324
Merged
Merged
Conversation
github-actions
Bot
added
bug
5 tasks done
Seungpyo1007
marked this pull request as ready for review
September 28, 2026 21:10
This was referenced Sep 28, 2026
Seungpyo1007
added a commit
that referenced
this pull request
Sep 29, 2026
Scanned the pda category (140 records, reviewed every match by hand) for the three duplicate patterns the tablet/watch (#324) and smartphone (#329/#330) passes turned up. Merged 28 redundant files into 21 keepers (140 -> 112), keeping the better-sourced side field by field and unioning source_urls. Pattern 1 - identical scrape imported twice under different row-id slugs (100% identical specs, differ only by source_row): - motorola/2002/accompli-388: kept ...-scrapegsma-344, deleted ...-1227. - mitac/2005/mio-a701: kept ...-scrapegsma-278, deleted ...-1161. Pattern 2 - same device under a brand-prefixed slug and a bare slug in sibling dirs. Same convention as the #330 GSMArena/phonedb merge: kept the bare GSMArena/Kaggle record (real specs + raw_gsmarena, sometimes Wikipedia), upgraded name to the brand-inclusive form, unioned source_urls to retain the phonedb reference, kept the keeper's soc/release_date, left verified as-is, deleted the phonedb twin: - treo-680 <- palm-treo-680-palm-lowrider ("Palm Treo 680") - treo-750v <- palm-treo-750v-htc-cheetah ("Palm Treo 750v") - treo-750 <- palm-treo-750-htc-cheetah ("Palm Treo 750") - treo-500v <- palm-treo-500v-palm-otto ("Palm Treo 500v") Treo 750 vs 750v and Treo 500 vs 500v were kept as distinct SKUs (only the matching-model twins were merged). palm-treo-500-palm-otto (2008 Treo 500) is a different device and was left alone. Pattern 3 - cross-source duplicate: the same PDA imported as a clean short-slug record and as a cigarplug/scrape-gsma row in the same base dir. Kept the clean record (correct soc FK, canonical slug), deleted the scrape twin, and repaired the keeper's ram_gb from the twin's cited memory_internal (the clean side carried the fabricated ram_gb=1 placeholder; the scrape side had the real value). The github.com/cigarplug/scrape-gsma URL was unioned so the repaired RAM is sourced: - 16 HP iPAQ pairs: h6310/h6315/h6320/h6325/hw6510/hw6515 -> 0.0625 GB (64 MB); hw6910/hw6915/rw6815/rw6818/rw6828/514 -> 0.0625; 610c/910c/data-messenger/ voice-messenger -> 0.125 GB (128 MB). - i-mate pda2 (two identical scrape twins, -1004 and -121) -> 0.125 GB. - i-mate pda2k -> 0.125 GB; i-mate pdal -> 0.062 GB. - palm treo-pro (3-way: bare + scrape-gsma-7066 + phonedb palm-treo-pro): kept the bare Wikipedia-cited record, ram 1 -> 0.125 GB (128 MB), name -> "Palm Treo Pro", unioned all four source_urls. Left unmerged and documented (ambiguous, per "leave ambiguous unmerged"): the GSM webOS palm-pre-scrapegsma-7065 and palm-pre-plus-scrapegsma-7063 overlap a region-split family (palm-pre[-plus]-palm-castle-c CDMA and palm-pre[-plus]-gsm-*-palm-castle-g EU/NA) but do not map 1:1, so no merge. The EU/NA regional variant pairs (pixi-gsm, pixi-plus-gsm, pre-2-gsm, pre[-plus]-gsm-castle-g) are legitimate separate SKUs and were left alone. No value was invented: every ram_gb repair took the real figure already present in the deleted twin's cited memory_internal. verified flags unchanged. Both gates pass: `python -m app.validate` and `integrity_check.py <data> --strict` (0 hard anomalies). Refs #296
This was referenced Sep 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Remove 183 duplicate records across 182 verified tablet/watch groups, retaining the more complete record, unioning source URLs, and preserving differing original source data under
raw_merged_records.The investigation comment abbreviates its group list; these counts reflect individually verified matches rather than forcing its reported total of 192. Ambiguous RAM tiers, year splits, LTE/Wi-Fi distinctions, Surface Duo, and the MediaPad Vogue discrepancy remain unmerged. Three Prestigio 7.0 HD/Prime/Ultra Plus revisions are distinct devices and remain separate; Watch Magic remains unmerged because its OS could not be sourced confidently.
Repair the repeated 1500 mAh / 150 g import placeholders: 170 fields were replaced using same-device sibling evidence before merging, and 362 fields with no unambiguous replacement are null. Resolve nine watch RAM discrepancies and ten pixel-density discrepancies from the original CSV, rebuild tablet storage options without RAM entries, and correct the iPad Pro 12.9 (2015) battery from 307 to 10,307 mAh after checking the comma-separated source value. Synchronize representative variant memory when corrected.
Closes #296
Source
Existing references are retained, and references used for corrections are included in the changed records.
Validation
python -m app.validatepassed across the repository.git diff --checkpassed.Checklist
develop