Skip to content

data(website): seed the new website category with 40,084 Wikidata records - #138

Merged
Seungpyo1007 merged 2 commits into
mainfrom
data/website-category
Jul 30, 2026
Merged

Seungpyo1007 merged 2 commits into
mainfrom
data/website-category

Conversation

@Seungpyo1007

@Seungpyo1007 Seungpyo1007 commented Jul 30, 2026 •

Copy link
Copy Markdown
Member

What

Seeds the website category added in TechEngine #48 with 40,084 records from Wikidata's Q35127 subtree, bumps the submodule to the engine that serves them, and publishes the dump.

Dataset: 1,113,452 → 1,153,536 records, and the API gains a 13th collection at /v1/websites.

Why this source

The dataset's own device categories are exhausted, so the question was what adds genuinely new entities rather than more of what we already hold. Two candidates were measured before committing to either:

  • Steam / SteamSpy top-up of game — rejected. 90% of its titles already exist, and the apparent 9% remainder is mostly name variants of records we have (PUBG: BATTLEGROUNDS vs our playerunknowns-battlegrounds, Grand Theft Auto V Legacy vs grand-theft-auto-v). Importing it would have inflated the count with duplicate entities.
  • Wikidata websites — 47,631 items in the subtree, no overlap with any existing category, and 89% carry an official URL. That is far better populated than the electronics categories probed earlier this session (digital cameras: 34 items total).

What was imported

47,631 QIDs → 40,273 English-labelled items → 40,084 records. 189 dropped for an unusable name, 243 slug collisions took a -<qid> suffix, and Windows-reserved stems (CON, PRN, LPT1…) are skipped — a con.json from the game import previously made the tree unindexable by git.

Fields are written only where the source had them: homepage_url (85% of imported rows), launch_date, languages, owners. Everything is verified: false and cites its Wikidata item.

Notes on the dump

  • The submodule moves to the engine commit carrying both the category and the seed fix from TechEngine data(soc): GPU enrichment round 2 (13 older chips) #49 — without that fix, seeding this tree failed on the first insert (SQLite Date type only accepts Python date objects as input) because the loader coerced dates by the field name release_date and this category's column is launch_date.
  • A partial dump writes a manifest describing only the collection it generated, so the websites entry was merged into the real site/public/v1/index.json rather than replacing it — the other twelve collections are intact (verified in the diff).

python -m app.validate → Data validation passed (the bundled validator gained the matching WEBSITE_REQUIRED rule).

Closes #1

Seeds the new `website` category (TechEngine §6.12) with the Q35127 subtree:
40,084 records, all verified:false, each citing its Wikidata item.

Fields are written only where the source had them — homepage_url on 85%,
plus launch_date, languages and owners where present. Nothing is inferred.

Also mirrors the engine's WEBSITE_REQUIRED rule into the bundled validator
{slug, name, source_urls, verified} with slug, launch_date and source-url checks.

Import notes: 47,631 QIDs in the subtree yielded 40,273 English-labelled items;
189 were dropped for an unusable name, and 243 slug collisions took a
-<qid> suffix. Windows-reserved stems (CON, PRN, LPT1 …) are skipped by the
importer — a `con.json` from the game import previously made the tree
unindexable by git.

Refs #1
Bumps the TechEngine submodule to the commit that carries the website category
(§6.12) plus the seed date-coercion fix, and publishes site/public/v1/websites
for all 40,084 records.

The manifest keeps its existing twelve collections and gains websites; the
partial dump writes a manifest describing only what it generated, so the entry
was merged into the real one rather than replacing it.

Refs #1
@Seungpyo1007 Seungpyo1007 added data Dataset changes enhancement New feature or request labels Jul 30, 2026
@github-actions github-actions Bot added the app Validator or application code changes label Jul 30, 2026
@Seungpyo1007 Seungpyo1007 moved this from Todo to In Progress in TechAPI-Project Jul 30, 2026
@Seungpyo1007
Seungpyo1007 merged commit 13b566c into main Jul 30, 2026
5 checks passed
@github-project-automation github-project-automation Bot moved this from In Progress to Done in TechAPI-Project Jul 30, 2026
@Seungpyo1007
Seungpyo1007 deleted the data/website-category branch July 30, 2026 09:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

app Validator or application code changes data Dataset changes enhancement New feature or request

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

TechAPI dataset roadmap and status: all categories (1989-2026)

1 participant