mix f1000.import.frequency — rank the top N words per language #17

New issue

Open

opened 2026-09-26 13:58:33 +00:00 by belvedere · 0 comments

belvedere commented

2026-09-26 13:58:33 +00:00

Collaborator

Objective: Populate Word rows with real frequency ranks.

Source: hermitdave/FrequencyWords (MIT licence, ~1.5k stars). Layout:
content/2018/<lang>/<lang>_50k.txt, one word count pair per line, already sorted by
descending frequency. Example head: you 28787591, i 27086011, the 22761659.

Files:

Create: lib/mix/tasks/f1000.import.frequency.ex

Steps:

Fetch <lang>_50k.txt with Req (AGENTS.md forbids :httpoison/:tesla/:httpc).
Parse word count; drop noise — punctuation-only tokens, possessives like 's, digits.
Take the first N valid words after filtering to reach the 1000 target.
Upsert into Word keyed on (language_id, text) — idempotent: re-running must not
duplicate. Use Ash.bulk_create with upsert? true.
Support --language es --limit 1000 flags; cache the raw download under
priv/data/cache/ so re-imports do not re-download.

Verify:

mix f1000.import.frequency --language en --limit 1000
mix f1000.import.frequency --language en --limit 1000   # no duplicates, same row count

**Objective:** Populate `Word` rows with real frequency ranks. **Source:** `hermitdave/FrequencyWords` (MIT licence, ~1.5k stars). Layout: `content/2018/<lang>/<lang>_50k.txt`, one `word count` pair per line, already sorted by descending frequency. Example head: `you 28787591`, `i 27086011`, `the 22761659`. **Files:** - Create: `lib/mix/tasks/f1000.import.frequency.ex` **Steps:** 1. Fetch `<lang>_50k.txt` with **Req** (AGENTS.md forbids `:httpoison`/`:tesla`/`:httpc`). 2. Parse `word count`; drop noise — punctuation-only tokens, possessives like `'s`, digits. 3. Take the first N *valid* words after filtering to reach the 1000 target. 4. Upsert into `Word` keyed on `(language_id, text)` — **idempotent**: re-running must not duplicate. Use `Ash.bulk_create` with `upsert? true`. 5. Support `--language es --limit 1000` flags; cache the raw download under `priv/data/cache/` so re-imports do not re-download. **Verify:** ```bash mix f1000.import.frequency --language en --limit 1000 mix f1000.import.frequency --language en --limit 1000 # no duplicates, same row count ```

belvedere added this to the M2 — Lexicon model & data pipeline milestone

2026-09-26 13:58:33 +00:00

belvedere added the

labels

2026-09-26 13:58:33 +00:00

belvedere referenced this issue