Lexicon.Word resource with frequency ranking #66

New issue

Closed

opened 2026-09-26 13:58:57 +00:00 by belvedere · 1 comment

belvedere commented

2026-09-26 13:58:57 +00:00

Collaborator

Objective: One row per surface form per language, carrying its corpus frequency rank.

Files:

Create: lib/first_thousand_words/lexicon/word.ex

Steps:

Attributes: text, normalized (downcased, unaccented, for matching/typing checks),
lemma, part_of_speech, frequency_rank (integer), frequency_count (integer).
Relationships: belongs_to :language.
Identity: unique on (language_id, text). Index on (language_id, frequency_rank).
cast must not include language_id (AGENTS.md: programmatic fields are set explicitly).

Verify: an IEx query returns the top 20 English words ordered by rank.

Pitfall: the upstream corpus contains non-word noise tokens ('s, 't, single punctuation,
numbers). Filtering belongs in the importer, but the schema must tolerate whatever survives.

**Objective:** One row per surface form per language, carrying its corpus frequency rank. **Files:** - Create: `lib/first_thousand_words/lexicon/word.ex` **Steps:** 1. Attributes: `text`, `normalized` (downcased, unaccented, for matching/typing checks), `lemma`, `part_of_speech`, `frequency_rank` (integer), `frequency_count` (integer). 2. Relationships: `belongs_to :language`. 3. Identity: unique on `(language_id, text)`. Index on `(language_id, frequency_rank)`. 4. `cast` must not include `language_id` (AGENTS.md: programmatic fields are set explicitly). **Verify:** an IEx query returns the top 20 English words ordered by rank. **Pitfall:** the upstream corpus contains non-word noise tokens (`'s`, `'t`, single punctuation, numbers). Filtering belongs in the importer, but the schema must tolerate whatever survives.

belvedere added this to the M2 — Lexicon model & data pipeline milestone

2026-09-26 13:58:57 +00:00

belvedere added the

labels

2026-09-26 13:58:57 +00:00

belvedere commented

2026-09-26 14:02:00 +00:00

Author

Collaborator

Duplicate of #14 — created by a re-run whose pagination helper only saw the first page of issues. Closing to keep the backlog clean. Repo owner can delete this outright.

belvedere closed this issue

2026-09-26 14:02:00 +00:00