nickkeers/first-thousand-words
Watch
1
0
Fork
You've already forked first-thousand-words
 0
Code Issues 51 Pull requests Projects 1 Releases Packages Wiki Activity Actions

Lexicon.Word resource with frequency ranking #14

New issue
Open
opened 2026-09-26 13:58:32 +00:00 by belvedere · 0 comments
belvedere commented 2026-09-26 13:58:32 +00:00
Collaborator
Copy link

Objective: One row per surface form per language, carrying its corpus frequency rank.

Files:

  • Create: lib/first_thousand_words/lexicon/word.ex

Steps:

  1. Attributes: text, normalized (downcased, unaccented, for matching/typing checks),
    lemma, part_of_speech, frequency_rank (integer), frequency_count (integer).
  2. Relationships: belongs_to :language.
  3. Identity: unique on (language_id, text). Index on (language_id, frequency_rank).
  4. cast must not include language_id (AGENTS.md: programmatic fields are set explicitly).

Verify: an IEx query returns the top 20 English words ordered by rank.

Pitfall: the upstream corpus contains non-word noise tokens ('s, 't, single punctuation,
numbers). Filtering belongs in the importer, but the schema must tolerate whatever survives.

**Objective:** One row per surface form per language, carrying its corpus frequency rank. **Files:** - Create: `lib/first_thousand_words/lexicon/word.ex` **Steps:** 1. Attributes: `text`, `normalized` (downcased, unaccented, for matching/typing checks), `lemma`, `part_of_speech`, `frequency_rank` (integer), `frequency_count` (integer). 2. Relationships: `belongs_to :language`. 3. Identity: unique on `(language_id, text)`. Index on `(language_id, frequency_rank)`. 4. `cast` must not include `language_id` (AGENTS.md: programmatic fields are set explicitly). **Verify:** an IEx query returns the top 20 English words ordered by rank. **Pitfall:** the upstream corpus contains non-word noise tokens (`'s`, `'t`, single punctuation, numbers). Filtering belongs in the importer, but the schema must tolerate whatever survives.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
area
auth
 Authentication and authorisation

  
area
data
 Word frequency + translation import pipeline

  
area
domain
 Ash resources, domains, policies, migrations

  
area
infra
 Releases, deployment, CI

  
area
stats
 Progress tracking, dashboards, analytics

  
area
study
 Spaced-repetition engine, decks, cards, reviews

  
area
tooling
 Local dev environment, mix tasks, editor tooling

  
area
ui
 LiveView practice experience and app shell

  
duplicate
Created by an over-eager re-run; safe to delete

  
future
Explicitly deferred beyond v1; tracked so it is not forgotten

  
kind
bug
 Something is broken

  
kind
chore
 Setup, tooling, mechanical work

  
kind
decision
 An architectural decision that needs recording

  
kind
docs
 Documentation

  
kind
feature
 New user-facing or domain capability

  
kind
spike
 Time-boxed investigation; output is a written finding, not code

  
kind
test
 Test coverage work

  
prio
blocker
 Blocks other work; do first

  
risk
high
 Correctness-sensitive or hard to reverse

No labels
area
auth
area
data
area
domain
area
infra
area
stats
area
study
area
tooling
area
ui
duplicate
future
kind
bug
kind
chore
kind
decision
kind
docs
kind
feature
kind
spike
kind
test
prio
blocker
risk
high
Milestone
Clear milestone
No items
Projects
Clear projects
No items
No project
First thousand words
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
nickkeers/first-thousand-words#14
No description provided.