mix f1000.import.concepts — build concepts and link words #18

New issue

Open

opened 2026-09-26 13:58:33 +00:00 by belvedere · 0 comments

belvedere commented

2026-09-26 13:58:33 +00:00

Collaborator

Objective: Create Concept rows and WordConcept links from the translation source chosen
by the spike, so a source-language word shares a concept with its target-language counterpart.

Files:

Create: lib/mix/tasks/f1000.import.concepts.ex
Create: lib/first_thousand_words/import/translation_source.ex (behaviour)
Create: lib/first_thousand_words/import/wikdict_source.ex (implementation)

Steps:

Define a TranslationSource behaviour (candidates(word, from, to) -> [{word, confidence}])
so the concrete source can be swapped without touching the importer.
For each language in the enabled set, walk its 1000 ranked words in rank order and attach
each to a concept; create the concept on first sight using the English gloss as the anchor.
Record sense_rank and confidence on each WordConcept.
Make it resumable: skip words already linked, so a crash at word 600 does not restart.
Refuse to run if the frequency import has not run for a language.

Verify:

mix f1000.import.concepts --source es
# then in IEx: a concept for a known word resolves to both an EN and an ES word

Pitfall: do not create a concept per translation pair — that produces thousands of
one-to-one concepts and defeats the pivot. A concept is a meaning, and both words point at it.

**Objective:** Create `Concept` rows and `WordConcept` links from the translation source chosen by the spike, so a source-language word shares a concept with its target-language counterpart. **Files:** - Create: `lib/mix/tasks/f1000.import.concepts.ex` - Create: `lib/first_thousand_words/import/translation_source.ex` (behaviour) - Create: `lib/first_thousand_words/import/wikdict_source.ex` (implementation) **Steps:** 1. Define a `TranslationSource` behaviour (`candidates(word, from, to) -> [{word, confidence}]`) so the concrete source can be swapped without touching the importer. 2. For each language in the enabled set, walk its 1000 ranked words in rank order and attach each to a concept; create the concept on first sight using the English gloss as the anchor. 3. Record `sense_rank` and `confidence` on each `WordConcept`. 4. Make it **resumable**: skip words already linked, so a crash at word 600 does not restart. 5. Refuse to run if the frequency import has not run for a language. **Verify:** ```bash mix f1000.import.concepts --source es # then in IEx: a concept for a known word resolves to both an EN and an ES word ``` **Pitfall:** do not create a concept per translation pair — that produces thousands of one-to-one concepts and defeats the pivot. A concept is a *meaning*, and both words point at it.

belvedere added this to the M2 — Lexicon model & data pipeline milestone

2026-09-26 13:58:33 +00:00

belvedere added the

labels

2026-09-26 13:58:33 +00:00

belvedere referenced this issue