Spike: pick the translation source for cross-language links #16

New issue

Open

opened 2026-09-26 13:58:32 +00:00 by belvedere · 0 comments

belvedere commented

2026-09-26 13:58:32 +00:00

Collaborator

Objective: A written recommendation, with measured numbers, for where translations come
from. Time-box: one day. Output is a document, not code.

Candidates to compare (all offline-usable, no runtime API dependency):

WikDict (https://download.wikdict.com/dictionaries/sqlite/<release>/) — pre-built
per-pair dictionaries derived from Wiktionary, released as <src>-<tgt>.sqlite3 for many
pairs; licence CC-BY-SA. Verify the current release directory rather than trusting this note.
Kaikki.org / Wiktextract per-language JSONL
(https://kaikki.org/dictionary/<Language>/kaikki.org-dictionary-<Language>.jsonl) —
rich: senses, forms, translations, examples. Large (the English extract was ~3.3 GB on
2026-09-26), so treat it as an offline batch job.

Measure for each: coverage of the top 1000 words of the target language; how well senses
disambiguate; licence and attribution obligations; total download size; how long a full import
takes; whether we can avoid holding a multi-GB file on the app server.

Steps:

Download a bounded sample of each.
Compute coverage against the frequency list for two languages.
Write docs/decisions/0002-translation-source.md with a recommendation and the numbers.

Verify: the doc contains a coverage table and an explicit recommendation with rationale.

Pitfall: do this before building the importer — the schema details (sense_rank,
confidence) depend on which source wins.

**Objective:** A written recommendation, with measured numbers, for where translations come from. Time-box: one day. Output is a document, not code. **Candidates to compare** (all offline-usable, no runtime API dependency): - **WikDict** (`https://download.wikdict.com/dictionaries/sqlite/<release>/`) — pre-built per-pair dictionaries derived from Wiktionary, released as `<src>-<tgt>.sqlite3` for many pairs; licence CC-BY-SA. Verify the current release directory rather than trusting this note. - **Kaikki.org / Wiktextract** per-language JSONL (`https://kaikki.org/dictionary/<Language>/kaikki.org-dictionary-<Language>.jsonl`) — rich: senses, forms, translations, examples. Large (the English extract was ~3.3 GB on 2026-09-26), so treat it as an offline batch job. **Measure for each:** coverage of the top 1000 words of the target language; how well senses disambiguate; licence and attribution obligations; total download size; how long a full import takes; whether we can avoid holding a multi-GB file on the app server. **Steps:** 1. Download a bounded sample of each. 2. Compute coverage against the frequency list for two languages. 3. Write `docs/decisions/0002-translation-source.md` with a recommendation and the numbers. **Verify:** the doc contains a coverage table and an explicit recommendation with rationale. **Pitfall:** do this before building the importer — the schema details (`sense_rank`, `confidence`) depend on which source wins.

belvedere added this to the M2 — Lexicon model & data pipeline milestone

2026-09-26 13:58:32 +00:00

belvedere added the

labels

2026-09-26 13:58:32 +00:00

belvedere referenced this issue

2026-09-26 14:02:01 +00:00

Spike: pick the translation source for cross-language links #68