Yaduha
An open-source framework for translating into languages that have no or very little training data. Yaduha implements LLM-Assisted Rule-Based Machine Translators (LLM-RBMT), a paradigm where large language models operate grammar-based tools rather than translating directly, guaranteeing grammatical correctness in the target language without requiring any parallel corpora.
Published at AmericasNLP @ NAACL 2024. Comparative study at LoResMT @ EACL 2026.
Background
There are roughly 7,000 languages spoken in the world today, and nearly half are endangered. Modern machine translation systems (Google Translate, DeepL) are trained on billions of translated sentence pairs scraped from the internet. This works well for languages like Spanish or Chinese that have massive online presences, but it fails completely for languages where that data does not exist.
Owens Valley Paiute (OVP) is a critically endangered language of the Numic branch of the Uto-Aztecan family, spoken in the eastern Sierra Nevada of California. There are no publicly available monolingual or bilingual corpora for OVP. You cannot train a data-hungry neural model on data that doesn't exist, so a fundamentally different approach is needed.
Before neural methods, machine translation was done with hand-written grammar rules and dictionaries (Rule-Based Machine Translatio; RBMT). RBMT produces grammatically correct, predictable output and requires no training data, but it is extremely labor-intensive to build, brittle, and limited to the sentence structures someone has manually programmed. LLM-RBMT combines the correctness guarantees of RBMT with the flexibility and linguistic understanding of large language models.
How LLM-RBMT Works
The central idea is that the LLM never translates directly into the target language. Instead, it operates a set of rule-based translation tools. Given the correct tools, LLMs can be good at translating simple sentences even for languages they have never seen much like a linguist who speaks many languages can follow a grammar guide for a new one.
A language package defines the target language’s grammar as a set of sentence types: Python classes (Pydantic models) whose fields represent linguistic features. Each sentence type has a __str__ method that renders the validated fields into a grammatically correct string in the target language.
For OVP, the sentence types capture a rich set of grammatical features: proximity (proximal -ii vs. distal -uu on subjects and objects), person (first, second, third), plurality (singular, dual, plural), inclusivity (OVP distinguishes taa “you and I,” taagwa “we inclusive,” and nüügwa “we exclusive”), reflexivity (reflexive possessives are marked differently than non-reflexive forms), and nominalization (verbs can become nouns — sawa-dü “the one who cooks” from sawa “to cook”).
Verbs take tense/aspect suffixes: -ku (completive/past), -ti (present ongoing), -dü (present), -wei (future/will), -gaa-wei (future/going to), and -pü (have x-ed). Transitive verbs require an object pronoun prefix on the verb — for example, u-puni-ku “saw him/her (distal).” Consonant mutations follow fortis/lenis rules (p↔b, t↔d, k↔g, s↔z, m↔~w) when suffixes attach.
Each of these features is constrained to valid values from the lexicon. Because the LLM’s output is forced to conform to the Pydantic model’s schema via structured output (constrained decoding), it can only produce valid combinations. The sentence type’s __str__ method then deterministically assembles the correct surface form.
The Translation Pipeline
Translating from English into the target language is a three-stage process. Consider the following English to OVP translation example:
Sentence Segmentation
The LLM decomposes the input into simple subject-verb (SV) and subject-verb-object (SVO) sentences that match the defined sentence types. Adjectives, adverbs, prepositions, and indirect objects are stripped — a deliberate trade-off that sacrifices some expressiveness for correctness. Subjects and objects can be nominalized verbs (e.g., “the runner” becomes “the one who runs”). It is better to produce something correct and simple than something complex and wrong.
“The men are walking a dog.” (past continuous)
“The men are drinking coffee.” (past continuous)
Structured Translation
Each simple sentence is translated by having the LLM fill in a Pydantic model — selecting a subject from the lexicon, choosing the correct verb, setting tense/aspect, and for SVO sentences, selecting an object and its morphological markers. The LLM’s output is constrained to the model’s schema via structured output (constrained decoding), so it can only produce valid combinations. The sentence type’s __str__ method then renders the validated fields into the target language, applying morphological rules like consonant mutation, suffix attachment, and pronoun prefixing. When the system encounters vocabulary it doesn’t know, it either substitutes a semantically close word from the lexicon (“prepare a meal” becomes “cook food”) or marks unknown words with brackets to be transparent about what it cannot translate. Both strategies preserve grammatical correctness.
lemma
tense_aspect
head
proximity
plurality
__str__applies morphological rules →Back-Translation Verification
Since no gold-standard translations exist, the system evaluates itself through back-translation: each OVP sentence is translated back into English by encoding its structured fields into a parts-of-speech list and having the LLM generate natural English from it — the LLM again never sees raw OVP text. The semantic similarity between the original English input and the back-translated English is measured using sentence embeddings. If the meaning survives the round trip, the OVP translation likely captured the intended meaning.
Comparing Five Translation Approaches
In a comparative study, we evaluated five fundamentally different approaches to translating into OVP. Each represents a different point on the spectrum between structured control and free-form generation.
Pipeline (LLM-RBMT with Structured Output)
The approach described above. The LLM operates a multi-step pipeline: segment the input, fill a Pydantic model whose schema encodes the grammar, and deterministically render the output. Structured output (constrained decoding) ensures the LLM can only produce valid field combinations. The output is guaranteed grammatically correct by construction.
Cost: ~$0.015/sentence (gpt-4o). Time: ~6–9 seconds. Highest and most consistent quality.
Builder (LLM-RBMT with Word-by-Word Construction)
A variant of LLM-RBMT where the LLM builds sentences one word at a time. At each step, the model is shown the available vocabulary options in OVP with their English translations, and it picks the best match. Because the LLM sees the actual available words at each decision point, it is better at finding close substitutes when the exact word is missing — for example, choosing tuunapi (food) instead of a bracketed [meal] when “meal” is not in the lexicon. Output is also guaranteed grammatically correct, but the approach requires one LLM call per word selected, making it significantly more expensive and slower.
Cost: ~$0.58/sentence (gpt-4o), ~$0.02 (gpt-4o-mini). Time: ~11–19 seconds. Quality comparable to Pipeline; fewer bracketed unknowns.
Instructions (Prompt-Only Baseline)
No tools, no structured output. The LLM is given a complete “grammar book” in its system prompt — all vocabulary lists (33 nouns, 14 transitive verbs, 22 intransitive verbs, 12 pronouns), all suffix rules, word ordering rules, and fortis/lenis consonant mutation tables — and asked to translate directly. This is the simplest approach to implement and the fastest to run, but output is not guaranteed to be grammatically correct. For example, the model might produce waakü-dü uhu-uu for “He works,” which violates the rule that subject pronouns like uhu cannot take the subject determiner suffix -uu. With a stronger model (gpt-4o), performance is surprisingly close to the structured approaches; with a weaker model (gpt-4o-mini), errors increase sharply.
Cost: ~$0.003/sentence (gpt-4o). Time: ~1 second. Fastest and cheapest; quality varies with model capability.
Fine-tuned (Trained on Generated Data)
Standard fine-tuning on gpt-4o and gpt-4o-mini, trained on 393 parallel sentences generated by the Pipeline translator. The training set was designed to cover every verb, noun, and pronoun at least three times, plus 125 sentences with deliberately missing vocabulary to teach the model how to handle gaps. Despite careful dataset design, this approach performed worst of all five. 393 sentences is simply not enough data to teach a model to generalize OVP grammar. This result underscores the fundamental problem: for extremely low-resource languages, there is not enough data to train data-dependent methods, even when the data is synthetically generated.
Cost: ~$0.001/sentence. Time: ~1–2 seconds. Lowest quality; limited by training corpus size.
RAG (Retrieval-Augmented Generation)
The LLM is given function-calling access to an external OVP dictionary — it can search English definitions and retrieve example sentence pairs. The model looks up relevant words and examples, then generates a translation using the retrieved context. Despite having access to the largest vocabulary of any approach, RAG underperformed. The core problem is that many outputs are semantically aligned but grammatically incorrect. For example, translating “Romeo wrote a letter” produces Romeo i-dü-mui-pü piponibü, which attempts reasonable word choices but assembles them into an ungrammatical construction. Without structural constraints, the model cannot reliably compose words into valid OVP sentences, and ungrammatical output receives a score of 0.0 in evaluation because it cannot be reliably back-translated.
Cost: ~$0.11/sentence (gpt-4o). Time: ~7–11 seconds. Most vocabulary access; grammatical correctness not guaranteed.
Evaluation
Standard machine translation metrics like BLEU measure word overlap against a reference translation. But reference translations do not exist for OVP — there is no gold standard to compare against. We therefore developed an evaluation framework that measures how much meaning is preserved through the translation pipeline, using five complementary metrics across two categories.
Metrics
Lexical metrics measure surface-level overlap between the back-translated English and the original input:
- BLEU — n-gram precision, the traditional MT metric. Penalizes semantically faithful translations that use different words.
- chrF++ — character n-gram F-score with word unigrams and bigrams. Less sensitive to exact word choice than BLEU.
Semantic metrics measure meaning preservation regardless of exact wording:
- BERTScore — compares contextualized token embeddings between the original and back-translated sentences.
- COMET — a learned metric trained to predict human judgments of translation quality.
- Semantic Similarity — cosine similarity between sentence embeddings (all-MiniLM-L6-v2). This is the primary metric used for evaluation.
Backwards vs. Comparator Scores
Each translation is evaluated with two scores. The backwards score measures the similarity between the original English and the back-translated English — how much meaning survives the round trip through OVP. The comparator score does the same thing but replaces all out-of-vocabulary words with placeholders first, isolating the system’s performance on vocabulary it actually knows. This prevents similarity scores from being inflated by untranslated English words that pass through unchanged.
A high backwards score with a low comparator score means the translation is structurally correct but missing vocabulary (“The king wore the crown” → backwards 0.995, comparator 0.541 — correct structure, but “king,” “crown,” and “wear” are not in the lexicon). A low backwards score with a low comparator score means significant meaning was lost.
Baseline Contextualization
Raw similarity numbers are hard to interpret without context. To establish a meaningful baseline, we computed semantic similarity across 11,175 pairs of unrelated sentences, yielding a mean of 0.569 (σ = 0.059). This represents the “null hypothesis” — the score you would expect between two sentences that have nothing to do with each other. Scores above 0.746 (μ + 3σ) indicate the translation very likely preserves core meaning. We applied this same baseline contextualization to all five metrics.
Results
We evaluated all five approaches on 150 English sentences evenly distributed across six sentence types: subject-verb, subject-verb-object, two-verb, two-clause, complex, and nominalization. Each approach was tested with both gpt-4o and gpt-4o-mini. Outputs that violated OVP grammar were automatically assigned a score of 0.0 — grammatical correctness is non-negotiable for endangered language work, where community trust in translation tools is essential.
Cost and Speed
| Approach | Model | $/sentence | Seconds |
|---|---|---|---|
| Pipeline | gpt-4o | $0.015 | 9.4 |
| gpt-4o-mini | $0.001 | 5.9 | |
| Builder | gpt-4o | $0.581 | 18.9 |
| gpt-4o-mini | $0.024 | 11.0 | |
| Instructions | gpt-4o | $0.003 | 1.3 |
| gpt-4o-mini | $0.000 | 0.8 | |
| Fine-tuned | gpt-4o | $0.001 | 1.3 |
| gpt-4o-mini | $0.000 | 2.0 | |
| RAG | gpt-4o | $0.107 | 7.3 |
| gpt-4o-mini | $0.006 | 10.7 |
The Pipeline translator achieved the highest and most consistent backwards similarity scores (~0.85) across sentence types, at a cost of about $0.015 per sentence with gpt-4o or $0.001 with gpt-4o-mini. The Builder achieved comparable quality, particularly on the comparator metric (where it benefits from choosing available vocabulary rather than inserting placeholders), but at roughly 40 times the cost due to its per-word LLM calls. Instructions with gpt-4o performed surprisingly well but with higher variance; with gpt-4o-mini, quality dropped noticeably and grammatical violations appeared. Fine-tuning performed worst overall — 393 training sentences were simply not enough to teach the model OVP grammar. RAG underperformed despite having the most vocabulary access, because dictionary lookups alone cannot enforce grammatical rules.
In a separate evaluation, the Pipeline translator achieved 98% accuracy on OVP-to-English translation across 100 randomly generated test sentences, confirming that the structured representation captures meaning reliably in both directions.
Translation Examples
The following examples illustrate the range of outcomes and how the backwards/comparator scoring reveals different failure modes.
All vocabulary available. Meaning fully preserved.
The sentence structure is correct — subject determiner suffix, object proximity suffix, object pronoun prefix, and verb tense suffix are all properly applied. But “king,” “crown,” and “wear” are not in the lexicon, so the backwards score is high (the English words pass through) while the comparator score reveals the vocabulary gap.
The conjunction “and” and the specific activity “hiking” were lost during sentence segmentation. The meaning is partially preserved but degraded — an inherent limitation of reducing complex sentences to simple SV/SVO forms.
The pronoun uhu cannot take the subject determiner suffix -uu — that suffix applies only to nouns. This is ungrammatical and receives an automatic 0.0. The Pipeline and Builder approaches cannot produce this error because their schemas prevent it.
Building a Language Package
A language package is a Python module that defines the target language’s vocabulary and grammar. The core work is defining sentence types — Pydantic models whose fields are constrained to valid lexical entries and whose __str__ method applies the language’s morphological rules to render a grammatically correct sentence. Each sentence type also provides example pairs (English sentence, filled model) that serve as few-shot demonstrations for the LLM.
No expertise in machine learning is required. The package author needs knowledge of the target language’s grammar and vocabulary, and basic Python. The framework handles the LLM interaction, structured output enforcement, back-translation, and evaluation.
Packages are installed as Python plugins via pip and registered through standard entry points. Yaduha currently supports four LLM backends (OpenAI, Anthropic, Google Gemini, Ollama) and ships with a REST API and web dashboard for testing translations.
Yaduha Studio
A browser-based development environment for building and testing language packages. Edit Python code with Monaco Editor, validate sentence types and vocabulary against the framework’s schema using Pyodide (Python running in-browser via WebAssembly), test translations in real time, and sync changes with GitHub.
pip install yaduha