Language engineering · Research software · Java & Python
Radixor
A deterministic multilingual stemmer that turns morphological lexicons into compact executable transformations. Language knowledge is prepared offline; the live search path executes an immutable compiled model rather than consulting a large text dictionary entry by entry.
Morphology compiled into a search primitive
The original Egothor idea was to learn transformations from lexical evidence instead of hand-writing a separate stemming program for every language. Radixor completes that model with explicit reduction semantics, immutable runtime structures, versioned binary persistence, deterministic ambiguity handling and interoperable Java/Python runtimes.
A patch is not a stored final answer. It is an instruction that transforms an input form into a canonical representation. This allows evidence learned from many lexical families to share structure and lets the compiler generalize compatible transformations beyond a flat word-to-output table.
One engine, many languages
The runtime is language-independent. Linguistic knowledge enters through versioned model data, so adding or improving a language is primarily a model-compilation problem rather than a new source-code algorithm.
Ambiguity is retained
The same model can expose one preferred result or all locally supported candidates in deterministic ranked order. A downstream search system decides how much ambiguity it wants to activate.
Build-time complexity, simple runtime
Induction, reduction and validation happen offline. Production lookup traverses the frozen model and applies a compiled patch command to the token.
Model artifacts are first-class
Language models are independently versioned, integrity-checkable deployment artifacts rather than hidden resources baked into a single executable.
Java and native Python
The project provides Java integration and a native Python/Rust runtime that consume the same versioned compiled-model concept.
Controlled extension
A compiled Java model can be reopened through a builder, extended with domain-specific evidence, rebuilt, tested and deployed as a new artifact instead of being mutated live.
- Linguistic evidence is curated offline instead of improvised at query time.
- Multilingual models remain explicit versioned artifacts.
- Deterministic runtime behaviour keeps stemming usable inside controlled systems.
- Research outputs are translated into deployable engineering assets.
What was actually measured
The 2026 twenty-language study evaluates morphological conflation as a relation. In plain language, it asks two questions at very large scale: do forms that belong together end up together, and do unrelated forms stay apart? It does not require every system to spell its output like a dictionary lemma.
From millions of words to trillions of pair decisions
The evaluation uses independently curated UniMorph-derived paradigms and counts the relation induced by deterministic stemming. This makes false joins and missed joins measurable without confusing stem spelling with stem quality.
- Positive pairs tell us whether genuine variants stay together.
- Negative pairs tell us whether unrelated forms are wrongly merged.
- Lemma spelling is not treated as the whole story.
- Ambiguity can be studied separately for primary and all-candidate output.
The quality result in practical terms
The candidate-aware result is especially useful for understanding the design. A single context-free stem must choose one answer even when morphology is genuinely ambiguous. Radixor keeps alternative evidence available first; an indexer, query expander or contextual layer can then choose a policy appropriate to the application.
Compact models instead of live dictionary lookup
The source lexicons are valuable because they contain scholarly morphological evidence; they do not need to remain the runtime representation. Radixor distils that evidence into patch programs and a reduced trie.
99.92 MB of lexical evidence becomes 16.55 MB of compiled model data
Measured on decompressed payloads, excluding outer GZip wrappers and package metadata. Across all twenty models, the compiled v7 streams use 2.50 bytes per evaluated form versus 15.12 bytes in the source payload.
A visible knowledge–quality–cost envelope
One additional English experiment deliberately compiles models from progressively smaller deterministic slices of the available lexical rows, then evaluates against the complete resource. The point is not to find one “magic” dictionary size; it is to show what is traded away when knowledge is removed.
Three anchor points
For non-specialist readers this is often easier to interpret than a large table: less lexical evidence buys only limited runtime benefit but steadily erodes stemming quality, especially on changed forms.
| Dictionary rows used | Exact agreement · all forms | Exact agreement · changed forms | Changed-form runtime |
|---|---|---|---|
| 100% | 97.478% | 97.197% | 71.6 ns/token |
| 50% | 95.262% | 86.107% | 61.1 ns/token |
| 10% | 92.868% | 76.516% | 47.0 ns/token |
Absolute latency is machine- and workload-specific. The curve is most useful as an engineering illustration: removing lexical evidence mainly hurts forms that actually require a transformation, while runtime changes more modestly and is not strictly monotonic.
Twenty default languages
A quick visual pass through the standard set makes the language spread easier to absorb than a plain comma-separated list.
Flags are used where they help orientation. Where a sovereign-state flag would be misleading, a language badge is used instead.
Research lineage and reproducibility
Radixor is the contemporary completion of a research line that began in Egothor in 2001: infer compact word-to-stem edit programs from lexical examples, organize them in a trie and execute the resulting model in a search system. The modern work adds explicit semantic reduction, uniform-subtree contraction, immutable compiled representations, evidence-ranked alternatives, model provenance, Java/native interoperability and publication-grade evaluation artifacts.
The evaluation is designed to be auditable. Model identifiers and descriptor hashes are recorded with result rows; the quality snapshot contains the raw confusion counts and derived metrics; generated tables and figures can be rebuilt from the machine-readable data rather than being manually transcribed.