LanguageDetector

Protocols:

• LanguageDetector.detect(text: str) -> tuple[LanguageScore, ...]
• LanguageDetectorProvider.language_detectors() -> tuple[LanguageDetector, ...]

What

LanguageDetector is the boundary for saying which languages a piece of text is written in. It is asked once per readable page, after every reader has produced its pages and before anything is embedded, and it is handed text rather than bytes — which is the whole reason it is not a SourceObserver.

The answer is a tuple of LanguageScore, ordered highest confidence first, each carrying a lowercase ISO 639-1 code and a number in [0, 1]. An empty tuple means no opinion: a text too short to judge, or an engine installed but unconfigured. Absent is never zero, the way a capability with no calibrated confidence reports none rather than 0.0.

One distribution ships through the same entry-point group a third party uses. indx-language-lingua wraps lingua, behind the lang extra. A stock pip install indx detects nothing, and its blocks carry no languages key at all — the same honest cliff as observing nothing and resolving no URI.

Why

Preflight never decodes content, and a language is a fact about characters. A language_hint sat on the router’s private PreflightContext from slice 2 to slice 13 and was never once assigned, because nothing on that side of the system could have filled it: an observer counts pages and reports a text-layer state without reading a word. Deleting it and asking after the read is what makes the field answerable at all.

A capability kind was the other candidate and is ruled out by construction: CapabilityKind is closed, the ladder over it is closed, and a detector routes nothing. It is the Chunker shape exactly — a port applied after reading, changing execution output rather than a routing decision.

Contract

The input is one text, and nothing else: a detector distribution depends on indx-interfaces alone and never sees the source bytes, the media type, or the plan. What it may not do is claim more than the characters support — the executor writes the answer onto a block, where it becomes something an index filters on.

A raise is a verdict about the detector rather than about the source, so the executor logs it and asks the next one, unlike an observer’s raise. The text being annotated has already been read and paid for, and losing an encode over an annotation would be the wrong trade.

LanguageDetectorProvider is optional the way ChunkerProvider is. A detector names no ID, joins no descriptor, and is not advertised on the capability snapshot at all: installing one moves nothing a plan is bound to.

Responsibilities and guarantees

• Answer with nothing rather than guessing. A short string scores confidently and wrongly in every statistical detector; a floor on input length is the detector’s own to set, and indx-language-lingua sets 20 characters.
• Report ISO 639-1, lowercase. The code is what an index stores; an engine’s own enum name is its own business.
• Order descending, so a caller reading [0] reads the answer.
• Keep module scope cheap. Discovery imports every provider module while building a snapshot, so the engine import belongs inside detect(), and building the model belongs behind the first call.

One attribute sits outside the protocol deliberately, read with a False default, because a member declared on it is a member isinstance and the type checker both demand. builtin = True is what indx-language-lingua sets, and the registry orders installed detectors ahead of it — first answer wins a page, so what indx ships has to be asked after what a deployment installed to override it.

Place in the system

DocumentExecutor.encode asks the first detector with an opinion for each page that produced text, writes that answer to the page block’s metadata under LANGUAGES_METADATA_KEY, and writes the document’s own answer to the document block. Pages nothing read contribute nothing rather than contributing a zero.

The document answer is the mean of its pages weighted by how much text each score was computed over. A flat mean lets a title page holding six words outvote a chapter, which is the same mistake weighting already fixed for generic-ocr’s self-reported confidence.

The key is reserved: EncodeRequest.metadata refuses languages outright rather than letting a caller’s label be silently replaced by a detector’s answer. Caller-supplied metadata and the detected half share the document block’s metadata dict for exactly that reason — one slot, one collision rule, stated at the trust boundary.

POLICY_VERSION did not move for any of this, and the capability snapshot ID is byte-identical before and after: a detector is an annotation on output, not a routing decision.

Related reading

• Protocol overview
• Executor
• Chunker
• SourceObserver
• Language models