Language detector

A language detector says which languages a piece of text is written in. It is asked once per readable page, after every reader has produced its pages and before anything is embedded, and it is handed text rather than bytes — which is why this is not a source observer: preflight never decodes content, and a language is a fact about characters. indx-language-lingua ships this way, through the same entry-point group your distribution uses. Package and install it as the overview describes.

from typing import Any

from indx_interfaces import (
    CapabilityDescriptor,
    CapabilityId,
    LanguageDetector,
    LanguageScore,
)


class DeclaredLanguageDetector:
    def detect(self, text: str) -> tuple[LanguageScore, ...]:
        # Nothing is how a detector says "no opinion": a text too short to
        # judge, or an engine installed but unconfigured. Absent is never
        # zero — the next detector is asked, and the block simply carries
        # no language.
        if len(text.strip()) < 20:
            return ()
        return (LanguageScore(language="ja", confidence=0.9),)


class Provider:
    def descriptors(self) -> tuple[CapabilityDescriptor, ...]:
        # This distribution annotates what other capabilities read; it reads nothing.
        return ()

    def create(self, capability_id: CapabilityId) -> Any:
        raise ValueError(f"this provider declares no capabilities, so not {capability_id}")

    def language_detectors(self) -> tuple[LanguageDetector, ...]:
        return (DeclaredLanguageDetector(),)

LanguageDetectorProvider is optional, like ChunkerProvider. A detector names no ID, joins no descriptor, and is not advertised on the capability snapshot at all—what it changes is an annotation on execution output, which no outstanding plan recorded. The first detector with an opinion wins a page; installed detectors are ordered ahead of the ones indx ships.

Answers are ordered highest confidence first and use lowercase ISO 639-1, because the code is what an index stores and your engine’s enum name is your engine’s business. Raising means “mine, and broken”—a verdict about the detector, not the source—so the executor logs it and asks the next one instead of failing an encode the run already paid to read.

The executor writes your page answers onto each page block’s metadata under languages, and writes the document’s own answer—the mean of its pages, weighted by how much text each score was computed over—onto the document block. That key is reserved: a caller supplying languages on EncodeRequest is refused rather than overwritten.

Checklist

• Answer with nothing rather than guessing. A short string scores confidently and wrongly in every statistical detector.
• Report ISO 639-1, lowercase, ordered descending.
• Keep module scope cheap; the engine import belongs inside detect(), and building the model belongs behind the first call.

Reference

• LanguageDetector protocol
• The lingua detector indx ships