PageEntityExtractor and ChunkEntityExtractor
The ports that find named entities in one page or one chunk, and say where they are.
Protocols:
PageEntityExtractor.id: ExtractorIdPageEntityExtractor.extract(text: str) -> Mapping[str, tuple[EntitySpan, ...]]PageEntityExtractorProvider.page_entity_extractors() -> tuple[PageEntityExtractor, ...]ChunkEntityExtractor.id: ExtractorIdChunkEntityExtractor.extract(text: str) -> Mapping[str, tuple[EntitySpan, ...]]ChunkEntityExtractorProvider.chunk_entity_extractors() -> tuple[ChunkEntityExtractor, ...]
An entity extractor finds the names in a text and says where they are. The
answer maps a label — person, organization, location, date, money,
invoice_number — to EntitySpan entries, each carrying the surface text, a
[start, end) character range, and a confidence in [0, 1].
The label is a free string. Nothing in indx-interfaces enumerates entity
labels, for the reason nothing enumerates facets: the useful set is a domain’s
rather than a contract’s, and 登録番号 matters to a deployment reading Japanese
invoices in a way PERSON does not.
There is deliberately no DocumentEntityExtractor. Pages already tile a
document with no gaps, so a whole-document port would window its input
internally to produce what the page port produces directly, and its spans would
index a concatenation rather than a block.
This is the second axis of the classifier split. A classifier answers what this
text is; an extractor answers what is named in it, and where. Both are
annotations on output, neither routes anything, and both are opt-in by ID — but
they cannot share a return type. LabelScore is a label and a number, with
nowhere to put a location, and an entity without its location is a fact you
cannot show a user or check against the page.
Squeezing entities into the classifier port was the obvious cheap move and it is the one thing this design refuses: it would have advertised entity extraction while discarding the spans that are the point (ADR-0029).
Contract
Section titled “Contract”A span indexes the text of the block that names it. Not the text the
extractor was handed if that text was assembled, not a document-wide
concatenation, not bytes — characters, into one block’s own text field, which
the response already carries. This is the whole addressing rule, and it is why
an extractor is asked with one unit at a time
(ADR-0030).
Character offsets rather than byte offsets is load-bearing for Japanese, where
the two differ by a factor of three.
The input is the full text of one page or one chunk. An extractor with a token window sees a long page whole and slices it itself, returning spans in the input text’s coordinates rather than its window’s.
A ChunkEntityExtractor requires chunk granularity, refused with a 422 before
the fetch when the request did not ask for CHUNK. Unknown IDs and external
implementations under data_residency refuse the same way, with the same
reasoning: a silently skipped extractor is an answer the caller believes was
given and was not. IDs share one namespace with the classifier ports.
A raise is logged and the unit skipped. device, cost_usd and builtin sit
outside the protocol and are read with defaults.
Responsibilities and guarantees
Section titled “Responsibilities and guarantees”- Return spans that index the text you were given, from its first character. An offset that is right about a window and wrong about the input is worse than no span, because nothing downstream can detect it.
- Do not return overlapping spans for one label. If two readings compete, return the one you can defend and let a second extractor disagree.
- Report a confidence you can defend. A gazetteer hit and a model’s softmax are both self-reports, and neither is calibrated against a labelled corpus — the benchmark has no ground truth to calibrate against (ADR-0035).
- Leave a label out rather than emitting an empty tuple for it.
- Keep module scope cheap, and build the engine behind the first call. A dictionary or a model is tens of megabytes and discovery must not pay for it.
- Advertise nothing when unable to run: no extra, no dictionary, no model, no extractor.
Place in the system
Section titled “Place in the system”DocumentExecutor.encode resolves the enabled IDs and checks residency before
the fetch, reads the document, chunks it when chunk granularity was asked for,
and asks each enabled extractor for each unit in request order.
Every answer is written to the document block’s metadata under
ENTITIES_METADATA_KEY, keyed by the block ID the entity was found in — page
blocks and chunk blocks alike. Chunk blocks themselves stay inert, carrying
text, bbox, embeddings and provenance and no metadata, which is the invariant
the executor’s own tests pin
(ADR-0031).
Beside the per-block spans, the document block carries the aggregation: a count
per (label, NFKC-normalized surface form), with the raw surface forms kept
beside the count. A count is not an identity. Coreference is out of scope,
so 「山田太郎」 and 「山田」 are two entries and nothing here claims they are one
person (ADR-0032).
entities is the third reserved metadata key, beside languages and
classification; EncodeRequest.metadata refuses it outright.
POLICY_VERSION does not move, and neither does the capability snapshot ID: the
field advertising extractor IDs is excluded from the content hash, so installing
an extractor invalidates no outstanding plan.