Entity extractor

An entity extractor finds the names in a text and says where they are: a label — person, organization, location, date, money, invoice_number — mapped to spans, each carrying the surface text, a character range, and a confidence. It is the other half of the classifier grid: a classifier says what a text is, an extractor says what is named in it. There are two, one per unit: PageEntityExtractor and ChunkEntityExtractor. There is deliberately no document one — pages already tile a document with no gaps. Package and install it as the overview describes.

import re
from collections.abc import Mapping
from typing import Any

from indx_interfaces import (
    CapabilityDescriptor,
    CapabilityId,
    Device,
    EntitySpan,
    ExtractorId,
    PageEntityExtractor,
)

INVOICE_NUMBER = re.compile(r"\b(?:INV|請求)[-‐]?\d{4,}\b")


class InvoiceNumberExtractor:
    # What a request names in `extraction.page_ids`. IDs are one namespace
    # across all five classifier and extractor ports.
    id = ExtractorId("acme-invoice-number")
    device = Device.CPU

    def extract(self, text: str) -> Mapping[str, tuple[EntitySpan, ...]]:
        # Offsets are characters into the text you were handed, counted from
        # its first character -- which is this block's own `text`.
        found = tuple(
            EntitySpan(
                text=match.group(),
                start=match.start(),
                end=match.end(),
                confidence=0.95,
            )
            for match in INVOICE_NUMBER.finditer(text)
        )
        # A label with nothing found is left out, not returned empty.
        return {"invoice_number": found} if found else {}


class Provider:
    def descriptors(self) -> tuple[CapabilityDescriptor, ...]:
        # This distribution annotates what other capabilities read; it reads nothing.
        return ()

    def create(self, capability_id: CapabilityId) -> Any:
        raise ValueError(f"this provider declares no capabilities, so not {capability_id}")

    def page_entity_extractors(self) -> tuple[PageEntityExtractor, ...]:
        return (InvoiceNumberExtractor(),)

A span indexes the text of the block that names it. Not the window your model sliced, not a document-wide concatenation, not bytes — characters, into one block’s own text field, which the response already carries, so a caller resolves your span against data it already has. Characters rather than bytes is load-bearing: in UTF-8 Japanese the two differ by a factor of three. If your engine has a token window, slice the input yourself and return spans in the input’s coordinates; an offset that is right about your window and wrong about the input cannot be detected by anything downstream.

A caller enables yours with {"extraction": {"page_ids": ["acme-invoice-number"]}} on encode. Installed IDs are advertised on the capability snapshot outside its content hash, so installing yours moves no plan. Raising means “mine, and broken” — the executor logs it, skips that unit, and asks the next extractor rather than failing an encode the run already paid to read. device and cost_usd are read off your class with defaults, exactly as for a classifier: declare Device.EXTERNAL for a hosted model, and a request carrying data_residency is refused before the source is fetched rather than silently sending the text off-box.

For chunks instead

Declare through chunk_entity_extractors(), and a request names your ID in extraction.chunk_ids. Everything else is identical. Two consequences: chunk granularity is required — naming a chunk extractor in a request that did not ask for CHUNK is a 422 before the fetch — and you will be called once per chunk rather than once per page, so a model that costs a call per invocation costs a great many more.

What the executor does with your answer

Every span is written to the document block’s metadata under entities, keyed by the block ID it was found in — page blocks and chunk blocks alike. Chunk blocks themselves stay inert. Beside the per-block spans the executor writes the aggregation: a count per (label, NFKC-normalized surface form), with the raw surface forms kept.

A count is not an identity. Coreference is out of scope, so 「山田太郎」 and 「山田」 are two entries and nothing claims they are one person. Normalization merges 「ＡＢＣ株式会社」 with “ABC株式会社” and stops there.

entities is reserved: a caller supplying it on EncodeRequest is refused rather than overwritten.

Checklist

• Return spans that index the text you were given, from its first character.
• Leave a label out rather than returning an empty tuple for it, and do not return overlapping spans for one label.
• Report a confidence you can defend, and do not claim it is calibrated — there is no ground truth in the benchmark to calibrate against.
• Advertise nothing when you cannot run — without your extra, your dictionary or your model — rather than an extractor that raises.
• Keep module scope cheap; the engine import belongs inside extract(), and a dictionary or model behind the first call.

Reference

• PageEntityExtractor and ChunkEntityExtractor protocols
• Extend: classifiers
• Encoding models