Chunker

A chunker decides what a chunk is: the embeddable unit a retriever indexes. It is asked once per encode, after every reader has produced its pages and before anything is embedded. It receives the same bytes every reader saw. A chunker that knows the format can reopen them for the structure a flat string lost. One that only needs text reads it off the page outputs. indx-chunker-pdf (line boundaries from PDFium text runs) and indx-chunker-page (the one-chunk-per-page floor) ship this way, through the same entry-point group your distribution uses. Package and install it as the overview describes.

from typing import Any

from indx_interfaces import (
    CapabilityDescriptor,
    CapabilityId,
    Chunker,
    ChunkPiece,
    PageChunks,
    PageOutput,
)


class ParagraphChunker:
    def chunk(
        self, content: bytes, media_type: str, outputs: tuple[PageOutput, ...]
    ) -> tuple[PageChunks, ...]:
        # Nothing is how a chunker says "not mine": answers are merged per
        # page, first in chunker order wins, and an unclaimed page goes to
        # the next chunker in line.
        if media_type != "text/plain":
            return ()
        return tuple(
            PageChunks(
                page=output.page,
                pieces=tuple(
                    ChunkPiece(text=paragraph)
                    for paragraph in output.text.split("\n\n")
                    if paragraph.strip()
                ),
            )
            for output in outputs
            if output.text
        )


class Provider:
    def descriptors(self) -> tuple[CapabilityDescriptor, ...]:
        # This distribution draws boundaries, it reads nothing.
        return ()

    def create(self, capability_id: CapabilityId) -> Any:
        raise ValueError(f"this provider declares no capabilities, so not {capability_id}")

    def chunkers(self) -> tuple[Chunker, ...]:
        return (ParagraphChunker(),)

ChunkerProvider is optional, like SourceObserverProvider: a provider that reads pages and never draws a boundary never implements it. A chunker names no ID, joins no descriptor, and is not advertised on the capability snapshot at all—what it changes is how execution output is cut, which no outstanding plan recorded. Installed chunkers are ordered ahead of the ones indx ships; indx-chunker-page alone sets fallback = True and sorts after everything, so an answer for every page is the last answer asked for.

Each piece is non-empty text with an optional bbox normalized the way Block.bbox is; the executor mints every page:N/chunk:M ID from piece order. Raising means “mine, and broken”—a verdict about the chunker, not the source—so the executor logs it and asks the next chunker instead of failing an encode the run already paid to read. Never raise for bytes you cannot open: the readers already gave those their verdict.

Checklist

• Answer with nothing for a document you have no boundary for; never invent pieces for a page you did not understand.
• Order pieces in reading order—the executor’s chunk indices and IDs are derived from it.
• Keep module scope cheap; a heavy import belongs inside chunk().

Reference

• Chunker protocol
• PDF line chunker
• Page-floor chunker