Chunker

The port that decides what a chunk is.

Protocols:

Chunker.chunk(content: bytes, media_type: str, outputs: tuple[PageOutput, ...]) -> tuple[PageChunks, ...]
ChunkerProvider.chunkers() -> tuple[Chunker, ...]

What

Chunker is the boundary for deciding what a chunk is — the embeddable unit a retriever indexes. It is asked once per encode, after every reader has produced its pages and before anything is embedded, with the same bytes every reader saw: a chunker that knows the format can reopen them for the structure a flat string lost, and one that only needs text reads it off the page outputs.

A chunker claims pages by answering with PageChunks for them, and disclaims a document by answering with nothing. Answers are merged per page, first in chunker order wins, so a chunker is free to be honest about partial knowledge — the PDF chunker has no layout for a scanned page OCR read, and saying nothing about it hands that page to the next chunker in line, indx-chunker-lines, which cuts the lines the OCR reader located into chunks with rectangles.

Six chunker distributions ship with indx through the same entry-point group a third party uses. indx-chunker-pdf splits PDF pages on the text-run layout PDFium reports, merged into visual lines, each chunk carrying a normalized bbox. indx-chunker-pptx cuts a PowerPoint slide into its shapes: one chunk per text-bearing shape or table and one image chunk per raster picture, each at the box the file states, with group transforms applied and placeholder frames inherited from the layout and master; its version 2, opt_in, cuts the same chunks and gives a table chunk its layout.table, a box per cell. indx-chunker-page is the floor: it claims every readable page of any media type with its whole text as one chunk, which is exactly what build hardcoded before this port existed. An install without it — a stock pip install indx — chunks nothing, the same honest cliff as observing nothing and resolving no URI. indx-chunker-lines cuts the lines a reader located (PageOutput.lines, which generic-ocr fills from the recognizer’s boxes and which never reaches the wire) into one chunk per line at its rectangle, for any media type and without reopening the bytes; a page whose reader stated no lines is disclaimed. indx-chunker-layout is version 2 of pdf: the same PDFium lines, grouped into one chunk per paragraph at the box around its lines, and opt_in, so it cuts nothing until a request or the deployment names pdf@2 (ADR-0069). It renders nothing: a page no reader read is left to version 1, which a named version leads and never replaces. Its version 3, also opt_in, adds ruled tables: the page’s rules become a grid whose cells carry the text inside them and their own boxes. The same distribution ships version 2 of lines: the lines a reader located, grouped by the same geometry, so a scan gets paragraphs too (lines@2, also opt_in). indx-chunker-grid ships delimited@1, xlsx@1 and docx@1, all opt_in: a CSV or TSV page parsed from the text its reader decoded, a worksheet walked by indx_ooxml.sheet, or a Word section walked by indx_ooxml.wordml, cut into its tables, each a chunk whose layout.table carries the cells (merges as spans), with the other text – a title above a table, a Word paragraph – as paragraph chunks of their own.

Why

blocks.build minted one chunk per page and marked it as a deferral: split on real structure once something downstream has an opinion about size. The recipe store and the export index are that downstream, and chunk_texts is the only path to a vector, which made chunking the thing that silently decided retrieval quality — measured concretely by the fixture invoice, whose generator draws every glyph as its own text run: boundary knowledge existed in the bytes and had nowhere to be said.

A capability-supplied channel on PageOutput was the other candidate and was rejected for uniformity: a boundary is an opinion about retrieval, not about reading, and a port applied after all reading lets one implementation draw boundaries for pages that different capabilities read.

Contract

The inputs are the loaded source bytes, the detected media type — the same pair PageReader.read and SourceObserver.observe receive — plus every PageOutput the run produced. A chunker distribution therefore depends on indx-interfaces and nothing else.

The output is a tuple of PageChunks, each naming a 1-based page and carrying ordered ChunkPieces. A piece is non-empty text or non-empty image bytes — exactly one, see below — and an optional bbox, normalized the way Block.bbox is: one validate_bbox serves both, so a piece that validates becomes a block that validates. Like PageOutput, it is deliberately not a Block: the executor mints every page:N/chunk:M from piece positions, so ID uniqueness stays with the one thing that can see the whole document.

A piece without a bbox carries a bbox_reason instead, always: no_geometry when the chunker knows no positions, hidden when the source marks the shape hidden, unresolved when the format has positions and this shape’s could not be resolved. The model refuses a piece with neither and a piece with both, so the reason is the chunker’s to state and never a default the executor fills in.

A text piece may state a layout: its role, and for a table role the grid of cells (ADR-0070). The executor copies it to the chunk block untouched. The piece’s text is still the plain rendering, rows on lines and cells tab-separated, and a table split across pieces repeats its header rows in each.

An absent page means “not mine” and the next chunker is asked. Raising means “mine, and broken” — and unlike an observer’s raise, it is a verdict about the chunker rather than the source, so the executor logs it and moves to the next chunker instead of failing an encode the run already paid to read. Chunk text is the chunker’s to write and need not reconstruct the page text: document and page blocks keep the reader’s text either way.

ChunkerProvider is optional, the way SourceObserverProvider is: a provider that reads pages and never draws a boundary never implements it. A chunker states an id and an integer version (ADR-0067): the ID names what it cuts (pdf, pptx, lines, page) and the version which way of cutting it, so a better boundary for the same pages ships as a new version beside the old one. The pair is unique across an install, the ID shares the annotators’ namespace, and the registry refuses a chunker that states neither. It joins no descriptor, and the snapshot advertises the installed pairs on chunkers outside its content hash — what a chunker changes is how execution output is cut, which no outstanding plan recorded and no routing decision consults. Installing one moves nothing a plan is bound to.

Responsibilities and guarantees

Answer with nothing for a document it has no boundary for, so merging can continue; never invent pieces for a page it did not understand.
Never raise for a source it cannot open — the readers already gave those bytes their verdict, and a chunker has no standing to override it. Raising is reserved for the chunker’s own defects, and costs it the document, not the caller the encode.
Order pieces in reading order; the executor’s index and chunk IDs are derived from that order.
A page without text is chunked as a picture or not at all. The executor enforces it — text claimed for an unreadable page is dropped from the answer, an image is kept — because “a chunk of nothing is not addressable content” must not depend on every chunker remembering it.
Keep module scope cheap. Discovery imports every provider module while building a snapshot, so a heavy import belongs inside chunk().

Three attributes sit outside the protocol deliberately, read with a False default, because a member declared on it is a member isinstance and the type checker both demand. builtin = True is what the shipped chunkers set, and the registry orders installed chunkers ahead of them — claiming it can only cost precedence, so nothing verifies it. fallback = True is what indx-chunker-page alone sets, and sorts after everything: an answer for every page must be the last answer asked for, or nothing more specific would ever get one. opt_in = True is what a version sets to stay out of the order a request that names nothing is served by: it is installed, advertised with default: false, and asked only when EncodeRequest.chunking names it. At most one version of an ID may leave it unset, and the registry refuses two, because two defaults for one ID is install order deciding the output again. Two more are advertisement and nothing else: media_types, the types the chunker cuts (unset means any, which is what lines and page are), and summary, one phrase saying what a chunk is under this version. Both ride snapshot.chunkers, so a caller choosing between versions for a document can show the ones that apply and say what each does; neither is a gate, because a chunker that claims nothing already costs a page nothing.

Place in the system

DocumentExecutor.encode computes the chunk map once — only when the request asks for chunk granularity, since a chunker may reopen the document — and hands the same map to the embedding path and to blocks.build, so the chunk IDs vectors attach to and the chunk blocks that exist are one enumeration by construction. How many chunks a page yields is therefore installation-dependent, the way resolvable schemes and observable media types already are; one chunk per page is the floor wherever indx-chunker-page is installed.

A chunk that is a picture

A ChunkPiece carries text or image and never both or neither. The image half exists for one case: a page no capability could read. Such a page has no text for any chunker to cut, so it used to leave a page block carrying its reason and nothing retrievable at all — chunk_map dropped every page without text, and a chunk is the only thing document embedding ever vectorizes. indx-chunker-pdf renders it instead, at INDX_CHUNK_DPI (default 120), and an image embedder puts it in the same space as everything else.

Which pages qualify is deliberately narrow. A page that read as an empty string was read — it is blank — and rendering it would spend a rasterization to embed white; only a page a reader reported failed or unreadable is rendered. And a chunker may claim such a page only with an image: a page’s text is a reader’s verdict, reached down the plan’s ladder and past output validation with a capability ID and a trace event attached, so text asserted by a chunker for a page the ladder gave up on would launder an unvalidated read into the result. Pixels claim nothing — they are the page — so they are admitted where invented text is dropped. A page that will not rasterize keeps no chunk and fails no encode.

This is the one part of chunking that moved POLICY_VERSION, and only indirectly: rendering made DOCUMENT_EMBEDDING_MODALITIES gain image, so a space whose only document lane is an image went from an unsatisfied plan to a routable one. Nothing else here is a routing decision.