Why three chunkers

A chunk is the unit a caller retrieves and the unit the playground draws over the page, so a chunk wants two things: text a reader produced, and a box saying where on the page it sits. The readers do not all leave the same thing behind, and that is the whole reason there are three built-in chunkers rather than one.

×

The PDF chunker reopens the bytes with PDFium, the engine the native reader used, and cuts where the file’s own layout put the glyphs. It needs the file to be a PDF with a text layer, so it disclaims a scanned page: it has no geometry for text OCR produced. It is also the one that renders a page nothing could read into a single image chunk, so that page has a vector at all.

The lines chunker never opens the bytes. An OCR engine knows where every line it read sits, and generic-ocr states that on PageOutput.lines, a field of the port model that never reaches the wire. The lines chunker cuts one chunk per located line at its box, for any media type, without a second engine run (ADR-0047). A DXF drawing’s text arrives located the same way. A reader that states no lines gets nothing from it.

The page chunker is the floor. It claims every readable page of any media type with its whole text as one chunk and bbox_reason: no_geometry, which is what a vision model’s answer, a CSV or an email has. It sets fallback = True, so it sorts after everything and only fills pages nothing more specific claimed. Without it, a stock pip install indx returns no chunk blocks.

One page, one writer

The executor asks the chunkers in order and merges per page, first answer wins (ADR-0012). Answering nothing for a page is how a chunker says “not mine”, so each can be honest about what it knows.

×

That is the ministry notice in the samples: twenty-five native pages and three scanned 別添 pages in one document. No single chunker could cut all twenty-eight with boxes, because the boxes come from two different places. The trace names who cut what, and the playground’s route box repeats it per page.

The same three, in versions

A better boundary for the same pages is a new version of the same chunker, installed beside the old one (ADR-0067): pdf v2 and lines v2 group the same lines into paragraphs. The three families stay, because the geometry still comes from three places; what a version changes is how much of it one chunk spans.

×

The slide chunker is the same idea for a fourth source of geometry, the shapes a .pptx file states, and is left out here only because it never competes with these three for a page.