DocumentStore and VectorIndex

Ports: DocumentStore and VectorIndex, declared in indx_interfaces.protocols over the models in indx_interfaces.store (ADR-0071).

What

DocumentStore keeps encoded documents: a StoredDocument is a StoredDocumentSummary beside the whole EncodeResult. VectorIndex keeps one ChunkPoint per chunk – its text, page and box, and a BlockEmbedding per space it was embedded in, possibly none – and answers a ChunkQuery with ChunkHits. Both are keyed by an owner and a source digest.

class DocumentStore(Protocol):
    def put(self, owner: OwnerId, document: StoredDocument) -> None: ...
    def get(self, owner: OwnerId, digest: SourceDigest) -> StoredDocument | None: ...
    def list(self, owner: OwnerId) -> tuple[StoredDocumentSummary, ...]: ...
    def delete(self, owner: OwnerId, digest: SourceDigest) -> bool: ...
    def purge(self, now: datetime) -> int: ...


class VectorIndex(Protocol):
    def upsert(
        self,
        owner: OwnerId,
        digest: SourceDigest,
        points: tuple[ChunkPoint, ...],
        spaces: tuple[EmbeddingSpace, ...],
        expires_at: datetime,
    ) -> None: ...
    def delete(self, owner: OwnerId, digest: SourceDigest) -> None: ...
    def search(self, owner: OwnerId, query: ChunkQuery) -> tuple[ChunkHit, ...]: ...
    def purge(self, now: datetime) -> int: ...

Why

Two ports because the backends split there: pgsql with pgvector implements both, a vector database such as qdrant implements only VectorIndex, and an in-memory store implements both. A backend that holds points never fakes a record store. Three ship: memory (indx-store-memory), sqlite (indx-store-sqlite, one file named by INDX_STORE_SQLITE_PATH, so what was kept survives a restart) and postgres (indx-store-postgres, a database named by INDX_STORE_POSTGRES_URL, shared by every instance). All three rank a leg with indx-ranking, so the same points give the same hits in the same order whichever keeps them.

A ChunkQuery is one leg, a vector in a named space or a text, because both backends answer both natively and a lexical port added later would be a second port. It is not both at once: a similarity and a term score share no scale, so the caller fuses the legs by rank and keeps each one’s score.

Contract

• Owner. Every method but purge takes an OwnerId first. Nothing crosses owners: a digest another owner kept is, to this owner, a digest never kept. The owner is never a model field, so nothing on the wire can name another one.
• Identity. Within an owner, a document is its source digest. StoredDocument refuses a summary whose digest is not its result’s plan.source_digest.
• Replacement. put replaces a document with the same digest, and upsert replaces every point the digest had.
• Idempotence. delete of a digest that is not there is False (records) or a no-op (points).
• Expiry on read. get, list and search never return anything whose expires_at is at or before now. purge(now) reclaims space for every owner and returns how many documents went; correctness never waits for it.
• Expiry on both ports. upsert is handed the document’s expires_at, because an index cannot join to the records to learn it.
• Spaces. upsert is handed the EmbeddingSpace of every vector in its points, because a vector column or a collection is created with a dimension and a metric. A vector leg scores by its space’s metric, higher nearer.
• Order. list returns the most recently updated first. search returns at most top_k hits, best first; a vector leg only over points with a vector in the space it names, a text leg over every point with text, and, when digests is set, only from those documents. A score is on the backend’s own scale and is comparable within one response only.
• Synchronous, like the facade (ADR-0024). A caller runs it off the event loop.

Discovery

A backend is an entry point in indx.document_stores or indx.vector_indexes whose value is a zero-argument factory. The factory reads its own settings. A deployment names one of each, and the two may come from different distributions. Neither joins indx.capabilities: a store is never planned, and the capability snapshot’s hash must not move for it.

Related reading

• Extension protocols
• Embedder: what produces a query vector in a space a ChunkQuery names.