Building One Pipeline

The Lab sweeps many configurations and scores them against an eval set. Most of the time that is what you want. cg.build() is the piece underneath it, for the times it is not: you already know the configuration, you don’t have an eval set yet, or you want to poke at chunks, timings, or an index directly.

Note

cg.build() runs the indexing half of a pipeline only — parse, chunk, embed, index. It does not run queries or score anything. That is what the BuiltPipeline it returns is for.

The call

import contextgrid as cg

corpus = cg.Corpus.from_dir("./documents")
config = cg.Config(parser="markdown", chunker="recursive:512", embedder="tfidf", index="dense")

pipeline = cg.build(config, corpus)

Config is a frozen dataclass — every field is a spec string ("recursive:512", "tfidf", …), which is what makes a configuration paste-able and diff-able. The axes and their spec grammar are covered per-axis under Axes; Config covers the dataclass itself. Corpus is covered in Corpus — Corpus.from_dir(path), Corpus.from_files(paths), and Corpus.from_texts({...}) for building one in memory without touching disk.

Run against a two-file corpus:

import contextgrid as cg

corpus = cg.Corpus.from_texts({
    "refunds.md": "refunds take within 30 days of purchase. digital goods are not refundable once downloaded.",
    "shipping.md": "express shipping arrives the next business day.",
})
config = cg.Config(parser="markdown", chunker="recursive:512", embedder="tfidf", index="dense")
pipeline = cg.build(config, corpus)

print(type(pipeline).__name__)
print("chunks:", len(pipeline.chunks))
print("index_bytes:", pipeline.index_bytes)
print("embed_tokens:", pipeline.embed_tokens)
print("build_ms:", pipeline.timings.build_ms)

BuiltPipeline
chunks: 2
index_bytes: 168
embed_tokens: 21
build_ms: 0.391...

Full signature:

cg.build(
    config: Config,
    corpus: Corpus,
    *,
    cache: Cache | None = None,
    stats: CacheStats | None = None,
    llm: LLM | None = None,
) -> BuiltPipeline

cache and stats are covered below. llm is only needed if config.transform, config.retrieval, or config.generator names something model-backed (hyde, agentic, llm, …) — see Transforms, Retrieval, and Generating an Answer. Building a config that names one of those without passing llm raises before anything is indexed, not partway through.

What you get back

BuiltPipeline is a plain dataclass holding everything the indexing stage produced, plus the methods that use it — search(), answer(), run_queries(), chunk_by_id(), all covered on the next two pages.

The fields

Field	Type	What it is
config	Config	The configuration that produced this pipeline
parses	dict[str, ParsedDocument]	Every source file’s parse, keyed by source id
chunks	list[Chunk]	What actually got indexed and can come back from a search
index	Index	The built index — pipeline.index.search(...) if you need to bypass search()
embedder	Embedder | None	None exactly when config.embedder is None (e.g. plain bm25)
timings	Timings	Wall-clock per stage — see below
warnings	WarningLog	Everything that went sideways without stopping the build
index_bytes	int	index.size_bytes()
embed_tokens	int	Tokens spent embedding the corpus
vectors	Any	The chunk vectors the index was built from, or None with no embedder
reranker	Reranker | None	Built from config.reranker, None if unset
transform	QueryTransform	Built from config.transform; a NoTransform identity when unset, never None
retrieval	RetrievalStrategy	Built from config.retrieval; a SimpleRetrieval default when unset
ingested	Ingested | None	What was indexed vs. what a hit resolves to — see Ingestion
trace	RetrievalTrace	Accumulates what the retrieval strategy did across every query run through search()
assembler	ContextAssembler	Turns retrieved chunks into the text a generator sees — built even without a generator
generator	Generator | None	None unless config.generator named one — see Generating an Answer

transform, retrieval, and assembler are the three fields on this list that are never None — each has a no-op default so downstream code doesn’t need to branch on whether the axis is switched on. embedder, reranker, ingested, and generator can genuinely be None.

Timings

Timings(
    parse_ms: float = 0.0,
    chunk_ms: float = 0.0,
    embed_ms: float = 0.0,
    index_ms: float = 0.0,
    query_ms: list[float] = [],
)

Kept per stage on purpose — “this config is slow” isn’t actionable, “this config spends 90% of its time in the reranker” is.

• build_ms (property) — parse_ms + chunk_ms + embed_ms + index_ms. Everything build() itself did.
• query_ms — starts empty. build() never queries anything; each call to search() or run_queries() appends one entry.
• percentile(fraction) — latency at a percentile, e.g. timings.percentile(0.95) for p95. Empty query_ms returns 0.0 rather than raising.
• as_dict() — the above as a flat dict (parse_ms, chunk_ms, embed_ms, index_ms, build_ms, query_p50_ms, query_p95_ms, query_p99_ms), which is what the reporting side reads.

print(pipeline.timings)

Timings(parse_ms=0.119..., chunk_ms=0.086..., embed_ms=0.167..., index_ms=0.019..., query_ms=[])

Caching across builds

Pass a Cache to reuse work between calls — the same instance, called twice:

import contextgrid as cg

corpus = cg.Corpus.from_texts({
    "refunds.md": "refunds take within 30 days of purchase.",
    "shipping.md": "express shipping arrives the next business day.",
})
config = cg.Config(parser="markdown", chunker="recursive:512", embedder="tfidf", index="dense")

cache = cg.MemoryCache()
cg.build(config, corpus, cache=cache, stats=cache.stats)
cg.build(config, corpus, cache=cache, stats=cache.stats)
print(cache.stats)

CacheStats(hits=5, misses=5, writes=5, by_stage={'parse': [2, 2], 'chunk': [2, 2], 'embed': [1, 1]})

Caution

cache=cache alone reuses cached work but does not update cache.stats — build() takes cache and stats as two separate parameters, and nothing wires one to the other for you. Leave off stats=cache.stats and every build looks like it hit nothing, hits or no hits.

DiskCache works the same way and survives the process, for reusing work across separate contextgrid run invocations — see Caching.

When it fails

If a parser reads no text at all from the corpus — the wrong parser for the file type, or a scanned PDF with no text layer — build() raises CorpusError before anything downstream runs, rather than silently indexing zero chunks:

import contextgrid as cg

corpus = cg.Corpus.from_texts({"a.pdf": "hello world"}, media_type=cg.MediaType.PDF)
config = cg.Config(parser="text", chunker="recursive:512", embedder="tfidf", index="dense")

try:
    cg.build(config, corpus)
except cg.CorpusError as e:
    print(e)

the 'text' parser read no text from 1 file in corpus 'corpus' (a.pdf), so none of them are in
this index at all and nothing can be retrieved. These files are application/pdf. Usually the
parser does not read the file types in this corpus -- check 'text' against them -- or the
files have no text layer and need OCR first.

If some files parse and others don’t, build() doesn’t raise — it drops the unreadable ones and adds a PARSER_FALLBACK warning to pipeline.warnings instead, naming which source id was skipped. Check pipeline.warnings.entries (or pipeline.warnings.summary()) after a build you’re not sure about.

Text files have to be UTF-8

The text and markdown parsers read UTF-8 and nothing else. A file that isn’t UTF-8 is not decoded approximately — it’s skipped, exactly like a file whose media type the parser declines, with a PARSER_FALLBACK warning naming it and saying what it looks like instead:

from pathlib import Path
import contextgrid as cg

Path("docs").mkdir(exist_ok=True)
Path("docs/good.md").write_text("# Returns\n\nRefunds take five days.\n")
Path("docs/latin1.md").write_bytes("Café orders are final\n".encode("latin-1"))
Path("docs/b.md").write_bytes(bytes(range(256)) * 16)

pipeline = cg.build(
    cg.Config(parser="markdown", chunker="recursive:256", embedder="tfidf", index="dense"),
    cg.Corpus.from_dir("docs"),
)
for w in pipeline.warnings:
    print(w.code.value, "-", w.message)
print(sorted({c.doc_id for c in pipeline.chunks}))

parser_fallback - 'b.md' is not UTF-8 text -- invalid start byte at byte 128 -- so it is not in this index at all. Nothing in it can be retrieved. It looks like a binary file with a text extension -- most of its bytes are not text at all. Check what it really is, and drop it from the corpus or give it the right extension so a parser that reads that format can be chosen.
parser_fallback - 'latin1.md' is not UTF-8 text -- invalid continuation byte at byte 3 -- so it is not in this index at all. Nothing in it can be retrieved. It looks like text in another encoding, most likely Latin-1 or Windows-1252. Convert it and run again: iconv -f windows-1252 -t utf-8 latin1.md
['good.md']

Losing a document is bad. Indexing a broken one is worse, and much harder to notice: decoding Latin-1 bytes as UTF-8 turns Café into Caf�, which gets embedded, retrieved and scored like any other text. The only symptom used to be an anchor_not_found warning naming the parser — so people changed parsers, twice, and never suspected the file.

Note

A leading UTF-8 byte order mark is dropped before parsing and never appears in document.text or in any offset. Without that, the first line of a BOM’d file reads as # Returns, which is not a heading, and the document’s first heading disappears from every heading_path — silently, and Windows editors write a BOM by default.

If every file in the corpus fails to decode, the CorpusError above carries these warnings forward rather than guessing at the parser:

the 'markdown' parser read no text from 1 file in corpus 'docs' (b.md), so none of them are in
this index at all and nothing can be retrieved. These files are text/markdown. Usually the
parser does not read the file types in this corpus -- check 'markdown' against them -- or the
files have no text layer and need OCR first. What the parse actually reported: 'b.md' is not
UTF-8 text -- invalid start byte at byte 128 -- ...

Unreadable files

Corpus.from_dir and Corpus.from_files raise CorpusError when a matched file cannot be opened at all — a file the current user has no permission to read is the usual case:

try:
    cg.Corpus.from_dir("./docs")
except cg.CorpusError as e:
    print(e)

docs/n.md could not be read: Permission denied. Every file the corpus patterns match has to be
readable by the user running contextgrid -- fix its permissions, or narrow the corpus so it is
not matched: `Corpus.from_dir(path, patterns=[...])`.

The original PermissionError is kept as the exception’s __cause__, so nothing is lost — but the type you catch is the documented one from Errors, not a bare OS exception.

Build vs. the Lab

Reach for cg.build() directly when you want one configuration and nothing else — a script, a notebook, a service that serves a fixed pipeline. Reach for cg.Lab when you want to compare configurations against each other on an eval set: it calls build() under the hood, once per point in the grid, and adds scoring, caching defaults, cost estimates, and a leaderboard on top.

Next: Searching covers what a BuiltPipeline does with a query, and Generating an Answer covers turning a search result into text.