Exporting Runs

context-grid is not a RAG framework and does not serve production traffic. What it produces is a decision, and a decision is only worth making if it can leave the tool. The winning configuration comes out as YAML you can commit, as runnable Python, and as a one-page report you can paste into a team decision doc — because nobody adopts a tool, they adopt an argument that came out of one.

write_bundle: everything, in one call

write_bundle writes the report, the raw results, the winning config and the manifest into a directory. Give it a real Results from a Lab run or Runner. Everything on this page runs against the same tiny two-file corpus — create it first:

from pathlib import Path

Path("my-docs").mkdir(exist_ok=True)
Path("my-docs/shipping.md").write_text(
    "Express shipping costs $15 and arrives within two business days of order "
    "placement. Standard shipping is free and takes 5 to 7 business days.\n"
)
Path("my-docs/refunds.md").write_text(
    "You have 30 days from the delivery date to return an item for a full "
    "refund. Items must be unused and in original packaging.\n"
)

import contextgrid as cg

CORPUS_DIR = "./my-docs"
corpus = cg.Corpus.from_dir(CORPUS_DIR)

evalset = cg.EvalSet(
    id="support-docs",
    version=1,
    items=[
        cg.EvalItem(
            id="q1",
            question="How much does express shipping cost?",
            gold=[cg.GoldSpan(span=cg.Span(doc_id="shipping.md", start=0, end=75), grade=1)],
        ),
        cg.EvalItem(
            id="q2",
            question="How long do I have to return an item?",
            gold=[cg.GoldSpan(span=cg.Span(doc_id="refunds.md", start=0, end=78), grade=1)],
        ),
    ],
)

lab = cg.Lab(corpus)
lab.grid(chunker=["recursive:128", "structural:128"], embedder="tfidf", index="dense")
results = lab.run(evalset)

manifest = cg.build_manifest(results.best("recall@5").config, corpus, evalset)
written = cg.write_bundle(
    results, "./bundle", manifest=manifest, name="support-docs-sweep",
    corpus=CORPUS_DIR, evalset=None,
)
for path in written:
    print(path)

bundle/report.md
bundle/results.json
bundle/winning-config.yaml
bundle/use_winning_config.py
bundle/manifest.json

resultsResults

The runs a sweep produced.

directorystr | Path

Where to write the bundle. Created if it does not exist.

metricstrdefault recall@5

Which metric decides the winner and heads the leaderboard.

manifestManifest | Nonedefault None

Attached to the report and written as manifest.json when given.

namestr | Nonedefault None

The sweep’s name, for the report’s title. Falls back to results.meta["name"].

corpusstr | Path | Nonedefault None

The documents the sweep ran over. Without it, winning-config.yaml falls back to a flat listing of pipeline fields — a record, not a config you can hand back to the tool. See the note on winning_config_to_yaml below.

evalsetstr | Path | Nonedefault None

Pass this too and the re-run can be scored, not just executed.

Exactly these five files are written, and only these:

File	Written by	Always written?
report.md	results_to_markdown	Yes
results.json	results_to_json	Yes
winning-config.yaml	config_to_yaml or winning_config_to_yaml	Only if results.best(metric) is not None
use_winning_config.py	config_to_python	Only alongside winning-config.yaml
manifest.json	manifest.save	If manifest= was given, or built from corpus= plus evalset=

Caution

Writing twice into one directory clears the first run out. Any of the files above that an earlier run left is deleted before this one writes, so a directory can only ever describe one run. Without that, a second run writing fewer files than the first left the previous run’s winning-config.yaml and use_winning_config.py in place — handing you the wrong experiment’s pipeline to re-run, next to a manifest.json that said it never happened. The way to write fewer files is the config file’s report.formats key (formats: [json] under report:, which is what contextgrid run reads); write_bundle itself has no formats parameter and always writes the full set above.

Only the file names in the table (plus experiment.yaml, which contextgrid run copies) are touched. Anything else in the directory is left alone. Files cleared that this run does not write are named in a UserWarning.

results_to_json

Every configuration and every number, for offline analysis — including the per-question scores, so a sceptic can re-run the statistics rather than take the summary on trust:

import json

print(list(json.loads(cg.results_to_json(results, manifest=manifest)).keys()))

['mode', 'cache', 'warnings', 'runs', 'manifest']

The top-level shape is {"mode", "cache", "warnings", "runs", "manifest"} (manifest only present when you pass one). Each entry in runs carries config, label, metrics, timings, cost, chunks, index_bytes, scored_queries, unresolved_gold, confidence_interval, by_type, failures, no_ground_truth, per_query, answers, retrieval and warnings — everything write_bundle shows in report.md, plus the raw per-question numbers the report only summarises.

results_to_markdown

The one-page report. Ordered the way somebody reads it, not the way it was computed: the conclusion first, then the evidence, then the caveats — a report that opens with a methodology section does not get read.

print(cg.results_to_markdown(results, manifest=manifest, name="support-docs-sweep"))

# support-docs-sweep — retrieval configuration comparison

## What to use

markdown · recursive:128 · tfidf · dense scored best on recall@5 at 1.000, across 2 configurations, scored on 2 questions. markdown · recursive:128 · tfidf · dense and markdown · structural:128 · tfidf · dense are not distinguishable on this eval set (n=2). The gap of +0.000 on recall@5 sits inside the confidence interval +0.000 to +0.000, so it is consistent with no difference at all. They scored identically on every single question, so this is not a close call between two different configurations -- they are behaving the same way. It runs locally at no cost per query, answering at under 1 ms p95.

> **The top two are not statistically distinguishable.** Either is a defensible choice on this evidence; pick on cost or latency instead.

## Score

**100/100 over 2 dimension(s): chunk, retrieval (not measured: embed, generation, parse)**

| Dimension | Score |
|---|---:|
| `chunk` | 1.000 |
| `retrieval` | 1.000 |

Comparable only against another score computed over the same dimensions.

## Leaderboard

| Configuration | recall@5 | p95 ms | $/1k queries | Chunks |
|---|---:|---:|---:|---:|
| `markdown · recursive:128 · tfidf · dense` | 1.000 | 0.0 | 0.0000 | 2 |
| `markdown · structural:128 · tfidf · dense` | 1.000 | 0.0 | 0.0000 | 2 |

## Which decision mattered

- **Chunker**: no measurable difference between the values tried.

> **These are averages over runs, not controlled comparisons.** In `ofat` mode each value appears in a different number of configurations, so a value that happens to sit in the baseline arm is averaged over different company than one that does not. Treat this as a pointer to what to sweep properly in `factorial` mode, not as a measured effect.

## Warnings

- **one_chunk_per_document**: recursive:128 produced 2 chunk(s) from 2 document(s), so each document is a single chunk and these scores rank documents rather than passages. The chunker axis cannot change a number it never touched -- sweep smaller sizes, or measure on longer documents
- **one_chunk_per_document**: structural:128 produced 2 chunk(s) from 2 document(s), so each document is a single chunk and these scores rank documents rather than passages. The chunker axis cannot change a number it never touched -- sweep smaller sizes, or measure on longer documents
- **evalset_at_ceiling**: every one of the 2 configurations scored recall@5 = 1.000. Nothing here can be ranked: the eval set is answered perfectly by all of them, so the sweep measured no difference rather than finding none. Ask harder questions, or compare at a smaller cut-off (recall@1) where there is room to separate them

## Reproducing this

- Manifest: `5abb81d925f2`
- Corpus: `a973d96ba2ec` (2 files)
- Eval set: `support-docs` v1 (`9bd2c880051b`)
- Resolution: coverage at 0.5
- context-grid 0.9.5 on Python 3.13.5

Two runs with the same manifest hash must produce identical numbers.

Caution

Read this fixture’s own warnings before you read its leaderboard. All three are real output, and together they say the sweep above measured nothing. It is kept that way on purpose — this page is about the shape of the bundle, and a two-file corpus makes it run in a second — but the leaderboard is not an example of a comparison worth acting on.

one_chunk_per_document, twice: both recursive:128 and structural:128 turned each of the two documents into exactly one chunk, because each document is shorter than 128 tokens. The chunker axis therefore never changed the text being indexed, so the two rows are the same pipeline under two names, and the identical 1.000 scores are what that looks like.

evalset_at_ceiling: two questions, both answered perfectly by everything. Even had the chunkers differed, there was no room left in the metric to show it.

That is the honest reading, and the report says so itself — which is the point. The tool is built to tell you when a number it just printed did not measure anything, and the sample above is a run where it had to. For a sweep whose leaderboard means something, give the chunker axis documents longer than its sizes and the eval set questions that not everything gets right; see Eval Set Quality for how many questions a given gap actually needs.

resultsResults

metricstrdefault recall@5

manifestManifest | Nonedefault None

Adds a “Reproducing this” section at the bottom, naming the manifest hash, corpus hash, eval set id and version, and the resolution policy used.

limitintdefault 15

How many leaderboard rows to print.

namestr | Nonedefault None

The sweep’s title. A caller with no name of its own should pass nothing rather than invent one — the bare word "experiment" is treated as no name at all, since that is what ExperimentConfig fills in when a config was never read from a file.

Sections appear only when they have something to say: generation columns (faithfulness, answer_relevancy, groundedness, citation_accuracy) show up only if a run actually generated answers, “Why the rest failed” only when the winner had failures, and “Warnings” only when a warning survived the run.

build_manifest and Manifest

“Reproducible” is a claim until something makes it checkable. Manifest is a hash-pinned record of everything that could change a number — the corpus contents, the eval set and its version, every parameter, the resolution policy, and library versions.

winner_config = results.best("recall@5").config
manifest = cg.build_manifest(winner_config, corpus, evalset)
print(manifest.short_hash)
print(manifest.to_dict())

5abb81d925f2
{'manifest_hash': '5abb81d925f227a314dcadf5fafea3347f173e2eb8a632478ba2f9a50352de41', 'config': {...}, 'corpus_hash': 'a973d96ba2ec299caee982fe89d454d7ee0706ddfe887e0db595e1c3fc1ba9a6', 'corpus_files': 2, 'evalset_id': 'support-docs', 'evalset_version': 1, 'evalset_hash': '...', 'resolution': {'policy': 'coverage', 'threshold': 0.5}, 'versions': {'contextgrid': '0.9.5', 'python': '3.13.x', 'platform': 'darwin'}, 'seeds': {}, 'created_at': '2026-...', 'notes': ''}

configConfig

corpusCorpus

evalsetEvalSet

resolverSpanResolver | Nonedefault None

Which resolution policy and threshold to record. Defaults to SpanResolver().

seedsdict[str, int] | Nonedefault None

notesstrdefault ""

Every Manifest field:

configdict[str, Any]

The winning Config, as config.as_dict().

corpus_hashstr

corpus.content_hash() — changes if any source file’s content changes.

corpus_filesint

len(corpus).

evalset_idstr

evalset.id.

evalset_versionint

evalset.version.

evalset_hashstr

A hash of every question and its gold evidence — catches an eval set edited without its version being bumped.

resolutiondict[str, Any]

{"policy": ..., "threshold": ...}.

versionsdict[str, str]

contextgrid, python, platform, plus numpy, pymupdf, pdfplumber, openai, anthropic when installed and imported.

seedsdict[str, int]

Whatever you passed in.

created_atstr

ISO timestamp. Excluded from the hash.

notesstr

Free text. Excluded from the hash.

Note

created_at and notes are recorded but excluded from .hash(). Two runs with the same manifest hash must produce identical numbers. If they do not, something outside the manifest is affecting results, and that is a bug worth finding.

Other Manifest methods: .hash() (the full sha256), .short_hash (first 12 characters), .save(path) / Manifest.load(path) to round-trip through JSON, and .matches(other) to compare two manifests’ hashes directly.

config_to_yaml

The pipeline’s fields as flat YAML — hand-written rather than via a YAML library, since the core installs with numpy and nothing else:

print(cg.config_to_yaml(winner_config, manifest=manifest))

# context-grid configuration
#
# manifest: 5abb81d925f2
# corpus:   a973d96ba2ec (2 files)
# evalset:  support-docs v1
#

ingestion: null
parser: markdown
chunker: "recursive:128"
embedder: tfidf
index: dense
transform: null
retrieval: null
reranker: null
k: 10
candidates: 50
generator: null

This is a record of the winning pipeline’s fields, not something contextgrid run can read back — there is no corpus: and no grid: wrapper. For a file you can actually re-run, use winning_config_to_yaml.

winning_config_to_yaml

The real thing: an experiment file contextgrid run accepts directly.

from pathlib import Path
from contextgrid.config.schema import ExperimentConfig, RunConfig

experiment = ExperimentConfig(
    corpus=Path(CORPUS_DIR).resolve(),
    run=RunConfig(headline="recall@5", k=winner_config.k),
    name="winning-config",
)
print(cg.winning_config_to_yaml(winner_config, experiment, manifest=manifest))

# context-grid configuration
#
# manifest: 5abb81d925f2
# corpus:   a973d96ba2ec (2 files)
# evalset:  support-docs v1
#
# Re-run this file directly:  contextgrid run winning-config.yaml
#

name: "winning-config"

corpus: "/Users/you/my-docs"

# One value per axis: this file names a single configuration, not a sweep.
grid:
  ingestion: null
  parser: markdown
  chunker: "recursive:128"
  embedder: tfidf
  index: dense
  transform: null
  retrieval: null
  reranker: null
  candidates: 50
  generator: null

run:
  mode: ofat
  k: 10
  headline: "recall@5"
  seed: 0
  resolution_policy: coverage
  resolution_threshold: 0.5
  machine_usd_per_hour: 0.0
  cache: memory
  model: null

corpus: and evalset: are written as whatever path experiment.corpus holds — str(experiment.corpus), verbatim. The example resolves it to an absolute path (/Users/you/my-docs above is this machine’s actual absolute path, shortened for the page) before building ExperimentConfig, and on purpose: this file usually lands in a report.out/ directory that sits below wherever the original config lived, and paths in a config resolve relative to the config file’s own location — a copied relative path would silently point somewhere else. write_bundle does this resolving for you automatically; calling winning_config_to_yaml directly, as above, makes it your job. k comes off the winning Config, not off experiment.run, since it is the config that actually won that this file describes.

budget_seconds and budget_usd are deliberately left out: they exist to stop a sweep that is taking too long or costing too much, and carrying them onto a single re-run could only cut that one run short.

config_to_python

Runnable Python that rebuilds one configuration — not a template with holes, this is code that actually runs:

print(cg.config_to_python(winner_config, corpus=CORPUS_DIR))

"""The winning configuration, as context-grid found it."""

import contextgrid as cg

# markdown · recursive:128 · tfidf · dense
# Any field not named below is at its default; `cg.Config()` puts it back.
config = cg.Config(
    chunker="recursive:128",
)

corpus = cg.Corpus.from_dir("/Users/you/my-docs")
pipeline = cg.build(config, corpus)

for chunk_id in pipeline.search("your question here"):
    print(chunk_id)

The corpus = line above is str(Path(CORPUS_DIR).expanduser().resolve()) — whatever you passed as corpus= to config_to_python, made absolute the same way. /Users/you/my-docs is this run’s actual path with the machine-specific prefix trimmed for the page; yours will show wherever your own CORPUS_DIR resolves to.

Field lines are read off Config at export time rather than hand-listed, so this cannot fall behind the dataclass. Only fields that differ from cg.Config()’s defaults are printed — the comment above config = cg.Config(...) says so explicitly. Call it without corpus= and the snippet is honest about that too:

print(cg.config_to_python(winner_config))

# Placeholder: this export was not told where the documents are.
corpus = cg.Corpus.from_dir("./documents")

explain_diff: the regression-triage tool

When a metric drops, diff the manifest against the last passing run — the changed line is the suspect. explain_diff writes that comparison in plain English:

manifest_a = cg.build_manifest(cg.Config(chunker="recursive:128"), corpus, evalset)
manifest_b = cg.build_manifest(cg.Config(chunker="recursive:256"), corpus, evalset)
print(cg.explain_diff(manifest_a, manifest_b))

1 thing(s) changed between these runs:
  config.chunker: 'recursive:128' -> 'recursive:256'

When nothing at all changed:

Nothing in these two manifests is different: same winning configuration, corpus, eval set, resolution policy, versions and seeds. On that evidence the two runs should have produced identical numbers for this one configuration.

That is not the same as the two runs being identical. A manifest records the configuration that won, not the sweep that found it, so two runs over different grids -- or over the same grid in different modes -- can both end up here.

If this configuration did score differently in the two runs, something outside the manifest is affecting results and that is worth finding.

diff(before, after) (also exported from contextgrid.report) returns the same information as a plain dict[str, tuple[Any, Any]], keyed like "config.chunker" or "corpus_hash", if you want to act on it programmatically instead of reading English.

Every file this page describes traces back to a real run through Lab and the grid; see Cost for where the $/1k queries column in report.md comes from, and Caching for how a warm cache changes the compute_seconds a manifest’s cost figures were built from.