CLI Reference

Installing context-grid also installs a contextgrid command. It reads the same YAML experiment file contextgrid.config.load reads in Python, so a sweep you built as a script can be handed to a teammate, a Makefile, or CI without them needing to know any Python at all.

contextgrid --help

usage: contextgrid [-h] [--version]
                   {run,init,check,profile,sweep,plugins,evalset,validate,diff} ...

Sweep retrieval configurations on your own documents.

positional arguments:
  {run,init,check,profile,sweep,plugins,evalset,validate,diff}
    run                 Run everything a config file describes.
    init                Write a starter config for this installation.
    check               Validate a config and say what it would run.
    profile             Measure a corpus and flag settings its shape rules
                        out.
    sweep               Run a matrix and print the leaderboard.
    plugins             List everything registered.
    evalset             Inspect an eval set and what it can support.
    validate            Check the scorer against a published benchmark.
    diff                Say what changed between two run manifests.

options:
  -h, --help            show this help message and exit
  --version             show program's version number and exit

Running contextgrid with no command prints this same help and exits 1. contextgrid --version prints context-grid <version> (e.g. context-grid 0.9.5) and exits 0.

Every command catches its own exceptions and prints error: <message> to stderr rather than a Python traceback, then exits 1. Ctrl-C counts — it prints one line and exits 130, the code every shell already uses for a process stopped by an interrupt:

error: stopped by Ctrl-C. Everything parsed, chunked and embedded so far is in the cache on disk, so re-running this command picks up from there rather than starting again.

That second sentence is true only for run.cache: disk. On the default memory cache the message says so instead, and tells you which setting would have kept the work.

The three commands below are the ones most people reach for first — init to get a config file, check to catch mistakes in it for free, run to actually sweep.

contextgrid init — write a starter config

contextgrid init contextgrid.yaml --corpus ./documents --evalset ./questions.jsonl

wrote contextgrid.yaml
edit it, then run:  contextgrid run contextgrid.yaml

pathpathdefault contextgrid.yaml

Positional, optional. Where to write the file.

--corpusstrdefault ./documents

The corpus: value written into the file.

--evalsetstrdefault ./questions.jsonl

The evalset: value written into the file.

--forcebooldefault false

Overwrite an existing file at path.

The file it writes lists every value each axis can take, as a comment next to the one or two values it actually sets — grid: chunker: [recursive:512, sentence:3] with # also available: chonkie:code, chonkie:recursive, ... underneath, one block like that per axis. Read it top to bottom once and you have the shape of the whole tool.

Without --force, running init again over an existing file refuses rather than clobbering it:

error: contextgrid.yaml already exists. Pass --force to overwrite.

Exit code 1 for that refusal (or a directory that can’t be written to), 0 once the file is written.

contextgrid check — validate before spending anything

contextgrid check contextgrid.yaml

contextgrid: 1 × 1 × 2 × 2 × 3 × 1 × 1 × 2 × 1 × 1 = 24 on paper, 5 to run in ofat mode (1 impossible combination(s) skipped), scored on recall@5
  ingestion   ['plain']
  parser      ['markdown']
  chunker     ['recursive:512', 'sentence:3']
  embedder    ['tfidf', None]
  index       ['dense', 'bm25', 'hybrid']
  transform   [None]
  retrieval   ['simple']
  reranker    [None, 'lexical']
  candidates  [50]
  generator   [None]

config is valid.

configpath

Positional. The YAML or JSON experiment file to check.

check parses the file, then does everything run would do except read a document, embed a vector, build an index or call a model:

• Confirms corpus: exists, and that it actually has files a registered parser can read (not just that the path is present — an empty ./documents from a clone without its data fails here, not four stages into run).
• Confirms evalset: exists, parses, and has at least one question in it. An eval set that parses fine but is empty is caught by run too — it exits 1, because a sweep that scored no questions measured nothing — but only after building and indexing the whole matrix to find out. check catches it for the price of opening one small text file.
• Builds one instance of every ingestion strategy, parser, chunker, embedder, index, transform, retrieval strategy, reranker and generator the matrix actually names, so a typo’d spec string (chunker: banana:999) or an out-of-range one (chunker: recursive:-5, candidates: -3) is reported now, in the same words run would use, rather than after the sweep has started spending time or money.
• Refuses any of those that needs a model when run.model is not set — transform: hyde, retrieval: agentic, generator: llm and the paid ingestion strategies (contextual, summary, hypothetical-questions, propositions) alike. Each error names run.model and lists the model-free values on that axis.
• Checks each of those specs against its optional dependency, so a plugin that imports fine but defers its real import to call time — marker is the one plugin in this package that does — is still caught rather than silently passed.
• Confirms report.out can actually be written, by writing one temporary file there and removing it. Writing the report is the last thing a sweep does, so a read-only output directory used to validate, run the whole matrix, and die on manifest.json with everything discarded. A report.out that does not exist yet is fine and is not created here — the nearest existing directory above it is what gets tested, since that is what mkdir needs.

Every problem it finds is printed as error: <message> and the command exits 1; a config with nothing wrong prints config is valid. and exits 0.

Note

This is why check is worth running in CI even for a config you’re confident in: it is the same cost as importing the package and building empty objects, not the cost of a sweep.

contextgrid run — run everything the config describes

contextgrid run contextgrid.yaml

  [1/5] markdown · recursive:512 · tfidf · dense
  [2/5] markdown · sentence:3 · tfidf · dense
  [3/5] markdown · recursive:512 · bm25
  [4/5] markdown · recursive:512 · tfidf · hybrid
  [5/5] markdown · recursive:512 · tfidf · dense · lexical@50
contextgrid: 1 × 1 × 2 × 2 × 3 × 1 × 1 × 2 × 1 × 1 = 24 on paper, 5 to run in ofat mode (1 impossible combination(s) skipped), scored on recall@5

configuration                                         recall@5   p95 ms     $/1k
--------------------------------------------------------------------------------
markdown · recursive:512 · tfidf · dense                 1.000      0.2   0.0000
markdown · sentence:3 · tfidf · dense                    1.000      0.0   0.0000
markdown · recursive:512 · bm25                          1.000      0.0   0.0000
markdown · recursive:512 · tfidf · hybrid                1.000      0.0   0.0000
markdown · recursive:512 · tfidf · dense · lexical@50    1.000      0.1   0.0000

markdown · recursive:512 · tfidf · dense scored best on recall@5 at 1.000, across 5 configurations, scored on 2 questions. ...

cache: 17 of 24 lookups reused (71%), chunk 6/10, embed 3/4, parse 8/10

wrote 6 files to /path/to/results

configpath

Positional. The YAML or JSON experiment file to run.

--quietbooldefault false

Suppress the [i/total] <config.label> progress lines that print to stderr as each configuration finishes. The leaderboard, summary and cache line on stdout are unaffected.

The [i/total] progress lines and every warning: line go to stderr — including anchor_normalised, which fires when your evidence matched only after whitespace was collapsed, because Markdown hard-wraps and a quote copied out of it usually spans a newline. The leaderboard, the plain-English summary sentence, and the cache: line go to stdout — so contextgrid run contextgrid.yaml > leaderboard.txt captures just the results, and 2> catches just the noise.

If report.out is set in the config (the default starter config sets it to ./results), run writes the formats named under report.formats there and prints wrote N files to <report.out> — see Reports for what each format contains.

Exit codes:

Exit	When
0	At least one configuration ran and scored at least one question.
1	The config file itself is invalid (same errors check would catch), or the sweep measured nothing — no configuration ran at all, or every one of them ran and scored zero questions.
2	The command line itself did not parse — an unknown flag, an unknown subcommand, a missing positional. argparse prints the usage message to stderr and exits before any config file is read.
130	Stopped by Ctrl-C. One line on stderr, no traceback, and no report written.

That 1 case matters for CI: a matrix whose only cell can’t be built, a run.budget_usd: 0.0 that leaves nothing to spend, and an eval set with no questions in it all used to print a leaderboard and still exit 0 — a green build for a sweep that measured nothing. Both run and sweep now treat nothing measured as a failure, whichever way it got there.

Note

sweep uses this same table, and it did not always. The check that turns nothing measured into exit 1 was wired into run alone, so the same empty eval set exited 1 through a config file and 0 through the flags-only command — the one more likely to be sitting in a CI job. sweep now prints the same error: lines and returns the same code. A partial sweep stopped by --budget-seconds still exits 0: it measured something, just not everything.

The reasons go to stdout, under the empty leaderboard they explain, and to stderr as error: lines. Both, deliberately: contextgrid run contextgrid.yaml > leaderboard.txt is the capture this page recommends, and a captured failure with no reason in it is worse than no capture at all. At most three distinct reasons print under the table; the rest are in the warnings.

A matrix whose only cell cannot be built — say embedder: [null] with index: [dense], a dense index with nothing to search:

error: no configurations were run, so nothing was measured
error: 1 combination(s) in this matrix cannot be built and were skipped -- a dense index with no embedder has nothing to search. The axes you wrote are almost certainly what you meant; this is just the product of them that is not

A matrix that ran every configuration and scored none of them — an empty questions.jsonl, or one whose every anchor names a document the corpus does not hold:

error: no configuration scored a single question, so nothing was measured
error: the eval set 'support-kb' has no questions in it, so every configuration was built and indexed and then scored nothing. Every metric is absent rather than zero, and none of them says anything about any configuration. Add questions to the eval set, or point `evalset:` at the file you meant

The leaderboard above those lines prints NOT_MEASURED in the metric column rather than 0.000, for the same reason: a zero is a measurement, and nothing there measured anything.

Note

Needing a model is not the same as being impossible, and the two fail in different ways. An impossible combination is skipped — the rest of the matrix runs without it. A plugin that needs a model and has none stops the command: transform, retrieval, generator and ingestion all refuse, name run.model, and list the model-free values on their own axis. Both check and run say it, in the same words, before anything is read or spent.

A sweep that runs some of its configurations before a run.budget_seconds cutoff still exits 0 — the leaderboard it printed is real, just partial, and a warning: budget_reached: ... line says so.

The other six commands, briefly

Each of these has its own --help; this is the one-line summary from contextgrid --help plus what it’s for.

Command	For
contextgrid profile <corpus> [--parser markdown]	Measure a corpus’s shape (file count, size, structure) and print hints about which axis values it rules out, before you spend a sweep finding out.
contextgrid sweep <corpus> <evalset.jsonl> [--parser ...] [--chunker ...] ...	A one-shot sweep from flags alone, no YAML file — the CLI equivalent of building a cg.Lab and calling .grid() in Python. Same exit codes as run, including 1 for a sweep that measured nothing. See The Lab.
contextgrid plugins [--family parser]	List every registered name on every axis, for this installation specifically — a name whose optional package is missing is marked - and the pip install line that fixes it is printed underneath.
contextgrid evalset <path>	Load a JSONL or CSV eval set and print its quality summary and question-type distribution, without running a sweep. See Eval Set Quality.
contextgrid diff <before.json> <after.json>	Explain what changed between two manifest.json files from earlier bundled runs. Two spellings of one configuration are not a change: retrieval: simple and no retrieval at all run the same search, so they are folded before anything is compared, along with ingestion: plain, transform: none and reranker: none.
contextgrid validate <benchmark.json> <corpus> [--limit N] [--recall-at-10 N]	Check this package’s own scorer against a published benchmark (LegalBench-RAG format) — a sanity check on the tool, not on your pipeline.

Sweeping an axis from flags: repeat the flag. --parser, --chunker, --embedder, --index and --reranker each collect every occurrence, so one value per flag and the flag as many times as you have values:

contextgrid sweep ./corpus ./questions.jsonl \
  --chunker recursive:256 --chunker recursive:512 --chunker sentence:3 \
  --index dense --index bm25

A comma-separated list is not supported and is not a silent no-op — --chunker recursive:128,recursive:256 reads as a single chunker named recursive with a stray second parameter, and the command says so and exits 1 before running anything.

contextgrid validate exit codes

validate is meant to be run in CI, so its verdict is in the exit code and not only in the report it prints.

Exit	When
0	Every metric you supplied a published number for is within 0.05 of it — or you supplied none, in which case the command only reports its own numbers and there is nothing to miss.
1	A published number was supplied and was not reproduced, or fewer than 95% of the benchmark’s gold spans fall inside the documents as loaded.

The second 1 comes first and stops the run: if the corpus is not the one the annotations were made against, nothing measured afterwards means anything, and the cause is loading rather than retrieval.

The first used to be a 0. contextgrid validate bench.json ./docs --recall-at-10 0.90 printed recall@10 differs by +0.100, outside the 0.05 tolerance and “anything left over is a problem with our scoring, not with the benchmark” — and then exited 0, so a scorer that missed the benchmark was a green build. That is the same trap run closed when it stopped exiting 0 for a sweep that measured nothing.

Note

Without --recall-at-10 there is no claim to check, so the command always exits 0. Passing it is what turns validate from a report into an assertion.

Caution

A config’s plugins: key — used to load your own custom Chunker, Embedder, or other plugin before the CLI resolves any spec string against a registry — behaves differently from Python. See Writing a Custom Plugin before relying on one from contextgrid run.