Metrics

Two inputs feed every metric in context-grid: qrels (which chunks are actually relevant to each question, and at what grade) and run (which chunk ids your pipeline returned, in rank order). Both are plain dicts:

qrels = {"q1": {"c1": 2, "c2": 1, "c3": 0}, "q2": {"c4": 2}}
run = {"q1": ["c3", "c1", "c5", "c2"], "q2": ["c9", "c4", "c1"]}

qrels[question_id] maps chunk id to grade (2 fully relevant, 1 partial, 0 irrelevant — same IR convention as GoldSpan.grade). run[question_id] is a ranked list of chunk ids, best first. In practice you get qrels from a SpanResolver and run from a pipeline search — see Diagnosing Failures and Search — but any dict in this shape works, which is what makes the examples below self-contained.

The six built-in metrics

Registered under cg.METRICS, resolved by name — not by cg.X import:

import contextgrid as cg
print(cg.available_metrics())
# ('hit_rate', 'map', 'mrr', 'ndcg', 'precision', 'recall')

Registry name	What it measures
recall	fraction of relevant chunks that made it into the top k. 0.0 if the question has no relevant chunks.
precision	distinct relevant chunks in the top k, divided by k itself — not by how many chunks the run actually returned. A run that only returns 3 chunks still divides by k=10.
hit_rate	1.0 if at least one relevant chunk is in the top k, else 0.0.
mrr	1 / position of the first relevant chunk in the top k, else 0.0.
map	mean average precision.
ndcg	normalized discounted cumulative gain, using the actual grade as the gain (so a grade-2 hit outweighs a grade-1 hit). 0.0 if the ideal ranking would also score 0.

Caution

map’s denominator is len(relevant) — the trec_eval/MAP@k convention — not min(len(relevant), k). A question with 20 relevant chunks caps at 0.25 at k=5 no matter how perfect your top 5 is. This is intentional, not a bug, but it is the most common “why does my MAP look low” surprise.

A chunk id counts once, however often it is repeated

run[question_id] is a list of distinct chunk ids. Every metric here treats it as one: a chunk that appears twice is one chunk that was retrieved and one slot that was wasted, not two retrievals. c1, c1, x at k=3 scores exactly what c1, y, x scores.

Without that rule the arithmetic breaks its own scale — recall@3 of 1.5 on a scale whose top is 1.0 — and a retriever scores better for the bug of returning the same chunk three times.

Caution

evaluate() and per_query() go further and refuse a run whose ranking repeats a chunk id for a question they were going to score, raising ValueError that names the question and the repeated ids:

import contextgrid as cg

try:
    cg.evaluate({"qa": {"c1": 2, "c2": 2}}, {"qa": ["c1", "c1", "c1", "c2"]}, ks=[3])
except ValueError as e:
    print(e)
# the run for query 'qa' returns the same chunk id more than once: 'c1'. A ranking is a list
# of distinct chunks, and scoring one with repeats in it would report a retriever's bug as a
# score. De-duplicate the ranking, keeping each chunk at its best position.

Rejected rather than quietly repaired, and rejected rather than clamped to 1.0. A repaired number goes on a leaderboard looking reasonable and leaves the broken retriever in place. If you are merging two runs yourself, de-duplicate first — keep each chunk at its best position.

A ranking for a question the qrels never judged is never averaged, so it is never checked.

You can pull a single metric out of the registry directly:

recall_metric = cg.get_metric("recall")
print(recall_metric)
# RecallMetric()

get_metric only accepts the plain registry name. Passing a cut-off along with it, like cg.get_metric("recall@5"), raises:

try:
    cg.get_metric("recall@5")
except Exception as e:
    print(type(e).__name__, e)
# UnknownPluginError no metric named 'recall@5'. Available: hit_rate, map, mrr, ndcg, precision, recall

Cut-offs: k, DEFAULT_KS, and the metric@k result keys

A metric on its own doesn’t know about k — k is passed in at call time. evaluate() scores every metric at every k in ks and names each result f"{metric}@{k}":

DEFAULT_KS = (1, 3, 5, 10, 20)  # cg.DEFAULT_KS

So "recall@5", "ndcg@10", and so on are result labels that evaluate() builds, not strings you feed back into get_metric or the registry. Keep that straight: the registry name is always bare ("recall"); the @k suffix only exists on the way out.

Every k must be at least 1. 0 and negative values raise ValueError rather than being scored — ranked[:-1] is a perfectly valid Python slice meaning “all but the last”, which is not a cut-off anybody asked for:

try:
    cg.evaluate({"q1": {"c1": 2}}, {"q1": ["c1"]}, ks=[-1], metrics=["recall"])
except ValueError as e:
    print(e)
# cut-off k must be at least 1, got -1. A k of 0 or below is not a smaller top-k, it is a
# slice that means something else.

evaluate() — score a whole run

def evaluate(
    qrels: Qrels,
    run: Mapping[str, Sequence[str]],
    *,
    ks: Sequence[int] = DEFAULT_KS,
    metrics: Sequence[str] = ("recall", "precision", "hit_rate", "mrr", "map", "ndcg"),
    warnings: WarningLog | None = None,
) -> dict[str, float]

import contextgrid as cg

qrels = {"q1": {"c1": 2, "c2": 1, "c3": 0}, "q2": {"c4": 2}}
run = {"q1": ["c3", "c1", "c5", "c2"], "q2": ["c9", "c4", "c1"]}

scores = cg.evaluate(qrels, run, ks=[1, 3], metrics=["recall", "ndcg"])
print(scores)
# {'recall@1': 0.0, 'recall@3': 0.75, 'ndcg@1': 0.0, 'ndcg@3': 0.5552773433538603}

Only questions with at least one chunk judged relevant (grade above 0) get averaged over. A question that’s in qrels but that run never answered for scores 0.0 for that question — that’s a real retrieval failure, not something evaluate() quietly excludes.

{} and {"c1": 0} are the same thing and score the same

Both spell “nothing here is relevant to this question”, so both are left out of the mean:

run_two = {"q1": ["x"], "q2": ["c9"]}
print(cg.evaluate({"q1": {"c1": 0}, "q2": {"c9": 2}}, run_two, ks=[3], metrics=["recall"]))
print(cg.evaluate({"q1": {},        "q2": {"c9": 2}}, run_two, ks=[3], metrics=["recall"]))
# {'recall@3': 1.0}
# {'recall@3': 1.0}

These used to disagree — the grade-0 spelling was scored as a question that scored zero and halved the mean, the empty one was excluded. A resolver writing a grade-0 row for every question it couldn’t resolve would have silently halved every number in a sweep. per_query() and mean_rank_of_first_relevant() follow the same rule.

If nothing in the qrels has a relevant chunk, evaluate() returns an empty dict — there was nothing to measure, and an empty result says that where a 0.0 would not.

Caution

metrics defaults to the six built-ins by name, not to “whatever’s currently registered” — a custom metric you cg.METRICS.register(...) yourself does not show up in evaluate()’s output unless you list its name in metrics= explicitly.

Passing a name that isn’t registered at all raises ValueError up front. But a registered metric that raises partway through scoring is caught per-metric and silently dropped from the result dict — it does not appear as 0.0, it just isn’t a key in the returned dict at all. Pass warnings=cg.WarningLog() to actually see it happened:

from dataclasses import dataclass
from typing import ClassVar, Mapping, Sequence

# A registered metric that raises partway through scoring -- not an unregistered name,
# which would raise ValueError before scoring even starts.
@dataclass(frozen=True, slots=True)
class BrokenMetric:
    name: ClassVar[str] = "broken"
    version: ClassVar[str] = "1"

    def evaluate(self, judgements: Mapping[str, int], ranked: Sequence[str], k: int) -> float:
        raise RuntimeError("oops")

cg.METRICS.register("broken")(BrokenMetric)

warnings = cg.WarningLog()
result = cg.evaluate(qrels, run, ks=[5], metrics=["recall", "broken"], warnings=warnings)
print(result)
# {'recall@5': 1.0}   -- 'broken@5' is just absent, not zero
for w in warnings.entries:
    print(w)
# CAUTION [score] (broken): the 'broken' metric raised RuntimeError('oops') and was left out
# of this run's results rather than silently scoring zero

This is also why RunResult.has(name) exists elsewhere in the toolkit — checking for a key’s presence, not trusting a 0.0, is the only safe way to tell “this metric scored zero” apart from “this metric never ran.”

per_query() — one metric, one k, per question

def per_query(qrels: Qrels, run: Mapping[str, Sequence[str]], metric: str, k: int) -> dict[str, float]

pq = cg.per_query(qrels, run, "recall", 3)
print(pq)
# {'q1': 0.5, 'q2': 1.0}

This is the shape Is the Winner Real? needs — a {question_id: score} dict per configuration, which its paired tests line up by question id. Only questions with at least one relevant chunk in qrels are included — same rule as evaluate(). An unregistered metric, a k below 1, and a ranking with a repeated chunk id all raise ValueError here too.

Character-level scores: catching context waste

Chunk-level recall answers “did a relevant chunk come back.” It says nothing about how much irrelevant text came back alongside it. A config can score recall@5 = 1.0 while burying the actual answer in a huge chunk full of padding — the character-level functions are what expose that:

Function	Signature	What it’s for
character_recall	(item: EvalItem, retrieved: Iterable[Chunk]) -> float	fraction of the gold evidence’s characters that appear somewhere in what was retrieved. 0.0 if the item has no gold spans.
character_precision	(item, retrieved) -> float	fraction of the retrieved characters that are actually gold. 0.0 if nothing was retrieved.
character_f1	(item, retrieved) -> float	harmonic mean of the two above.
retrieved_character_count	(retrieved: Iterable[Chunk]) -> int	total characters retrieved, overlapping spans merged and counted once.
gold_coverage_by_chunk	(item: EvalItem, chunks: Sequence[Chunk]) -> dict[str, float]	per-chunk fraction of the gold evidence each individual chunk holds — feeds a “this chunk holds 60% of the evidence” display, not a flat relevant/not-relevant mark.

import contextgrid as cg

item = cg.EvalItem(
    id="q1",
    question="What was the settlement amount?",
    gold=(cg.GoldSpan(cg.Span("doc1", 100, 270)),),
)

# same evidence, retrieved two different ways
big_chunk = cg.Chunk(id="c1", span=cg.Span("doc1", 0, 2000), text="x" * 2000)
small_chunk = cg.Chunk(id="c2", span=cg.Span("doc1", 100, 270), text="x" * 170)

print(cg.character_recall(item, [big_chunk]), cg.character_precision(item, [big_chunk]))
# 1.0 0.085
print(cg.character_recall(item, [small_chunk]), cg.character_precision(item, [small_chunk]))
# 1.0 1.0

Both chunks give perfect recall@k and perfect character_recall — the evidence is in both. But character_precision is 0.085 for the big chunk against 1.0 for the small one: only 8.5% of what came back is actually the answer. A generator reading the big chunk has to find the needle itself; a generator reading the small chunk doesn’t.

print(cg.retrieved_character_count([big_chunk, small_chunk]))
# 2000  -- small_chunk's span is entirely inside big_chunk's, so it adds nothing new

print(cg.gold_coverage_by_chunk(item, [big_chunk, small_chunk]))
# {'c1': 1.0, 'c2': 1.0}  -- both chunks individually hold 100% of this gold span

Three more scoring functions that aren’t on this page’s main list

cg.coverage_fraction, cg.recall_against_exact and cg.score_answer (with its result type cg.AnswerScore) are all top-level exports that score something, so people reasonably come looking for them here. They are documented in full on the page that owns the thing they measure — spans, approximate indexes and generated answers respectively. Definitions, ranges and a worked example for each are below so you don’t have to go and find out whether it’s the one you want.

None of them is a METRICS plugin: they don’t take (judgements, ranked, k), they can’t be named in evaluate(metrics=...), and they don’t appear in available_metrics().

coverage_fraction(target, others) -> float

How much of one span is covered by a set of other spans, as a fraction in [0, 1]. Overlaps between the others count once. This is the character-level machinery under character_recall above, exposed on its own — see Spans and Offsets.

0.0 when target is empty, and spans in a different document contribute nothing.

import contextgrid as cg

gold = cg.Span("refund.md", 100, 200)          # 100 characters of evidence
first = cg.Span("refund.md", 80, 150)          # covers 100-150
second = cg.Span("refund.md", 150, 190)        # covers 150-190

print(cg.coverage_fraction(gold, [first, second]))
# 0.9   -- 90 of the gold's 100 characters; 190-200 was never retrieved
print(cg.coverage_fraction(gold, [first]))
# 0.5
print(cg.coverage_fraction(gold, [cg.Span("other.md", 100, 200)]))
# 0.0   -- same offsets, different document

The gold span split across two chunks is the case it exists for: neither chunk alone holds enough to clear a per-chunk threshold, and together they hold 90% of the answer.

recall_against_exact(approximate, exact, k) -> float

What fraction of exact search’s top k an approximate index also found — the number that turns “quantization feels fine” into a decision. Both arguments are lists of Scored, which is not a top-level name. Full context in Indexes.

from contextgrid.index.base import Scored

exact = [Scored("c1", 0.9), Scored("c2", 0.8), Scored("c3", 0.7)]
approx = [Scored("c1", 0.9), Scored("c9", 0.75), Scored("c3", 0.7)]

print(cg.recall_against_exact(approx, exact, k=3))
# 0.6666666666666666   -- 2 of exact's 3 survived
print(cg.recall_against_exact(approx, exact, k=1))
# 1.0                  -- the top hit is the same one

Caution

An empty exact list returns 1.0, not 0.0 and not an error. It reads as “perfect recall” when what actually happened is “there was nothing to compare against” — check that your exact run returned something before you believe the number.

1.0

print(cg.recall_against_exact(approx, [], k=3))

score_answer(item, answer, context, gold_chunks=()) -> AnswerScore

Judges one generated answer without calling a second model. Deliberately lexical — it compares word sets, so it is coarse and checkable rather than sensitive and unaudited. Full treatment on Answering and Generation.

AnswerScore carries six fields and one property:

Field	Range	What it means
item_id	—	the EvalItem.id this scored
groundedness	[0, 1]	fraction of the answer’s words that appear in the context it was given. Low means invented or recalled from training.
citation_accuracy	[0, 1] or None	fraction of cited passage numbers that were actually in the context. None when the answer cited nothing.
evidence_overlap	[0, 1]	fraction of the gold chunks’ words the answer used. 0.0 unless you pass gold_chunks.
abstained	bool	the answer declined rather than guessed
should_have_abstained	bool	there was nothing to answer from — no resolved gold, or no chunks in the context
warnings	list of str	plain-English notes about what looked wrong
.abstention_correct	bool	abstained == should_have_abstained. A correct refusal scores as a success.

item = cg.EvalItem(
    id="q1",
    question="What is the refund window?",
    gold=(cg.GoldSpan(cg.Span("refund.md", 0, 46)),),
)
gold_chunk = cg.Chunk(
    id="g1",
    span=cg.Span("refund.md", 0, 46),
    text="Refunds are available within 30 days of purchase.",
)
context = cg.AssembledContext(
    text="Refunds are available within 30 days of purchase.",
    chunks=(gold_chunk,),
    tokens=12,
)

good = cg.Answer(text="Refunds are available within 30 days.", citations=(1,))
print(cg.score_answer(item, good, context, gold_chunks=[gold_chunk]))

AnswerScore(item_id='q1', groundedness=1.0, citation_accuracy=1.0, evidence_overlap=0.75, abstained=False, should_have_abstained=False, warnings=[])

Every word of the answer came from the context (groundedness=1.0), the one citation was real (citation_accuracy=1.0), and it used three quarters of the gold chunk’s words.

An answer that invented its content and cited a passage that was never there:

invented = cg.Answer(text="Contact the vendor by fax immediately.", citations=(1, 4))
score = cg.score_answer(item, invented, context, gold_chunks=[gold_chunk])
print(score.groundedness, score.citation_accuracy)
for w in score.warnings:
    print("-", w)

0.0 0.5
- only 0% of the answer's words appear in the context it was given, so most of it came from somewhere else
- cited passage(s) that were not in the context: [4]

And a refusal when there was genuinely nothing to answer from, which is a success:

empty = cg.AssembledContext(text="", chunks=(), tokens=0)
declined = cg.score_answer(item, cg.Answer(text="The passages do not contain the answer."), empty)
print(declined.abstained, declined.should_have_abstained, declined.abstention_correct)
# True True True

Note

should_have_abstained uses EvalItem.is_resolved, not is_answerable. An item whose anchor exists but which this parse lost has no evidence to answer from, so declining is right and is scored as right. is_answerable would be True for that item and would mark the refusal wrong.