Full example validation runbook¶

Use this workflow when a change can affect example detection, ground truth, fixtures, or validation harnesses. It is the source of truth for complete catalog accounting.

Choose the correct runner¶

Question	Runner
Is a single-library `v1`/`v2` case classified correctly?	`tests/validate_examples.py`
Does a case demonstrate a runtime effect?	`skills-src/evaluation/validation/scripts/run_example_runtime_smoke.py`
Are multi-library release bundles correct?	`skills-src/evaluation/validation/scripts/run_bundle_examples.py`
Do all non-compiler, non-bundle fixtures pass through public CLI workflows?	`skills-src/evaluation/validation/scripts/run_special_cli_examples.py`
Are audit, BTF, L3/L4/L5, Python API, reconcile, snapshot-pair, and KABI fixtures valid?	The dedicated pytest proof artifact below
Is every ground-truth case accounted for?	`skills-src/evaluation/validation/scripts/collect_full_example_matrix.py`
How accurate are evidence depths or external tools?	Benchmark/depth runners; measurement only

validate_examples.py alone is not the full catalog. A scan of directories that produce libv1.so/libv2.so also omits bundle, audit, fixture, Python, BTF, KABI, and other dedicated-owner cases.

Prerequisites and trust boundary¶

Run on Linux x86_64 with the repository development environment and the same tool dependencies as Examples Validation CI: gcc/g++, clang/clang++, CMake, Ninja, CastXML, and binutils.

A single default gcc/clang is not enough to build the full catalog: case115_bit_int_width_changed requires a C23 _BitInt-capable compiler (GCC 14+, or a recent Clang) — GCC 13 and earlier reject _BitInt(N) outright. The benchmark runners (scripts/benchmark_comparison.py, scripts/generate_benchmark_report.py) prefer a versioned gcc-15/gcc-14 binary on PATH over the bare gcc alias for this one case; without any of those installed, the case fails to build (reported ERROR) rather than building at reduced fidelity. tests/feature_probe.py-gated pytest lanes skip the case cleanly instead when the toolchain lacks the feature — the benchmark scripts do not (yet) have the same graceful fallback, so treat an ERROR on this case as a missing gcc-14+, not a product regression.

python -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[dev]"
mkdir -p results
export PYTHONPATH=.
export ABICHECK_TRUSTED_SOURCE_SMOKE_RUN=1

Activating the venv (rather than only installing into it) matters: every python/pytest invocation below relies on PATH resolving to .venv/bin/python, not an ambient interpreter that lacks the pinned dev extras. If you cannot activate (e.g. a non-interactive shell), replace every bare python command in this runbook with .venv/bin/python instead.

The source-smoke run mode executes fixture commands. Enable it only for a trusted checkout containing reviewed repository-owned fixtures. Its default is intentionally disabled. Without this opt-in, source-smoke-owned cases can be SKIP in both compiler lanes and the collector reports them UNRESOLVED.

Reproduce the full matrix¶

1. Validate dedicated owners¶

python skills-src/evaluation/validation/scripts/run_example_owner_proofs.py --json \
  > results/example-owner-proofs.json

The proof runner executes every dedicated owner separately and records its command, exit code, and bounded output in a machine-readable artifact.

2. Produce lane artifacts¶

python tests/validate_examples.py --toolchain gcc --json > results/validate-examples-gcc.json
python tests/validate_examples.py --toolchain clang --json > results/validate-examples-clang.json
python tests/validate_examples.py \
  case01 case04 case98 case105 case122 case129 case130 case131 case132 case133 \
  --artifact-variant build-source --json > results/validate-examples-build-source.json
python skills-src/evaluation/validation/scripts/run_example_runtime_smoke.py --json > results/example-runtime-smoke.json
python skills-src/evaluation/validation/scripts/run_bundle_examples.py --json > results/bundle-examples.json
python skills-src/evaluation/validation/scripts/run_special_cli_examples.py --json > results/special-cli-examples.json

3. Aggregate one row per case¶

python skills-src/evaluation/validation/scripts/collect_full_example_matrix.py \
  --gcc results/validate-examples-gcc.json \
  --clang results/validate-examples-clang.json \
  --runtime results/example-runtime-smoke.json \
  --build-source results/validate-examples-build-source.json \
  --bundle results/bundle-examples.json \
  --special-cli results/special-cli-examples.json \
  --proofs results/example-owner-proofs.json \
  --out results/full-example-matrix.json

The collector exits non-zero for UNRESOLVED or FAILED. Never use --allow-unresolved for a release or correctness gate.

4. Verify the gate¶

python - <<'PY'
import json
from pathlib import Path

d = json.loads(Path("results/full-example-matrix.json").read_text())
total = d["ground_truth_cases"]
assert len(d["results"]) == total
assert d["summary"] == {"COVERED": total}
assert not d["artifact_errors"]
assert not d["unresolved_cases"]
assert not d["failed_cases"]
direct = d["direct_coverage"]
# A case can be COVERED two ways: directly (a compiler lane or public-CLI
# workflow reached the canonical verdict) or via known-gap-oracle (a
# reviewed known_gap plus the case's own source_smoke oracle proves the
# canonical truth while every evidence tier still XFAILs — see case111).
# direct_coverage deliberately excludes the latter, so asserting
# direct["covered"] == total is wrong on a fully-green matrix whenever any
# case is covered that way. Assert the two provenances sum to the total
# instead of hard-coding which cases (if any) are oracle-only.
known_gap_oracle = d["coverage_by_provenance"].get("known-gap-oracle", 0)
assert direct["covered"] + known_gap_oracle == total
print(
    f"{total}/{total} COVERED; direct={direct['covered']}/{direct['total']}, "
    f"known-gap-oracle={known_gap_oracle}"
)
PY

The exact count comes from catalog/ground_truth.json; automation must not hard-code a historic count. When this runbook was added, the proven result was 181/181 COVERED. As of this writing that is 180 direct + 1 known-gap-oracle (case111) — a case only qualifies for known-gap-oracle provenance when its own declared source_smoke oracle proved the canonical verdict; see _single_library_status in collect_full_example_matrix.py.

Interpret results¶

Ground truth is invariant across scan depths. A reduced-evidence lane that cannot observe the expected change is an XFAIL detection gap, not a second lane-specific expected verdict. Case98 demonstrates this: L0–L2 observes NO_CHANGE against the single COMPATIBLE_WITH_RISK truth, while the L3 build-source lane proves the expected build-flag risk.
FAILED: a lane ran and contradicted ground truth, or a proof failed.
UNRESOLVED: no owner lane proved the case; inspect lanes, proof_lane, and note in that row.
artifact_errors: a required runner artifact was missing, malformed, partial, produced by the wrong runner/toolchain, or came from different ground truth.
A compiler-lane SKIP is acceptable only when another designated owner proves that case and the final row is COVERED.
provenance=compiler means GCC or Clang demonstrated a compilable pair. abicheck-cli-workflow means a public CLI command demonstrated a special input shape or richer build-source evidence. The dedicated-owner artifact remains a separate regression proof, but does not substitute for the required public-CLI artifact.
G20 audit risks are advisory at the default scan gate: the CLI emits the expected cross-check kinds/providers and exits 0 with COMPATIBLE. The special runner validates that contract instead of falsely requiring the comparison-only COMPATIBLE_WITH_RISK label.
Runtime statuses describe behavior; they do not replace verdict proof.

Agent checklist¶

Read catalog/ground_truth.json; it defines scope and ownership.
Use the smallest owner runner while iterating.
Before claiming full-catalog success, obtain every artifact above.
Record commit SHA, tool versions, commands, exit codes, and artifact paths.
Claim success only when collector JSON has one row per case and every row is COVERED, with no unresolved or failed cases.
Never substitute pair counts, scan-depth totals, benchmark accuracy, or one green compiler lane for the full matrix.
Keep generated binaries and ad-hoc results out of commits.

CI implementation: .github/workflows/examples-validation.yml. Keep this runbook synchronized with its full-matrix job.