Skip to content

compare --depth binary skipping matrix-wide --sources/--build-info is tested only on snapshot operands, which cannot prove it

Index: Known gaps

Status (re-verified 2026-10-09 at 456989f): OPEN. tests/test_scenarios.py:653 test_sc_scan_binary_depth_matrix_args was revived, but per docs/contribute/usecase-registry.yaml:1040-1055 it runs on stored JSON snapshots (tests/scenarios/ci_gating.yaml:120), so it cannot tell 'binary depth skipped collection' apart from 'snapshot inputs always skip it'.

Found while retiring tests/scenarios/ci_gating.yaml's SC-SCAN-BINARY-DEPTH-MATRIX-ARGS scenario and its automated test (test_sc_scan_binary_depth_matrix_args) in the ADR-068 Phase 6 doc follow-up: the scenario's premise was that a CI matrix builds one command template and varies only --depth, leaving --headers/--sources/ --build-info present on every rung, and the binary rung must ignore the deeper inputs and skip pattern-scan/L3 collection. The retired scan command exposed this as a checkable coverage array (layer/status rows) plus a pattern_scan.files_scanned field; verified live, compare -o json=... emits neither at any --depth. compare --depth binary does reach the same verdict/exit code on the scenario's fixture, but nothing currently asserts it skipped the deeper collection rather than merely projecting an already-collected superset down afterward -- the two are observably different only through the now-gone coverage block.

Tractable when picked up: decide what compare's own coverage-reporting surface should look like for this question (a --depth-scoped rung either was or wasn't collected), add it, and un-retire the scenario against it.