One Semantic Pipeline — unifying application, fact, identity, and outcome models¶
ADR: ADR-063 · Proposed — roadmap
ADR, partially implemented. Phase 0's Fact[T]/FactStatus infrastructure
and AI-readiness gates have landed, and a narrow Phase 1 slice has too; see
each phase's own "Landed"/"Still not landed" paragraph below in this plan
for the authoritative per-slice detail, plus
docs/_meta/one-semantic-pipeline-status.yaml for the concept-level
authority summary. ADR-063 itself no longer restates this detail — its own
duplicated status block was removed (PR 0, 2026-09-02) precisely because it
kept drifting out of sync with this plan.
Effort: XL, multi-quarter, phased — do not attempt as one PR. Depends on / sequences with: ADR-055, ADR-061,
ADR-062, ADR-042, ADR-046/048, ADR-049, ADR-050 (each partially implemented
already; see "Sequencing against in-flight ADRs" below).
Problem¶
AGENTS.md's own "Known gaps" section documents, over dozens of numbered findings, one recurring root cause: the same concept (an input, a config value, a fact's availability, an entity's identity, a semantic result) is represented more than once in the codebase, and the representations drift out of agreement. ADR-063 states the target architecture — and, in its own "Governing invariant" section, the one rule every phase here exists to enforce: one concept, one representation, everywhere it is used, never two. This document is the phased, file-level plan to get there without a rewrite and without ever leaving two live implementations of the same concept standing side by side for longer than one phase. That is not a style preference either document treats as negotiable: a phase whose own PR leaves the representation it was meant to replace still reachable by any caller is an incomplete phase, not a phase with follow-up work, no matter how much of the new representation it built.
Three constraints shape every phase below, taken directly from this repository's own conventions (AGENTS.md, ADR-061's migration discipline) and from the governing invariant above:
- Vertical slice, not flag day. Each phase ships one consolidation, behavior-preserving, independently mergeable, independently revertible.
- Delete after consolidating — same PR or the very next one, never "eventually." A phase is not done when the new path works; it is done when the old path it replaces is removed and nothing in the repository can still reach it. A phase that only adds is half a phase, and "half a phase, to be finished later" is exactly the accumulation pattern this plan exists to stop — it does not get counted as progress toward consolidation until the deletion half lands.
- Verify at the size of the change. A phase touching the compare/scan hot path re-runs the FP-rate gate, the tier-accuracy gate, and the mutation-score gate for any module it touches; a phase touching persisted schema adds a v(N) migration test and a round-trip test at production scale (per this repo's "third-party-boundary tests" and "toolchain pins" conventions).
Sequencing against in-flight ADRs¶
This plan does not start from zero. Four backing ADRs are already partially implemented, and their current state determines what each phase below can assume:
| Backing ADR | Current state | What this plan's phases assume |
|---|---|---|
| ADR-055 (typed request/result) | D1 implemented for compare only |
Phase 1 extends the existing CompareRequest/service_compare_pipeline.py shape to dump (a real DumpRequest/resolve_dump_request/execute_dump_request pair, already built) and finishes routing dump's own real execution onto it — scan's ScanRequest already exists separately and its candidate resolution already converges on the shared execution primitive, so Phase 1 does not extend this shape to scan, only to dump |
| ADR-061 (responsibility packages) | Phases 0-1 implemented; Phase 5 (model package) begun |
Phase 0/2/4/7 of this plan land inside the model/compare/policy packages ADR-061 already created; this plan does not create new top-level packages beyond what ADR-061 names |
| ADR-062 (storage v2) | Phase 0 primitives (abicheck/storage/: FactStatus/FactAvailability, occurrence-preserving identity, canonical encoding, version axes) implemented and inert — nothing wired to a producer/reader |
Phase 0/5 of this plan is the generalization of these primitives into the domain layer; Phase 8 of this plan is the wiring ADR-062 Phase 1 still needs, done jointly rather than twice |
| ADR-042 (compatibility/gate separation) | Implemented for JSON/SARIF/compare-release; workflows/aggregate/gate.py/fold.py still decode exit codes inline |
Phase 7 of this plan closes the gate.py/fold.py gap — not a redesign of ADR-042 itself. junit_report.py's own inline _is_failure computation is not one of the gaps Phase 7 closes: that phase's own corrected design leaves it exactly as it is, since it is a legitimate per-render function of each call's own SeverityConfig/relevant_ids, not a property a finding carries — there is no RunOutcome field for it to read instead, so it stays inline by design, not as an unclosed gap |
| AGENTS.md "PR C" (dump/scan typed convergence) | resolve_dump_request/execute_dump_request split landed; scan's candidate resolution already converges on the shared workflows.artifact.execute._resolve_side_snapshot_impl primitive that execute_dump_request itself calls internally (resolve_dump_request only validates evidence and builds a ResolvedDumpRequest — it never calls the primitive; service_input_resolution is only a delegating facade re-exporting this module's owner), but dump's own real ELF/PE/Mach-O execution still runs the legacy path, blocked on two named items (castxml availability for parity testing, --compile-db-filter typed surface — now closed) |
Phase 1 of this plan is exactly "finish PR C" for the dump half, not a new design |
Phase 2B-8B: consumer-cutover extension (accepted 2026-09-02)¶
An external review of the codebase as of merge commit aa78c37 (PR #1015)
assessed this plan against ADR-063's own governing invariant rather than
against each phase's individually-scoped acceptance criteria. Its findings
and its six proposed sub-phases were reviewed and accepted on
2026-09-02 (this PR) — they are now part of this plan's roadmap, not a
standing proposal. A per-concept, machine-readable summary of "who is
authoritative today, and what has to land before the legacy representation
can be removed" now lives in
docs/_meta/one-semantic-pipeline-status.yaml; update it in the same PR
that changes a concept's status rather than letting this prose and that
ledger drift apart the way ADR-063's own Phase 2/6 status paragraphs
briefly did (see PR #1019).
The review's conclusion: the primitives this plan calls for — Fact[T],
EntityId/ScopePath, SemanticIR, AnalysisPlan, RunOutcome, sectioned
storage, ReportDocument — are, in large part, built, but they have
mostly been added alongside the legacy representations they were meant to
replace, not yet made the sole authority those legacy representations funnel
through. That is a materially different claim from "phases remain," and
the review's own headline numbers make the shape of the gap concrete:
infrastructure/building-blocks roughly 75-80% built; actual consumer cutover
onto the one semantic model roughly 35-45% done; legacy-representation
removal under 15% done. Concretely, the review found:
SemanticIRis real, persisted, and (since Phase 6's third/fourth slices) covers records/enums/typedefs/functions/variables/constants on both header-AST backends — but no detector, verdict, or exit path reads it; the checker still reads the legacyAbiSnapshot.functions/types/… collections exclusively, withsemantic_ircomputed and stored beside them. Two independently-mutable representations of the same entities existing side by side is exactly the shape ADR-063's governing invariant exists to forbid, even though each individual phase that produced this state met its own stated acceptance criteria.- Phase 0/5 converted
Fact[T]representation and populated the registry (seeabicheck/model/fact_registry.py'sFACT_REGISTRYfor the current count — this document doesn't freeze it;KNOWN_UNCONVERTED_ELIGIBLE_FACTSis empty), but every migrated reader still resolves throughresolved_fact_value(fact, legacy_default)or an equivalent local helper doing the identical present-or-default collapse — a review finding on this PR named three:diff_param_qualifiers._fact_bool,diff_cxx_rules._fact_str_list,compare.surface_graph.fact_list. Each reads.is_present/.valuedirectly rather than calling the shared primitive, but all collapseNotCollected/Unsupported/Failedback onto the sameFalse/None/[]default the legacy field always held, so 5B's inventory of what to migrate is every such default-collapsing unwrap, not only literalresolved_fact_valuecall sites. This was Phase 0's own explicit, correct acceptance bar ("representation, not detector logic"), but it means no fact anywhere has reachedFactLifecycle.CONSUMED— a detector branching onFactStatusat all remains unscheduled work, named as out of scope by every phase that touches facts rather than owned by any of them. AnalysisPlan(Phase 4) is deliberately a narrow, side-effect-free preflight object by design (so--dry-runstays free of ambiguous resolution) — but that means no single resolved runtime object exists for real execution either; policy, pack, contract, and public-surface state are each still (re)resolved independently downstream of it.- Phase 3's closure-walk-not-graph-traversal outcome (see ADR-063's own
D5 "Amendment" note and
docs/contribute/known-gaps.md's three-review- round account) is a correct, reviewed decision given the demonstrated hazards of reading a mergeable evidence graph for a value with exactly one legitimate source — but it leaves D5's own text unfulfilled, andAbiSnapshot.surface_graphgenuinely has no production reader today. action/run.shstill decodes raw process exit codes and its own shellcaselogic into a verdict rather than readingrun_outcomefrom the report (Phase 7's own named, deliberate scope boundary — real, not an oversight, but still a second live representation of the same outcome).- Sectioned storage (Phase 8) is real and is now the default write shape,
but most sections still carry the pre-existing JSON encoding for their
fields rather than a typed domain DTO —
semantic_iris the only section actually promoted to one so far. buildsource/entity_identity.py'sCanonicalIdentity(L5) remains a second, deliberately out-of-scope identity system alongsideEntityId(Phase 2's own named scope boundary) — permanently duplicate under a literal reading of "one concept, one representation," pending a bridge or reclassification neither this plan nor ADR-063 currently schedules.
None of this contradicts any individual phase's own "Landed"/"Still not
landed" accounting elsewhere in this plan — every landed slice met its
own stated acceptance criteria at the time it landed. (ADR-063 itself no
longer carries this per-phase accounting at all: PR 0 removed its
duplicated status block, leaving this plan's own per-phase sections as the
primary source, plus the machine-readable ledger at
docs/_meta/one-semantic-pipeline-status.yaml for the concept-level
authority split — see the "Adopted sub-phases" table above for how the two
relate.) That is narrower than "every phase is complete": Phase 3's
own D5 text remains unfulfilled for the reason recorded in its "Amendment"
note, Phase 6's acceptance-criteria fixture (a closure-parameterized
template) is still unmet, and several later phases are explicitly still in
progress — each says so plainly in its own status paragraph elsewhere in
this document, and this review does not dispute any of those self-reports.
The review's point is that the plan's phase
list, read end to end, did not until this update contain a phase whose job
is "make the new representation the only one a consumer can reach" for
SemanticIR, Fact semantics, or a single resolved runtime context — only
Phase 9 (selectors) and Phase 10 (title only, unimplemented) were framed
around deletion/authority at all, and Phase 10 itself still documents that
detector migration and legacy-projection retirement are separate future
work it does not itself schedule. The six sub-phases below close that gap.
2026-09-05 re-assessment of this extension (accepted)¶
A second external review, of main at the merge of PR #1062, re-checked the
six sub-phases below against the implementation rather than against their
own completion labels. It was reviewed and accepted on 2026-09-05.
Its findings, reasoning, retirement table, and parallel-track
decomposition are owned by
duplication-and-convergence-assessment.md
("2026-09-05 re-assessment" and Phase 6), which already owns cross-cutting
retirement work — read it there rather than a second copy here.
Its governing correction, which applies to how every row below is closed:
"no current bug found" can justify preserving behavior, never a second
implementation of that behavior. investigated_declined is therefore a
separate disposition on the introduced → wired → authoritative → retired
ladder, and it leaves a removal gate open.
What that means for this table specifically — the rows it corrects, and which track closes each:
| Row | Correction | Track |
|---|---|---|
| 2B/6B | The landed typedef and constant cohorts are fidelity gates, not authority transfers: both indexes are built every comparison and the legacy projection adjudicates | T3 |
| 4B | Built, not unwired, on both paths — what is open is consumption: partial on compare (classify_compare_pair reads requested_depth), absent on dump |
T4 |
| 5B | The vtable "final closure" is an investigated decline of a behavioral change; the PDB fabrication path and the authority transfer are still open |
T9 |
| 7B | action/run.sh is partially migrated — the residual is the raw exit/stderr path |
T8 |
One finding bears on this plan's own guardrail design rather than on a row:
scripts/semantic_ir_cutover.py's module-level "no legacy attribute read"
rule is satisfied by the typedef selector while the legacy projection
still decides, so it is not evidence of authority. Each cohort's guard must
cover the whole dependency path — callers, adapters, selectors, producers —
and the legacy-writer retirement condition.
Adopted sub-phases — each slots after the parent phase it extends in
the numbering, and each starts at status not_started. This table is the
status owner for the six sub-phases themselves; the ledger
(docs/_meta/one-semantic-pipeline-status.yaml) tracks status one level
up, per concept (facts/identity/semantic_ir/…), and each
concept's own removal_gate names which sub-phase closes it — the two are
complementary, not the same data twice. Update the Status column below
when a first PR lands against a sub-phase:
| Sub-phase | Status | Closes | One-line goal |
|---|---|---|---|
| 2B — Identity consumer migration | in progress | Phase 2's remaining string-identity call sites | Migrate diff_filtering.py/type_reachability.py onto EntityId once DWARF-side blockers clear; split EntityId/OccurrenceId matching into a StableEntityId tier (cross-release, suppression-alias-safe) vs. a SnapshotLocalIdentity fallback, rather than a further attempt at globally-stable Anonymous/LocalToFunction ordinals (two prior attempts already reverted). The StableEntityId/SnapshotLocalIdentity split landed (abicheck/model/identity_tiers.py), and two post-parse consumers migrated onto it — diff_filtering.py's opaque-type suppression (compare/opaque_types.OpaqueTypeIndex, matching on StableEntityId first and the RecordType.name spelling second) and type_reachability.py's closure-walk record-tracking keys (SnapshotLocalIdentity rather than bare str). Bare-name-collision narrowing landed (2026-09-03, see the note below the table): OpaqueTypeIndex.complete gates contains(..., strict=...) on both sides' stable tier being provably complete for the comparison at hand, closing the collision exactly when doing so cannot drop a real suppression. The entity: alias promotion in finding_identity.resolve_change_identity/report_canonical_finding_id was investigated (2026-09-03) and declined rather than deferred: no currently-identifiable finding needs it (typedefs/constants already get a fine NORMALIZED tier off their own spelling, functions/variables already get the stronger CANONICAL mangled tier), so wiring it in would trade real suppression-file compatibility for a benefit that cannot be demonstrated today — see the ledger's identity concept for the full reasoning |
| 4B — Resolved execution context | in progress | The gap between AnalysisPlan's deliberately narrow preflight scope and real execution's need for one resolved object |
A ResolvedExecutionContext built only for real (non-dry-run) execution — effective config with per-field provenance, resolved compile/toolchain context, effective/available evidence depth, resolved policy/pack/contract — so downstream code stops independently re-reading .abicheck.yml, re-deriving precedence, or re-resolving severity scheme. First PR landed, in two slices (see Phase 4's own "Adjacent, additive infrastructure landed" note, below, under "Phases"): the ResolvedExecutionContext type itself (composing an AnalysisPlan with an already-resolved CompatibilityEvaluationConfig/CompileContexts, provenance access, and a resolution digest), then a second slice closing the "requested/effective/available depth" field list via a new EvidenceView (copied off AnalysisAssurance, never re-derived). A third slice (2026-09-03) landed the sub-phase's first real call site: service_compare_pipeline.resolve_compare_request now builds one from its own already-resolved AnalysisPlan and attaches it on ResolvedComparePair — additive, unread by any consumer yet, but no longer "a dataclass nothing outside its own tests constructs" for the compare path. A fourth slice (2026-09-04, "Track 3") closed the "unread" half of that gap for one concrete site: classify_compare_pair's own DiffResult.requested_depth stamp — previously a second, independent request.depth.lower() normalization living a few lines away from the identical one ResolvedExecutionContext.from_plan already performs — now reads pair.resolved_execution_context.requested_depth instead, when it agrees with the call's own request.depth (Codex review, PR #1047: this function's own two-phase split lets a caller pass a different request than built pair, and old/new are always projected to this call's request.depth, so a disagreement defers to it rather than reporting a depth the classification never actually saw), falling back to the direct computation for a caller that attaches no context at all (a hand-built ResolvedComparePair, as some unit tests still do). classify_compare_pair is the typed Python API's run_compare_request path specifically -- the native compare CLI calls compare_snapshots() directly and does not go through it. Correction to this row's own prior text: resolve_dump_request/execute_dump_request are not "fully unwired" -- execute_dump_request already calls with_assurance() (landed 2026-09-03, commit 280b6c614, the same day as the third slice above), the real post-execution caller this row previously said didn't exist yet; DumpResult.resolved_execution_context is fully built, assurance-enriched, and even carries a per-side compile_contexts entry when a header-AST parse ran -- but nothing downstream of execute_dump_request (no CLI, nothing outside the function itself) reads it back, so it remains a genuinely unread, not unbuilt, object. A fifth slice (2026-09-04) closed one more concrete site on the compare path: resolve_compare_request was carrying no compile_contexts at all (evaluation_config/compile_contexts were both named as deferred in the third slice's own note) even though each side's resolved CompileContext was reachable a few lines away -- switching _resolve_side/_resolve_old_side/_resolve_new_side from resolve_side_snapshot to _resolve_side_snapshot_impl recovers SideResolution.effective_compile_context per side, threaded through the identical safety gate the dump path already applied (lifted out of execute_dump_request's own inline conditional into one shared predicate so the two paths share one decision instead of two hand-copies -- Codex review, PR #1037, six rounds, the dump-path original). A sixth slice (2026-09-04, Codex review, fresh evidence on the same PR) relocated that shared predicate a second time, to workflows.artifact.compile_context_gate.side_effective_compile_context (a new leaf module, not workflows.artifact.execute as first landed): the initial placement grew service_compare_pipeline.py/workflows/artifact/execute.py past the 800-line production cap, and a bare SideResolution type import would have pulled the helper into the large, already-allowlisted workflows.artifact.execute -> service -> ... import cycle as a genuinely new member -- so the function now takes the bare CompileContext | None it actually reads off SideResolution.effective_compile_context instead of the whole object, needing no import from execute.py at all. evaluation_config remains unresolved on both paths (still None on every real ResolvedComparePair/ResolvedDumpRequest), and every other independently-re-derived value this sub-phase names (policy/pack resolution) is still read the old way — this slice closes two concrete sites (compile-context threading, plus the dedup itself), not the sub-phase |
| 5B — Fact semantic consumption | in progress | Phase 5's "no fact reaches CONSUMED" gap |
For each fact family, an explicit FactStatus → detector-meaning table (e.g. FAILED means "incomplete evidence," not "confirmed absent") instead of the uniform legacy-default collapse — inventory every present-or-default unwrap first (resolved_fact_value call sites and local equivalents like diff_param_qualifiers._fact_bool/diff_cxx_rules._fact_str_list/compare.surface_graph.fact_list), not only the shared primitive's own callers; first vertical cohort on the five fields with an existing fabricated-finding history (RecordType.bases/virtual_bases/vtable/vptr_offset_bits, Param.is_va_list), since those are where the behavior change is easiest to justify and test. First PR landed (see the note below the table): the shared compare_facts/FactComparison primitive plus two of the five fields' primary finding-emitting call sites (bases/virtual_bases in diff_types._diff_type_bases, is_va_list in diff_param_qualifiers.param_va_list_changes). Second PR landed (see the note below the table): every remaining reader of bases/virtual_bases/is_va_list audited and closed — one genuine pairwise fabrication risk (diff_cxx_rules._transitive_bases, feeding virtual_method_addition) gated the same way, the rest (single-snapshot reachability/classification aids) documented as already safe. vtable/vptr_offset_bits remain on the old collapse, deliberately (their own dedicated, higher-scrutiny slice — see below), so this sub-phase's own gate ("every detector... for at least one full fact family") is not yet closed. Third PR landed (see the note below the table): the dedicated vtable/vptr_offset_bits slice — diff_vtable_layout._is_polymorphic/diff_layout._check_vptr_introduced now read FactStatus directly (additive, per-record); the TYPE_VTABLE_CHANGED cluster (diff_types_vtable.py) itself was re-audited and found unsafe to convert without a diff_cxx_rules.virtual_method_addition-side fix blocked by an import-cycle constraint, so the removal gate remains open for that one cluster specifically. Fourth through seventh PRs landed (see the note below the table): closed the case-(a) field inventory's audit status entirely -- is_const/is_volatile/is_mutable/is_restrict/access gated directly; default and the five deprecated/is_scoped surfaces investigated, found not safely convertible without a legacy-hybrid load-path fix (a real end-to-end test regression, not a hypothetical), and left on their existing fact_provenance mechanism with the specific finding recorded. Eighth PR landed (see the note below the table): audited the entire remaining case-(b) field inventory (nineteen per-declaration fields plus six detector-unconsumed snapshot-level ones) and found it already safe -- zero findings, no code changes -- closing this sub-phase's audit scope for every model field carrying a Fact[T] sibling as of this session; the sub-phase's own remaining work was the vtable/TYPE_VTABLE_CHANGED cluster (vptr_offset_bits is fully gated — see the corrected note further down). Track 4's 5B final closure (2026-09-04, see the note below the table) closed that last cluster as a formal, investigated decline, after a three-round investigation: round 1 declined, round 2 landed a FactStatus decline once Codex review found it closed a real, reachable PDB-driven fabrication, and round 3 reverted round 2 after it silently regressed a real detection scenario (a hand-constructed/typed-API RecordType omitting vtable= -- indistinguishable via FactStatus alone from PDB's own non-evidence -- means non-polymorphic, not "unknown"). vtable_transition_is_evidenced's heuristic is unchanged from before this closure; the PDB fabrication remains a real, open, explicitly-documented gap. This closes 5B's removal gate for every named field family; the sub-phase's remaining open scope is the seven fact_provenance-gated fields from the fourth-through-seventh PRs, not a further audit sweep |
| 6B — SemanticIR checker cutover | in progress | The gap this review calls the single largest: SemanticIR computed and persisted but never read by the checker |
One read index over SemanticIR (entity(), occurrences(), functions(), records(), facts(), references()) with a legacy-flat-snapshot adapter producing the same read shape, migrated one detector family/cohort at a time, each cohort closing with an architecture-gate rule forbidding a direct legacy-collection read for that family |
| 7B — Boundary consumer migration | in progress | action/run.sh's raw-exit-code decoding (Phase 7's named scope boundary); the release fan-out's gate-pack-fold duplication (a distinct residual, not ADR-064's own GateOptions rewrite, which already landed 2026-09-02); the release fan-out's explicit rejection of a --pack-asserted contract.unresolved |
Action reads run_outcome/exit from the machine report instead of re-deriving a verdict from the raw process exit code and stderr text; a real per-pair executor for depth/suppression/policy/compile-context/--contract mode-and-domain already exists (service.run_compare) and ADR-064's GateOptions already resolves the release fan-out's severity/exit-code-scheme gate config exactly once (confirmed by the 2026-09-03 investigation below) — what remains is narrower still, since T6 landed 2026-09-05: apply_release_gate_pack no longer mirrors pack_application.apply_to_compare_config — both call one shared policy/gate_pack_fold.fold_gate_pack_severity, leaving only the two callers' different fold targets (raw strings vs. a resolved SeverityConfig) for the duplication-and-convergence-assessment plan's own P0 EffectiveGate/EffectiveEvaluationConfig target; and resolve_release_pack_application unconditionally rejects contract.unresolved for a release comparison — not for lack of a per-library PersistedContractContext (service.run_compare already creates one, record_release_resolved_config already merges into it after every pair), but as the rejection's own deliberate choice, unverified whether still necessary — no landed fix or confirmed-necessary rationale yet |
| 8B — Multi-artifact canonical storage | in progress | Phase 8's "one legacy blob per section, single-artifact only" residual | Typed DTOs for the remaining sections beyond semantic_ir; multi-artifact ProjectSnapshot packages; baseline-set/BundleFacts folded into sections instead of staying separate document shapes |
2B's bare-name-collision narrowing landed (2026-09-03). The gap
compare/opaque_types.py's own docstring named as still-open since the
opaque-type suppression migration: two unrelated types sharing a bare leaf
spelling in different scopes (ns1::Handle opaque, ns2::Handle a
different, visible declaration) could both be suppressed through the
spelling tier, since the pre-existing design deliberately fell back to
spelling on any stable-tier miss rather than trusting a miss as proof of
non-opacity.
Why a miss couldn't simply be trusted before this slice. Trusting it
unconditionally would have traded a real bug for a worse one: a mixed
header-AST/DWARF comparison, or one side loaded from an archived baseline
predating RecordType.entity_id population, would leave one side's stable
tier incomplete — a real, still-opaque declaration failing to resolve an
identity on just one side reads identically to "not the opaque one" under
an unconditional-trust rule, silently dropping a genuine suppression and
reporting a purely private layout change as breaking (a live false-positive
risk against this repo's FP-rate gate).
The fix: gate narrowing on a completeness signal, not on producer
identity — three review rounds, each tightening the predicate a weaker
one had gotten wrong. The final shape: OpaqueTypeIndex carries
stable_by_local, the stable ids resolved among just the declaration(s)
sharing each bare spelling, and intersect()'s complete is true only
when, for every spelling both sides agree is opaque, the two sides' own
stable_by_local sets for it are exactly equal and non-empty.
Two weaker predicates were tried and each fell to a real counter-example
before landing on this one. First: "every raw declaration independently
resolved some stable id" (per side, ANDed) — falsified by two producers
resolving different ids for the same declaration (e.g. disagreeing on
whether an enclosing scope segment is a namespace or a record), which
this per-side-only check cannot see at all. Second, once fixed to compare
the two sides' id sets per spelling: "the two sides' sets intersect" —
falsified by a genuine collision itself, where two distinct
declarations share one spelling and only one of them actually paired;
a shared id from the paired declaration keeps the intersection non-empty
while the other silently disagrees. Exact equality is what closes that
gap: it requires every id either side resolved for a spelling to have a
match on the other side, not merely that some id does. Equal-and-empty
had to be excluded too (frozenset() == frozenset() is True, but two
sides that both resolved nothing for a spelling are not in agreement —
counting them as paired let strict=True reject a change on a
contentless miss instead of falling through to the spelling tier, caught
by an existing behavior-preservation test regressing).
_downgrade_opaque_type_changes reads complete straight off the
already-intersected index: opaque.contains(c, ..., strict=opaque.complete).
strict=True only changes what happens on a stable-tier miss (never a
hit, and never when the change carries no resolvable identity at all) —
so this is additive, both-or-neither-gated capability layered on the
existing permissive default, the identical discipline
compare/typedefs.py's and compare/constants.py's own fidelity gates
already established for the SemanticIR cutover.
Investigation finding worth recording: the completeness precondition
already holds far more often than the original "needs its own slice"
caution assumed. Checking each current producer found RecordType.entity_id
populated unconditionally by castxml, clang, DWARF, and PDB (all via
entity_id_for_type), and BTF/CTF inherits it for free by routing through
dwarf_snapshot.py's shared builder (to_dwarf_metadata()). The realistic
incompleteness case today is narrower than "some current producer never
resolves one" — it is producer version/schema skew: an archived baseline
.abi.json from before this population existed, or a genuinely mixed
header-AST/DWARF comparison. OpaqueTypeIndex.complete degrades safely to
the pre-narrowing permissive behavior in exactly that case, so this
finding motivated implementing the gate rather than skipping it — it did
not remove the need for one.
What remains open, explicitly. A change that carries no resolvable
entity_id at all still falls straight through to the spelling tier,
collision and all — tests/test_opaque_identity_tiers.py's
TestKnownGapStaysDocumented pins this narrower residual case, not the
general collision (now closed by TestBareNameCollisionNarrowing).
Verification. tests/test_opaque_identity_tiers.py's new
TestBareNameCollisionNarrowing class: the collision-closing case, the
completeness-decline case (proving the gate actually degrades rather than
silently narrowing anyway), and a hit-is-unaffected case. Full fast unit
lane green; ruff check/mypy abicheck/ clean; check_fp_rate.py 0 FP /
0 FN, both deltas 0; check_tier_accuracy.py OK (top-tier correct,
under-call monotonic).
2B's entity: alias promotion: investigated, declined rather than
deferred (2026-09-03). The narrowing slice above closed a demonstrated
bug (a real false suppression, pinned as a test before the fix and closed
after it). This is the other half of 2B's stated remaining scope, and it
does not have one: finding_identity.resolve_change_identity already
folds Change.entity_id in as an entity: alias, but never promotes it
to primary_id/tier -- what report_canonical_finding_id (the
cross-backend suppression-key function) actually hashes. Promoting it
would mean trusting a stable EntityId ahead of (or instead of) the
existing tier a finding already gets, which changes the hash for any
already-stored finding_id: suppression rule matching that finding.
Checked, rather than assumed, what promotion would actually buy for every
family that currently reaches Change.entity_id at all: typedefs and
constants already resolve IDENTITY_TIER_NORMALIZED off their own alias/
qualified-name spelling, which both header-AST backends already agree on
-- the one real cross-backend risk in that data, a value spelling
difference ("char const*" vs "char const *"), is exactly what
canonicalize_values=True already normalizes, independently of
entity_id. Functions/variables reach IDENTITY_TIER_CANONICAL off their
mangled name whenever one exists, which is strictly more precise than an
entity-derived id could be. No currently-identifiable finding needs the
promotion to resolve correctly or consistently across backends.
That makes this the identical shape of decision extract/
semantic_normalizer.py's constants slice already declined for a different
field (a value-spelling canonicalizer with no observed cross-backend
divergence to fix) -- AGENTS.md's own "a canonicalizer with no known
target divergence to fix is a heuristic in search of a bug, not a fix for
one" applies here just as directly, now to an identity-tier promotion
rather than a spelling canonicalizer. Declined on that basis: not "blocked
on sign-off" (the stability precondition -- StableEntityId/
entity_id_is_cross_snapshot_stable, ADR-063 Phase 2's own designated
"entity: promotion gate" -- is already built and tested), but "no case
found that would justify the compatibility cost yet." Revisit only if a
real cross-backend or cross-detector identity collision surfaces that
narrowing/canonicalization alone can't close, with the suppression-file
compatibility trade-off put to a maintainer explicitly at that point, the
same bar the opaque-type narrowing slice above did not have to clear
because it introduced no compatibility change at all.
5B's first PR landed (2026-09-03). abicheck.compare.fact_comparison.
compare_facts is the shared primitive (moved there from model/fact.py
where it first landed — Codex review: deciding "does this differ" is a
compare/-owned question per model/AGENTS.md's own scoped contract, not
model/'s). It
classifies an (old_fact, new_fact) pair into FactComparability.COMPARABLE
/INCOMPLETE/UNSUPPORTED/NOT_APPLICABLE per the review's proposed
FactStatus → detector-meaning table, and returns the resolved
old_value/new_value only when COMPARABLE. Two of the five named
fields' primary finding-emitting call sites were migrated onto it:
diff_types._diff_type_bases (bases_fact/virtual_bases_fact — the two
comparisons are gated independently, so a comparable virtual_bases pair
with an incomplete bases pair still reports a plain hierarchy change,
just without the finer became/lost-virtual classification) and
diff_param_qualifiers.param_va_list_changes (per-parameter
is_va_list_fact — the old _fact_bool present-or-False collapse this
function used is removed; param_restrict_changes is unaffected, since
is_restrict has no per-parameter evidence gap on either producer). Each
migration is behavior-preserving for a pair with fully complete evidence on
both sides (PRESENT, confirmed-empty/confirmed-False included — see
tests/test_diff_types_bases_fact_status.py's and
tests/test_clang_param_va_list.py::TestParamVaListFactStatusGating's own
"both sides confirmed" controls) and changes behavior for a pair with
incomplete evidence on either side, where a finding used to be fabricated
from the gap and is now withheld instead. bases/virtual_bases
additionally decline on a PARTIAL fact (Codex review, PR #1033: a base
absent from a partially-covered list may simply live in the uncovered
part, so treating the covered portion as the complete set is the identical
fabrication risk as an incomplete fact); is_va_list — a single-parameter
bool, not a list with an unobserved remainder — still compares normally
under PARTIAL. Both checked against the FP-rate and per-tier-accuracy
gates (scripts/check_fp_rate.py/scripts/check_tier_accuracy.py), both
clean.
Deliberately not attempted in this PR — vtable/vptr_offset_bits
(diff_types_vtable.py/diff_layout.py/diff_vtable_layout.py): those
three modules already implement extensive, individually-reasoned
evidence-gap guards (_vtable_transition_is_evidenced and siblings) built
before Fact[T] existed, inferring "evidence missing" indirectly from
size_bits/virtual-signature heuristics rather than reading FactStatus
directly, with a documented history of multiple prior false-positive/
false-negative incidents per guard (see each function's own docstring).
Replacing that heuristic layer with a direct FactStatus read is very
likely a real improvement — FactStatus is strictly better evidence than
inferring a capture gap from size_bits/signature-set heuristics — but it
needs its own dedicated slice with equal scrutiny (each existing guard's
own Codex-review history re-verified against the new behavior), not a
drive-by change bundled into this PR's already-broader scope. Also
unconverted: every other reader of the two fields this PR did migrate
(diff_cxx_rules.py's _fact_str_list, diff_stdlib_impl.py,
diff_time64.py, idioms.py, surface_graph.py,
buildsource/header_graph.py, diff_cpp_patterns.py) — these are mostly
classification/matching aids rather than finding-emitting detectors in
their own right, so they carry a smaller fabricated-finding risk than the
two migrated call sites, but 5B's own removal gate ("every detector...
for at least one full fact family") is not closed until they are covered
too. Recorded here rather than silently left implicit, per this repo's own
"say so explicitly and record the gap" convention (AGENTS.md
"Decision-making principles").
5B's second PR landed (2026-09-03) — the "lower-risk batch" from the note
above. Every remaining reader of bases/virtual_bases/is_va_list
named above was audited individually (is_va_list has no other reader at
all: diff_param_qualifiers.param_va_list_changes/compare.va_list_diff
is its only call site). The audit split into two kinds:
- One genuine old/new pairwise fabrication risk:
diff_cxx_rules. _transitive_bases, called fromvirtual_method_additionto tell a genuinely new virtual slot apart from a compatible override of an inherited one. An incompletebases_fact/virtual_bases_factanywhere along the transitive walk used to read as "no bases here", which could make a real override look like a new slot and fire a spuriousVIRTUAL_METHOD_ADDED._transitive_basesnow returns whether every visited record's evidence was confirmedPRESENT(aPARTIALfact is treated as incomplete too, the same disciplinediff_basesalready applies to this identical field pair — an unlisted base could simply live in the uncovered remainder);virtual_method_additiondeclines to emit when either side's walk was incomplete, rather than fabricating. Behavior is unchanged whenever both sides' evidence is complete — every real producer (dwarf_snapshot.pyincluded) already statesbases/virtual_basesexplicitly, even empty, for everyRecordTypeit emits; only test fixtures that never set the field at all (reading asNOT_COLLECTED, not confirmed-empty) needed updating to match that real producer shape._owner_descends_from(diff_cxx_rules.py, feedingvtable_slot_is_override_reuse) also calls_transitive_basesbut is itself part of the vtable evidence-gap cluster below — it takes the new return value's set component only, unchanged behavior, left for that slice. Verified against the FP-rate gate (scripts/check_fp_rate.py) and the per-tier-accuracy gate (scripts/check_tier_accuracy.py), both clean, plus the full unit suite. - Five single-snapshot reachability/classification aids with no old/new
pair to gate at all:
diff_stdlib_impl._public_by_value_type_closure(feeds an OR of both sides — an evidence gap only narrows one side's closure, the other side's independent closure or a later re-run still catches it),diff_time64._fold_record_tokens(feeds a union of both sides' token sets — identical shape),idioms._collect_base_targets(single-snapshot ADR-027 anti-pattern scan — a gap only shrinksbase_targets, missing aPOLYMORPHIC_TYPE_NON_VIRTUAL_DTORrather than fabricating one),buildsource/header_graph._flat_structural_type_edges(one snapshot's own graph, and this package's own governing rule already caps every finding it can feed atAPI_BREAK_KINDS/RISK_KINDSwithout an artifact diff also proving the break), andcompare.surface_graph. fact_list(already explicitly justified before this PR — an evidence gap only omits a graph edge this phase's own D5 amendment already documents as non-authoritative for any decision today). None of the five needed a behavior change — each already fails toward under- detection, never fabrication — so each now carries an explicit ADR-063 Phase 5B docstring note recording why, closing the audit gap without a speculative rewrite of code that was already safe.diff_cpp_patterns. _is_empty_record'svtable_factread is the one item from the original list that turned out to belong to the vtable cluster, not this batch — see below.
Still open: vtable/vptr_offset_bits themselves (diff_types_vtable.
py/diff_layout.py/diff_vtable_layout.py's own guard cluster), plus
every downstream reader of those two fields specifically —
diff_cxx_rules.virtual_method_addition's own old_vtable != new_vtable
comparison, idioms.py's three vtable reads (_recognise_factory,
_has_virtual_destructor, _detect_non_virtual_dtor), and
diff_cpp_patterns._is_empty_record. This PR's own audit reconfirms the
prior PR's reasoning for leaving them alone: _vtable_transition_is_
evidenced and its siblings are exactly the kind of individually-reasoned,
multi-round-reviewed evidence-gap heuristic that needs its own dedicated
slice with equal scrutiny, not a change folded into an otherwise
lower-risk batch.
5B's third PR landed (2026-09-03) — the dedicated, higher-scrutiny vtable/vptr_offset_bits slice this note called for. Re-verified each of this cluster's existing guards against its own Codex-review history (per each function's own docstring, cross-referenced above) before touching anything, per the plan's own instruction that this needs "equal scrutiny," not a drive-by. The audit found the cluster's three modules split cleanly into two kinds:
-
Two self-contained per-record guards, safely convertible.
diff_vtable_layout._is_polymorphicnever had any evidence-gap gating at all — a record whose ownvtable_factwasNOT_COLLECTED(e.g. a persisted, pre-v21 direct-clang snapshot's blanket-empty vtable, perAbiSnapshot.clang_vtable_facts_reliable's own docstring) read as confirmed non-polymorphic, unlikediff_types_vtable/diff_layout's own vtable reads, which already gate on the whole-snapshotvtable_facts_reliableflag._is_polymorphicnow reads the record's ownvtable_fact.statusdirectly and degrades to its own pre-existingNone(indeterminate) return instead ofFalsewhen that fact wasn't actually collected and no other evidence (virtual_bases, or a positive result from the transitive base walk) settles the question — additive, since a real positive signal from any source still short-circuits before this check is reached.diff_layout._check_vptr_introducedgained a parallel, additive per-record check on its old-sidevtable_fact/vptr_offset_bits_fact: the existingvtable_facts_reliableparameter is a whole-snapshot flag and stays exactly as it was (removing it would have silently broken every existing test that constructs a legacy-shaped fixture by hand, without also hand-settingFact.not_collected()on each record — the two mechanisms are equivalent only via the realstorage.fact_backfillload path, not via directRecordType(...)construction); the new per-record check sits beside it and can only decline more often, catching a per-record gap the whole-snapshot flag cannot see (a mixed-producer/hybrid dump, or a future producer that leaves one record's layout facts uncollected). Neither change alters behavior for a confirmed-empty (Fact.present([])/Fact.present(None)) record — every existing test, including the hand-constructedvtable_facts_reliable=Falsefixtures, stayed green unmodified; new tests pin the new behavior (tests/test_g23_vtable_b2.py:: TestReconstructionFactStatus,tests/test_diff_layout.py's three newtest_vptr_*_fact_uncollected*/*_confirmed_non_polymorphiccases). -
The
TYPE_VTABLE_CHANGEDcluster itself (_vtable_transition_is_evidenced/_vtable_transition_rests_on_ unresolved_evidence), deliberately left unconverted — a genuinely new finding, not a restatement of the first PR's deferral. Tracing the cluster's own false positive (identical headers, no DWARF vtable capture on one side because a class's virtuals live in a translation unit only the other side's debug info covers) againstFactStatusdirectly showed the two are answering different questions: DWARF's own extraction reportsFact.present([])— genuinelyPRESENT, notNOT_COLLECTED— for that exact scenario, since from DWARF's own local, per-TU perspective it really did capture everything it saw. A directvtable_fact.statusread cannot see per-TU coverage loss at all, so it would not replace this cluster's heuristic (the class's-own-virtual- functions / size-delta fallback still does the real work there) — it would only add a decline for the disjoint, genuinely-uncollected case. Tracing that addition through the call graph found it unsafe:diff_ cxx_rules.virtual_method_additiondefers to this cluster ("TYPE_ VTABLE_CHANGEDcovers this case") specifically in the one-side-uncollected/other-side-populated shape, relying on today's heuristic — not aFactStatusread — to still find real evidence there (the class's own virtual functions differ) and fire. An unconditional decline onNOT_COLLECTEDwould silently desynchronize the two detectors, and fixing it properly (makingvirtual_method_additionconsult this cluster's own evidenced/not-evidenced verdict before deferring, rather than re-deriving a coarser answer fromold_vtable != new_vtable) needs an importdiff_types_vtable.py's own no-import-cycle leaf-module constraint does not allow without further restructuring (diff_cxx_rules.pyalready suppliesdiff_types_vtable.pywithvtable_slot_is_override_reuse; the reverse import would cycle). Both modules' own docstrings now record this finding in place, rather than only in this plan document, per this repo's own "say so explicitly and record the gap" convention.idioms.py's three vtable reads anddiff_cpp_patterns._is_empty_record— all single-snapshot classification aids with no old/new pair to gate, the same shape the second 5B PR already closed for its own five aids — were audited the same way (no behavior change: an uncollectedvtable_factthere can only under-detect, never fabricate).idioms.py's three call sites gained the identical docstring-note treatment the second PR used;diff_cpp_patterns.pyitself carries no equivalent note in-code, since that file's ownarchitecture/debt.yamlno-growth baseline (CLAUDE.md's own "Files that are large" section) had zero headroom for even a four-line docstring addition and this file's finding does not warrant raising it -- the audit conclusion is recorded here instead.
Verified against the fast unit suite (unchanged pass count beyond the new
tests), mypy abicheck/ (0 errors), and ruff check/ruff format --check
on every touched file.
Still open, narrower than before: the TYPE_VTABLE_CHANGED cluster's
own two guards (_vtable_transition_is_evidenced/_vtable_transition_
rests_on_unresolved_evidence) remain on the pre-Fact[T] heuristic for the
reasons traced above — this is not the same gap the first 5B PR recorded
(that gap was "not yet attempted"; this one is "attempted, found to need a
virtual_method_addition-side fix an import-cycle constraint blocks
without further restructuring"). 5B's own removal gate ("every detector...
for at least one full fact family," now read as "vtable, all
five fields gated" — vptr_offset_bits is fully gated elsewhere and does
not share this cluster's own gap, see the corrected note further down)
is therefore still open for this one cluster specifically, even though
every other reader of both fields (the two sibling vtable detectors, and
every single-snapshot classification aid) is now gated or explicitly
audited-safe. Closing it for
real needs either restructuring diff_cxx_rules.py/diff_types_vtable.py's
own dependency direction so virtual_method_addition can consult this
cluster's real verdict, or an equivalent shared primitive both sides can
depend on without a cycle — a real design question, not a two-line patch.
5B's fourth through seventh PRs (2026-09-03) closed the remaining
case-(a) field inventory's fixable half. storage/fact_backfill.py's
apply_legacy_fact_backfill rule table is the authoritative, exhaustive
list of every case-(a) field ADR-063 Phase 5 (the fact-registry phase)
ever converted — fifteen rules across RecordType.vtable/
vptr_offset_bits, Param.is_va_list/is_restrict, TypeField.is_const/
is_volatile/is_mutable/default/deprecated, Function.deprecated,
Variable.deprecated/access, RecordType.deprecated, EnumType.
deprecated/is_scoped. Three more of these had the identical
previously-unguarded shape the vtable/vptr_offset_bits slice closed —
a whole-snapshot reliability flag applied once at detector registration,
then a bare-value comparison per declaration, with no per-declaration
FactStatus check at all:
diff_types_field_facts._check_field_qualifier_pair—TypeField. is_const/is_volatile/is_mutable, each gated independently throughcompare_factsalongside the existingheader_cv_facts_reliableflag.diff_param_qualifiers.param_restrict_changes—Param.is_restrict, mirroringcompare.va_list_diff.diff_va_list_params's already-migrated treatment of the siblingis_va_listfield exactly.diff_symbols_variables.var_access_changes—Variable.access, gated alongsidecastxml_var_access_facts_reliable.
Each landed as its own commit with dedicated tests, verified against mypy/ruff/architecture/ai-readiness (0 errors), the FP-rate and tier-accuracy gates (unchanged), and the full pre-existing test suite for its own area (778/234/543 tests respectively, all passing unchanged) — confirming no real producer today hits the gap either fix closes, matching the vtable slice's own disclosed caveat.
The remaining six case-(a) fields — TypeField.default/deprecated,
Function.deprecated, Variable.deprecated, RecordType.deprecated,
EnumType.deprecated/is_scoped — were investigated and found NOT safely
convertible this way, a distinct conclusion from "not yet attempted."
These five surfaces (six rows; deprecated repeats per declaration kind)
share one detector-side mechanism, fact_provenance.py
(fact_known_qualified/both_known_backed_fact_qualified, itself
predating Fact[T] — G28 Phase 3), not the collapsed-default pattern the
other conversions replaced. Session investigation traced every producer
path this session could directly verify — both header backends' own
explicit construction, DWARF's field omission, and dumper_hybrid.
_backfill_*_facts's replace_with_fact_sync calls — and found the two
mechanisms agree on all of them, which briefly looked like a green light.
A real attempted conversion regressed a genuine end-to-end test
(tests/test_dumper_hybrid.py::TestNamespaceQualifiedMerging::
test_legacy_bare_keyed_hybrid_baseline_still_detects_transition): a
--ast-frontend hybrid snapshot persisted before this fact family's own
schema version carries no per-declaration deprecated_fact key in its
JSON at all, and the legacy-load correction deliberately does NOT force
such a snapshot's reconstructed deprecated_fact to NOT_COLLECTED (the
guarding clang_deprecation_facts_reliable flag reads True for a hybrid
producer specifically — an ordinary, fresh hybrid dump's own construction
already states this fact explicitly per declaration, so the flag has no
reason to distrust it). The legacy JSON format, unlike a fresh in-memory
construction, always serializes some value for deprecated (there is no
"omitted" concept once a dict round-trips through the constructor), so
reconstructing from it unconditionally backfills to Fact.present(value)
— including for a declaration neither backend actually confirmed on that
old snapshot. AbiSnapshot.fact_provenance (the separate, G28-Phase-3
per-declaration dict) is exactly what resolves that one ambiguity for this
one legacy-hybrid shape, and a direct Fact[T] status read cannot recover
it. TypeField.default was never attempted at all for a second,
independent reason: its existing gate (fact_same_producer_qualified)
answers a question Fact[T]'s status genuinely cannot — whether the two
backends' value representations are comparable at all (castxml's
verbatim source expression vs. clang's structural fingerprint), not merely
whether either collected something.
Closing the deprecated/is_scoped half properly needs either teaching
the legacy-load path to also consult fact_provenance when backfilling
deprecated_fact for a pre-qualification-fix hybrid snapshot specifically,
or accepting that this fact family's evidence-gap gating permanently stays
on the provenance-string mechanism (in which case a Fact[T] migration
would just be a same-behavior refactor, not a fix, and isn't worth the
risk). Recorded in diff_types_field_facts._diff_field_deprecated's own
docstring as well as here, per this repo's own "say so explicitly and
record the gap" convention. This closes the case-(a) inventory's audit
status entirely: storage/fact_backfill.py's fifteen-rule table is not
the whole case-(a) inventory (bases/virtual_bases are case-(a) too but
carry no rule there -- "no independent reliability signal" per that
module's own docstring -- and were gated in earlier 5B PRs alongside
is_va_list). Of the fifteen ruled fields specifically: seven are now
gated on a direct FactStatus read (vptr_offset_bits, is_va_list,
plus this session's is_const/is_volatile/is_mutable/is_restrict/
access — vptr_offset_bits's only detector consumer,
diff_layout._check_vptr_introduced's own direct-status pre-check, is
fully gated; see the corrected note further down, which this sentence
previously contradicted, Codex review, fresh evidence),
one (vtable) is partially gated with its own documented
residual TYPE_VTABLE_CHANGED cluster, and seven (default plus the five
deprecated surfaces plus is_scoped) remain on the pre-Fact[T]
fact_provenance mechanism with a substantiated, tested reason recorded
for each rather than left silently unaudited.
5B's eighth PR (2026-09-03) audited the entire remaining case-(b) field
inventory and found it already safe — no code changes. Every model field
carrying a Fact[T] sibling that isn't one of the fifteen case-(a) rows
above is case-(b): per each field's own docstring, its bare None/[]
resting value is already unambiguous ("not captured," with no separate
"confirmed absent" state to conflate it with — unlike the case-(a) fields,
where the same resting value legitimately means two different things).
Two parallel Explore audits swept every detector-side reader (diff_*
.py/compare/*.py/idioms.py/checker*.py — not the extractors, not
model/storage) of all nineteen remaining per-declaration fields:
Function.is_explicit/is_hidden_friend/hidden_friend_owner/
source_header/is_variadic/exception_spec/is_override/
contract_attributes/is_compiler_generated/elf_binding,
RecordType.is_final/is_abstract/data_size_bits/is_standard_layout/
is_trivially_copyable/qualified_name/source_header,
Variable.source_header/alignment_bits/elf_binding. A direct check
covered the remaining six, snapshot-level (not per-declaration) fields --
ElfMetadata.dynamic_flags/has_init/has_fini, Mach-O rpaths, PE
delay_imports, AbiSnapshot.ast_resolved_standard -- confirming zero
detector-side readers exist for any of them at all (only extractor/
storage/serialization code touches them), so there is nothing to audit on
this axis yet. EnumType.qualified_name/source_header were confirmed
safe by direct inspection (structurally identical to the audited
RecordType/Function siblings — the same generic type_map_key/
_add_header_declares machinery, not a field-specific path).
Every pairwise finding-emitting detector already declines correctly on a
None/None pair (mostly via diff_helpers.bool_transition(...,
skip_none=True) or an explicit is None or ... is None: return/return
[]/continue guard, or — for is_standard_layout/is_trivially_copyable
— an is True/is False comparison that silently excludes None without
needing an explicit guard at all, the same discipline diff_layout.py's
own module docstring already documents). Every single-snapshot
classification/matching/identity/graph-building aid treats None/empty
as "no evidence to add" (an or name fallback, an OR-term in an
evidence-presence check, or metadata passed straight through onto an
already-decided finding — e.g. Change.symbol_binding, consumed only by
suppression's optional binding: selector, where a missing value simply
fails to match rather than being read as a confirmed answer) — never as a
positive, fabricatable answer. Function.is_compiler_generated has no
detector-side reader at all (extractor-side only, in
buildsource/source_extractors/base.py), so it is currently dead on this
axis rather than unsafe.
This closes ADR-063 Phase 5B's audit scope for every model field
carrying a Fact[T] sibling as of this session: the fifteen case-(a)
rows (seven fixed, one partial with a documented residual cluster, seven
investigated and left on fact_provenance with a tested reason each) plus
all nineteen audited case-(b) fields (zero findings) plus the six
detector-unconsumed snapshot-level fields (nothing to audit). 5B's own
removal gate ("every detector... for at least one full fact family") is
satisfied for bases/virtual_bases/is_va_list/is_const/
is_volatile/is_mutable/is_restrict/access/vptr_offset_bits (fully
gated — vptr_offset_bits_fact's only detector consumer is diff_layout.
_check_vptr_introduced's own direct-status pre-check, above) and
documented-open for vtable's own residual TYPE_VTABLE_CHANGED cluster
(diff_types_vtable._vtable_transition_is_evidenced/
_vtable_transition_rests_on_unresolved_evidence still read vtable_fact
via the collapsed resolved_fact_value, not a direct status branch — see
the third PR's own account above; this cluster deliberately never consults
vptr_offset_bits_fact at all — see that module's own "NOT consulted
here" comment, so vptr_offset_bits carries no residual gap through this
cluster) and the seven fact_provenance-gated fields — the sub-phase's
remaining work is that one cluster (vtable only), not a further audit
sweep.
Track 4 — 5B final closure (2026-09-04): the TYPE_VTABLE_CHANGED
cluster's own residual gap, resolved as a formal, investigated decline.
Full three-round account (declined → landed → reverted) lives once,
canonically, in diff_types_vtable.py's own module docstring ("Track 4,
5B final closure" section) — not repeated here. Outcome only:
vtable_transition_is_evidenced's heuristic is unchanged from before this
closure (confirmed identical to its pre-closure state via git diff). A
real, reachable fabrication risk was found along the way — a PDB-derived
side's vtable_fact is unconditionally NOT_COLLECTED on every record,
independent of AbiSnapshot.clang_vtable_facts_reliable, and a
cross-backend comparison against one can fabricate a TYPE_VTABLE_CHANGED
finding from an unrelated size delta — and a fix for it was landed, then
reverted after it regressed a real, previously-passing scenario
(tests/test_abicc_scenario_parity.py::
TestLeafClassVirtualMethodAdditions::test_virtual_added_to_leaf_class):
vtable is a public, positional RecordType field, and omitting it at
construction — not just PDB's own extractor never setting it — resolves
to the identical NOT_COLLECTED status, which is exactly how this
codebase's own hand-constructed test fixtures (and any external typed-API
caller) spell "no vtable" for an ordinary non-polymorphic class.
This closes 5B's own removal gate for the vtable field family as a
formal, investigated decline — the same disposition 2B's entity: alias
promotion and 6B's own undone cohort items received — not left
ambiguous between "landed" and "declined." The PDB fabrication remains
real, reachable, and open; closing it for real needs a snapshot/producer-
level signal analogous to AbiSnapshot.clang_vtable_facts_reliable, not a
per-record FactStatus branch. The DWARF per-TU ambiguity is a separate,
still-unresolved gap needing evidence this predicate's inputs do not carry
today — see the canonical docstring for both.
Known gap surfaced by review (Codex security review, this PR, not
closed): a "decline rather than fabricate" evidence gate can be an
under-detection lever, not just a fabrication guard, for a detector that is
the sole signal for its ABI-break class. virtual_method_addition's own
docstring already states it is the only signal for its specific blind spot
(a DWARF/symbol-only snapshot whose vtable array cannot show the growth);
declining when bases/virtual_bases evidence is incomplete therefore
doesn't just miss one classification among several redundant ones — for a
hand-crafted or legacy-schema snapshot with the right shape (vtable_fact
already trivially spoofable pre-existing this PR, plus a bases/
virtual_bases_fact at NOT_COLLECTED/FAILED/PARTIAL), it can turn a
genuinely BREAKING VIRTUAL_METHOD_ADDED into a silent, --require-
complete-analysis-invisible FUNC_ADDED/COMPATIBLE. This is not unique
to this PR's diff: the identical shape already exists at the first 5B PR's
own migrated call sites (diff_bases, diff_va_list_params) — any
detector using compare_facts's INCOMPLETE branch to decline has the
same property whenever it is (or becomes) a sole signal for its class. A
real fix threads a detector's own decline into analysis_assurance.py's
rollup (a genuine new signal that module doesn't compute today — it is
explicitly "a rollup, not a new probe" over data the pipeline already
computes) so --require-complete-analysis can see it, or gates the
specific sole-signal detectors more conservatively than the general
compare_facts pattern. Neither is attempted here: it is a cross-cutting
design question for the whole "decline rather than fabricate" philosophy
this sub-phase established, not a two-line patch scoped to one call site,
and reversing just this PR's own decline would only reopen the fabrication
bug it fixes without closing the class (an attacker would target
diff_bases/diff_va_list_params instead). Recorded here per this
repo's own "say so explicitly and record the gap" convention rather than
left implicit or rushed.
7B's first PR landed (2026-09-03). Scoped narrowly, after inspection
showed most of action/run.sh's raw-exit-code dispatch is not actually
closable the way it first appears: the Click-usage-error-vs-real-verdict
disambiguation (_is_cli_error's stderr grep) has no run_outcome to read
in the first place (a genuine CLI usage error produces no report at all),
and scan's dedicated evidence-contract/budget-overflow/not-comparable exit
codes (7/5/6) are deliberately exit-code-only by ADR-037 D5's own design —
three prior stderr/marker-file signals for that exact axis were each shown
forgeable, which is why it has its own reserved exit code instead of a
report-based signal at all. Neither is a gap this sub-phase can close by
reading run_outcome harder. What is closable, and landed this PR:
_report_compat_verdict/_severity_gate_exit — the two shared functions
compare and scan alike consult to resolve the compatibility-axis verdict
and the severity-policy gate tier — now read the report's own
run_outcome.compatibility/run_outcome.gate (ADR-063 Phase 7 / D6) first,
falling back to the pre-existing verdict/severity.exit_code field (and,
for a non-JSON report, the rendered text) only when no run_outcome block
is present. Behavior-preserving wherever both sources exist (run_outcome
.compatibility is literally result.verdict, same as the legacy verdict
field; run_outcome.gate is the identical fold severity.exit_code
already encoded, just typed), so no existing tests/test_action_run_sh_*.py
fixture needed updating — tests/test_action_run_sh_run_outcome.py adds the
new coverage, giving each report a run_outcome value that deliberately
disagrees with its paired legacy field to prove the script actually reads
run_outcome rather than falling through to the unchanged path by
coincidence. The release fan-out's own "independent pair-semantics
reimplementation" half of this sub-phase's stated scope is untouched by
this PR — real, separately-scoped follow-up work (cli_compare_release.py/
cli_compare_release_helpers.py/cli_compare_release_matrix.py together
are ~2800 lines with their own extensive review history; a shared
pair-operation executor there needs its own dedicated slice, not a
drive-by extension of this one).
7B's release-fan-out investigation landed (2026-09-03) — a dedicated
slice, per the note above, that reads the ~2800 lines closely rather than
attempting a rewrite. The line count itself was stale (cli_compare_release
.py/cli_compare_release_helpers.py/cli_compare_release_matrix.py are
now 838/1277/718 lines — a fourth, sibling file the plan text above didn't
name, cli_compare_release_pairwise.py (743 lines, the actual per-pair
comparison engine: _run_compare_pair/_compare_one_library/the
sequential/parallel dispatch), was split out afterward and carries none of
the duplication this slice found). Reading every concrete axis the plan
text names against cli_compare_release_pairwise.py found most of them
already unified, through mechanisms that landed independently of this
sub-phase:
- Depth, suppression, pack policy/namespace overrides, compile context,
contract evaluation — every real per-library pair already routes
through
service.run_compare(_run_compare_pair's own docstring: "the single Tier-2 chokepoint... this is what keepscompare-releaseandcompareon one classification path"), which folds all five identically for a release pair and a single-paircompareinvocation of the same library. Nothing to unify here — it already is, with one qualification found on review, itself corrected on a second review pass (Codex, this PR, two rounds):cli_compare_receipt.resolve_release_pack_applicationunconditionally rejects a--packassigningcontract.unresolvedfor a release comparison (a hardPackManifestError, not a silent no-op) — but not because the release fan-out lacks a per-libraryPersistedContractContextfor that field's consumer (contract_coverage_exit._accepts_unresolved) to read, the first-round finding's premise. It does build and merge one:_run_compare_paircallsrecord_release_resolved_configafter every pair (cli_compare_release_pairwise.py), which folds the pack's resolved config intoresult.contract_contextviacontract_context. with_resolved_configwhenever--contractproduced one — and_compare_one_libraryreadscoverage_exit_floor(result)off exactly that context afterward. So the context-plumbing this field's consumer needs already exists; the rejection itself is the only blocker, not missing infrastructure. Whether that rejection remains necessary now that the plumbing exists, or is a stale defensive guard a future slice could safely lift, is unverified — not attempted here, and not to be assumed either way without re-readingresolve_release_pack_application's own reasoning against this evidence. So a release comparison cannot usecontract.unresolvedtoday, where a single-paircomparecan — a real, live boundary-consumer gap (the explicit rejection), not merely a duplication risk like the gate-pack fold below, and not a missing-context gap either. - Severity/exit-code-scheme gate resolution — ADR-064's own
GateOptions/resolve_release_gate_options(abicheck/policy/release_gate_options.py) already resolves this exactly once within the release fan-out (replacing three independent re-derivations that existed before that ADR — see that module's own docstring), closing the specific drift risk PR B's own note there had flagged as unsafe to fix reactively. What remains is narrower than "not unified":release_gate_options.py's own docstring states plainly thatapply_release_gate_pack"mirrors [pack_application. apply_to_compare_config's] logic... instead" of calling it, because the release fan-out has noResolvedCompareConfig-shaped object of its own to fold packs onto — distinct from ADR-064's ownGateOptionsrewrite (already landed 2026-09-02, per that ADR's own "Landed" note, and what closed the three-independent-re-derivations drift risk above): a full fold unification is the duplication-and-convergence-assessment plan's own P0EffectiveGate/EffectiveEvaluationConfigtarget (docs/contribute/plans/duplication-and-convergence-assessment.md), not this sub-phase's to redo reactively.tests/test_release_gate_pack_fold_parity.py(this PR) closes the actual residual risk instead: a Hypothesis property test pinning the two independently-reasoned fold implementations to agree on outcome for every generated pack contribution, so a change to one that silently drifts from the other fails there first — the "primitive-level property test" AGENTS.md calls for when a real unification isn't the safe move for one PR to make reactively. - Report composition — the release fan-out's per-library
--output-dirJSON write (_compare_one_library's ownto_json(...)call) has no shared code with single-paircompare's_render_compare_reportpipeline, but this is not a maintenance gap:compare-releaseexposes neither--use-casesnor a suppression-audit equivalent at all (confirmed by grep — no such flag exists anywhere in the three/four release modules), so there is nothing for a shared executor to fold in that the simpler directto_jsoncall is missing. Unifying this would mean adding--use-cases/suppression-audit support tocompare-releasefirst — a real, separate feature request, not a 7B-scoped refactor. - The build-config matrix pseudo-pair (
_collect_matrix_result,cli_compare_release_matrix.py) independently reloads suppression/policy and re-folds packs (_load_suppression_and_policy/policy_file_with_packs— the same shared helpers a real pair's resolution already goes through, called at a third site, not a second, divergent implementation). This is structurally required, not duplicated-by-oversight: the matrix findings are release-global (extra_changesfed toservice.compare_snapshotsagainst a pair of empty snapshots), resolved once for the whole release regardless of how many real library pairs exist — routing it through the real per-pair_run_compare_pair/_compare_one_librarypath makes no sense for a pseudo-pair with no actual old/new binaries. Its own docstring already states this reasoning; left as-is.
Net effect: the "shared pair-operation executor" the plan text's one-line
goal names is, on inspection, substantially already real — just distributed
across several independently-landed mechanisms rather than one function —
with two concrete, still-open gaps, not one (a second found on review,
Codex, this PR — see the amended first bullet above): the
apply_to_compare_config/apply_release_gate_pack duplication, which has
its own named, deferred follow-up (the duplication-and-convergence-assessment
plan's own P0 EffectiveGate/EffectiveEvaluationConfig target — not
ADR-064's GateOptions rewrite, which already landed 2026-09-02) and is now
guarded by a parity test rather than left to silent drift; and the release
fan-out's explicit rejection of a --pack-asserted contract.unresolved
(resolve_release_pack_application rejects it outright — not for a missing
PersistedContractContext, which service.run_compare already creates
per library and record_release_resolved_config already merges into, per
a second review round's own correction to the first round's premise
above, but as the rejection's own deliberate choice), which has no landed
fix or confirmed-necessary rationale
yet and remains a live boundary-consumer gap for a future slice to
investigate and close. Recorded here per this
repo's own "say so explicitly and record the gap" convention rather than
closing 7B's own status row on an incomplete account, or forcing a premature
rewrite of carefully-built, recently-landed code (GateOptions) to
manufacture a bigger diff.
The exit-code-scheme half of that one gap landed (2026-09-04). Reading
both fold functions side by side found their scheme resolution --
specifically the "which way does the scheme move" precedence with the real
regression history (Codex review, PR #1032) apply_to_compare_config's own
docstring already documents -- was expressible as one pure function over
primitive values (a pack's own explicit scheme, the resolver's already-
decided fallback scheme, whether the pack supplied a severity level, and the
pre-pack scheme), independent of which of the two different pre-resolution
shapes (ResolvedCompareConfig vs. six raw strings) the caller holds.
policy.release_gate_options.resolve_gate_pack_exit_code_scheme is that
function; both apply_release_gate_pack (same module) and
apply_to_compare_config (pack_application.py) now call it instead of
each re-deriving the identical three-tier precedence. The dependency points
the direction ADR-061 requires: pack_application.py is a legacy_root_
module, unrestricted by the architecture gate's layer-import check (that
check only fires for modules under a migrated package directory), so it
importing from policy.release_gate_options is not the "migrated layer
imports unclassified module" violation the reverse direction would be
(policy.release_gate_options's own module docstring, still accurate,
explains why it cannot import pack_application.py). tests/
test_release_gate_pack_fold_parity.py is kept as the black-box guard over
the whole fold, updated to record that its scheme half is now one function
rather than two.
What remains genuinely open: the severity-level application itself --
dataclasses.replace on an already-resolved SeverityConfig on one side,
six independent raw-string overrides on the other -- is the same update
expressed against two different pre-resolution data shapes, not duplicated
logic in the way the scheme resolution was. Collapsing that difference for
real needs the release fan-out to hold a ResolvedCompareConfig-shaped
object of its own to fold onto -- the duplication-and-convergence-assessment
plan's own P0 EffectiveGate/EffectiveEvaluationConfig target (not
ADR-064's own GateOptions rewrite, which already landed 2026-09-02 and is
what this fold already builds on), not attempted here.
The other 7B gap -- the contract.unresolved rejection -- closed
(2026-09-04, Track 2). Re-reading resolve_release_pack_application's own
reasoning against the plumbing this section already confirmed (service.
run_compare/record_release_resolved_config genuinely build and merge a
per-library PersistedContractContext) found no hazard the plumbing doesn't
already cover: contract.unresolved=warn never touches evidence, labels, or
GateDecision for any library (ADR-049 Section 6.2), only the orthogonal
contract-coverage exit floor, so applying it release-wide carries the same
"one pack value, threaded identically into every library's own resolved
config" shape policy.overrides/surface.internal_namespaces already use
without issue -- there was no genuine per-library-vs-release-wide semantics
mismatch to guard against, just an unconfirmed hazard the earlier round
correctly declined to assume away without checking. The unconditional
rejection is gone; resolve_release_pack_application now applies the same
contract_evaluation-gated CONTRACT_EVALUATION_ONLY_FIELDS check the
single-pair path already uses (rejecting the field only when this
release invocation has no --contract for anything to read it), using the
release's own real contract_evaluation value instead of the
hard-coded True that reopened the gap the manual check then had to close.
tests/test_pack_application.py::TestReleasePackApplication::
test_contract_unresolved_pack_now_applies_to_a_release_comparison proves it
end to end -- a release comparison whose --contract public domain cannot
prove its own evidence complete for hand-built snapshots (no real header-AST
provenance), which exits 1 from the coverage floor alone without the pack
and 0 with it, while contract_coverage_failures stays populated in both
(the ledger stays unsuppressible; only the exit's willingness to fail on it
changes). The sibling test
test_contract_unresolved_pack_rejected_without_contract_on_a_release
pins that a pack asserting the field is still rejected as decorative when
this release invocation passes no --contract at all. run_outcome's
ledger row above is updated to reflect this as closed rather than open.
8B's first PR landed (2026-09-03). storage.types_section_codec
.TypesSection is the "types" D8 legacy section's own typed DTO, wired
through storage.dto.types_to_dto/types_from_dto in place of the generic
legacy_section_to_dto pass-through envelope every other legacy section
still uses. "types" was chosen as the first section to promote precisely
because it is the simplest possible case: storage.legacy_sections
._SECTION_FIELDS["types"] is exactly one field, and that field is required
whenever the section is present at all (_REQUIRED_SECTION_FIELDS["types"]
names it) — so there is no schema-version-dependent field-presence sparsity
for a typed wrapper to get wrong, unlike the provenance/debug/binary
sections a first investigation also considered, each of which carries a
genuinely sparse, schema-version-dependent set of optional fields that a
naive typed-dataclass-with-defaults conversion risks silently reconstructing
incorrectly against storage.import_v1.export_legacy_snapshot's
byte-for-byte round-trip contract (verified against every real
tests/fixtures/schema/v*.json fixture, test_a_real_schema_fixture_round_
trips) — a real design problem those sections still need before they can
follow, recorded here rather than attempted without it. The on-disk section
payload shape is unchanged ({"types": [...]}, identical to what
legacy_section_to_dto already stored); what changed is that the DTO layer
no longer treats a "types" section as an arbitrary opaque blob
indistinguishable from any other legacy section. The remaining seven legacy
sections (binary/declarations/layout/debug/build/graph/
provenance), multi-artifact ProjectSnapshot packages, and folding
baseline-set/BundleFacts into sections all remain open, per this
sub-phase's own stated scope.
8B's second PR landed (2026-09-03). storage.graph_section_codec
.GraphSection promotes the "graph" D8 legacy section the same way, chosen
next by the identical heuristic: _SECTION_FIELDS["graph"] is exactly one
field (surface_graph), and split_legacy_document only ever creates a
"graph" section when that field is present (a section with none of its
fields present is omitted entirely, and "graph" has no other field it
could carry) — so a present "graph" section's payload has exactly one
possible shape, the same guarantee "types" relies on. This holds even
though _REQUIRED_SECTION_FIELDS["graph"] is empty: that table is derived
from schema v1 alone, and surface_graph postdates v1 (ADR-063 Phase 3 D5,
schema v29), so its absence there reflects the field's introduction date,
not genuine per-section optionality. Wired through storage.dto
.graph_to_dto/graph_from_dto and storage.import_v1, mirroring
types_to_dto/types_from_dto's wiring exactly; on-disk shape unchanged.
The remaining six legacy sections (binary/declarations/layout/debug/
build/provenance) all carry the genuinely sparse, schema-version-
dependent optional-field profile the first PR's note above describes, and
still need that design problem solved before they can follow the same
pattern.
8B's third PR landed (2026-09-03), closing "typed DTOs for the remaining
sections beyond semantic_ir" in full. storage.sparse_section_codec
solves the sparsity design problem the first two PRs left open: each of the
six remaining sections' fields is split by what storage.legacy_sections
._REQUIRED_SECTION_FIELDS already, independently proves about it (derived
empirically from tests/fixtures/schema/v1.json, the format's own oldest
fixture — any key that table lists is safe to require unconditionally in
any document this build can still read). A field in that set becomes a
real, always-present, named dataclass attribute (BinarySection.elf/.pe/
.macho, ProvenanceSection.library/.version, DebugSection.dwarf/
.dwarf_advanced, DeclarationsSection.functions/.variables/.enums/
.typedefs/.sycl); every other field in the section's own
_SECTION_FIELDS allowlist lives in extra, a validated (allowlist-
checked, canonically-frozen), never-defaulted pass-through mapping, so a
document missing an optional key entirely still round-trips with that key
simply absent from extra — never fabricated, never dropped. layout and
build have no field _REQUIRED_SECTION_FIELDS proves present since v1 at
all (both sections postdate schema v1 in full), so their entire content
stays in extra; still real, dedicated, versioned classes rather than the
generic pass-through, per the module's own "typed only in the sense the
required half of D8 actually supports today" scoping note. Wired through
storage.dto's six new *_to_dto/*_from_dto pairs and one new registry
(storage.import_v1._LEGACY_SECTION_CODECS) that replaced the by-then
three-branch if/elif chain in both import_legacy_snapshot and
export_legacy_snapshot — a lookup table scales to a ninth section kind as
a one-line addition instead of a second edit in two functions each time.
Every one of D8's eight named legacy section kinds now has its own DTO;
legacy_section_to_dto/legacy_section_from_dto remain defined as the
generic fallback a future, not-yet-specialized ninth section kind would
use, but are unreachable for any of today's eight. One real behavior
change, deliberately: import_legacy_snapshot now enforces a section's own
required fields structurally at import time too, not only at
export_legacy_snapshot's post-hoc missing_required_section_fields
check — a hand-built document missing e.g. provenance.version entirely
(never a real snapshot_to_dict() output, which always includes both
AbiSnapshot.library/.version since neither has a dataclass default) now
fails on import rather than passing silently through and only failing
later, if ever, on export.
8B's remaining two items investigated (2026-09-03): both explicitly
blocked, not merely unstarted. "Multi-artifact ProjectSnapshot
packages" and "baseline-set/BundleFacts folded into sections" are not a
one-line aspiration with no design behind them — storage-format-v2.md's
own A1.4/A1.5 sections carry a full, concrete design (a PackageManifest
.project_sections: Mapping[str, ObjectRef] field for cross-library
evidence stored once and shared by every ArtifactRef that needs it, a new
bundle_facts_store.py reader/writer reconstructing a BundleFacts-shaped
view so compare_bundle_from_facts's existing tests pass unmodified,
concrete file/test/acceptance-criteria lists). PackageManifest.
artifact_refs/.variant_refs are already generic tuples and
__post_init__ already validates an arbitrary-length collection
(storage/package.py) — the object model has supported this shape since
A1.1 landed. What is missing is a producer: every real writer today
(import_v1.import_legacy_snapshot, project_snapshot_legacy
.write_legacy_snapshot_package, sectioned_document's hard-coded single
artifact/variant id) is wired to exactly one artifact, and no test
round-trips a genuinely multi-artifact package successfully (only
duplicate-id/collision rejection tests construct more than one
ArtifactRef).
Why this isn't implemented here anyway. storage-format-v2.md's own
"Relationship to G38" section states the constraint directly: this plan's
Phase 1 and G38's own Phase 2 target the same eventual container, "They
are complementary and must not both grow a container format... Do not
implement a third persisted bundle shape." one-semantic-pipeline.md's
own Phase 8 text is equally explicit: "any remaining legacy baseline-set/
BundleFacts-only code path once the ProjectSnapshot import adapter
covers it — per ADR-062's own phasing, not accelerated here." This is a
recorded governance decision, not a scope judgment this session is free to
override — proceeding without the G38-side coordination the design itself
requires would risk building exactly the "third persisted bundle shape"
both texts name as the failure mode to avoid. Recorded here, per this
repo's own "say so explicitly and record the gap" convention, rather than
attempted against the plan's own stated blocker or left silently
unaddressed.
Recommended sequencing: 2B and 6B are
the highest-value pair, in that order — 2B closes the last identity-provider
gap 6B's own migration would otherwise trip on, and 6B is what actually
starts retiring the legacy AbiSnapshot.functions/types/… read path
rather than adding a further passenger beside it. The review's explicit
recommendation is not to keep widening SemanticIR's/FactRegistry's
producer coverage (more backends, more fields) until at least one full
vertical slice — request → SemanticIR → a FactStatus-aware detector →
RunOutcome → report, with the legacy read path for that one detector
family actually removed — exists as a proof of the pattern the remaining
phases would repeat. Widening producers without first proving a consumer
cutover increases the number of representations to keep in sync faster than
it retires any of them, which is precisely the failure mode ADR-063 exists
to stop.
A mechanical definition of done for the whole ADR, accepted as the
target — evaluated once 6B (or an equivalent) has landed at least one real
cohort, not before: the pipeline is complete only when (1) every front end builds one
request type; (2) every real execution resolves one execution context;
(3) every extractor's output reaches one canonical SemanticIR (Phase 6's
own type — no new type is proposed here; AbiSnapshot.semantic_ir remains
the persisted container); (4) the checker
reads no backend-specific or legacy AbiSnapshot collection
(.functions/.types/…) directly, only SemanticIR through the read
index Phase 6B defines;
(5) every detector consuming an availability-bearing fact explicitly
branches on its FactStatus; (6) stable-entity matching goes through
EntityId, with string identity as a named legacy fallback only;
(7) public surface has one authoritative query, whatever that query's own
final design turns out to be after Phase 3's amendment (see below);
(8) release/artifact-set fan-out calls the same pair operation single
compare does; (9) RunOutcome -> ExitDecision is the only path a
consumer follows to a numeric exit code, never the reverse (exit code back
to inferred semantics, except inside a documented legacy-report compat
adapter); (10) the Action and every renderer read decisions instead of
re-deriving them; (11) current writers emit canonical, typed section DTOs,
with legacy shapes confined to import adapters; (12) an architecture/AST
check finds no legacy reader of a migrated representation outside its
compatibility layer; (13) live, legacy-flat, sectioned, and ProjectSnapshot
inputs describing the same comparison agree on semantic digest, finding
identity, verdict, assurance, gate, and RunOutcome; (14) the checks behind
(12)/(13) run as required, not merely informative, CI gates — worth
flagging because PR #1011 disabled required-status-check enforcement and
post-merge verification on this repository, so today's AI-readiness/
architecture gates are informative only and do not themselves block a
regression from merging.
Phase 3 (D5) amendment — accepted: rather than a further attempt to make
compute_public_surface() read the mergeable evidence graph (three review
rounds already found that unsafe for the reasons ADR-063's own D5
"Amendment" note and docs/contribute/known-gaps.md record), formally split
the concept D5 conflated into two: a SemanticReferenceIndex
(deterministic, built from SemanticIR/snapshot declarations alone,
authoritative for the public-surface closure walk — what
referenced_identifiers_by_node() already computes today, just not yet
sourced from SemanticIR) and the existing evidence graph
(multi-producer, provenance/confidence-bearing, for explanation and L5
analysis, never for a decision with exactly one legitimate source). This
would formally close D5's own literal "traversal over one authoritative
graph" text with a decision that matches what already shipped, rather than
leaving that text describing a design three review rounds rejected.
Recorded here as accepted; ADR-063's own D5 text (this same PR) carries
the matching amendment note.
On this plan document's own size and structure: this document has grown
past fifteen thousand lines, carrying multi-round PR review history inline
with each phase's design. The review's suggestion — move each phase's own
review-round history to docs/contribute/plans/history/one-semantic-pipeline/
phase-N.md and keep this document to "where are we, what blocks the next
step, what's the next bounded PR" per phase, plus a small machine-readable
authority ledger (concepts: {primitive, producers, consumers, authority,
removal_gate} per concept, generated into a status table rather than
hand-copied) — is a reasonable direction for a future documentation-only PR,
but is not attempted in this update: it is a structural change to how this
plan is maintained, not a status correction, and risks losing citations
(commit hashes, PR numbers, specific Codex-round findings) that later
phases' own text still points back to. Recorded here as a named, deferred
proposal rather than silently acted on.
Phases¶
Phase 0 — Fact[T] in the domain layer (builds on ADR-062 Phase 0)¶
Goal. A detector cannot observe a field's value without first observing
its availability. None/[]/a boolean flag stop being overloaded to mean
both "confirmed absent" and "not collected."
Design. abicheck.storage.availability_status.FactStatus is the leaf
vocabulary this phase reuses — but ADR-061's dependency direction is
storage -> model, so Fact[T] (living in model/) may not import
from storage. This phase therefore relocates FactStatus/
Confidence/the status-order tuples from abicheck/storage/
availability_status.py into abicheck/model/availability.py (a leaf
module, no dependency on anything but the standard library, matching that
module's own existing "none of it needs to know what a stored record or a
ledger looks like" framing), and abicheck/storage/availability_status.py
becomes a re-export shim for one release so existing storage.* imports
keep working. FactAvailability (the ledger record) stays in storage/,
since it legitimately depends on model, not the other way around.
Add abicheck/model/fact.py: a generic Fact[T] with three fields,
not two — status: FactStatus, value: T | None, and diagnostics:
tuple[str, ...] = () — mirroring storage.FactAvailability's own
existing shape, which already separates its value-bearing fields from
diagnostics for exactly this reason (see that record's own field
comments). A first draft of this phase described Fact[T] as just
FactStatus plus the T payload and let Fact.failed(reason) store its
reason there — review correctly caught that this cannot typecheck
(reason is a diagnostic string, not a value of type T) and would
either violate the declared generic type or silently drop the diagnostic
FactAvailability's wire shape already preserves. With the third field:
Fact.failed(reason) is Fact(status=FAILED, value=None,
diagnostics=(reason,)); Fact.unsupported()/Fact.not_applicable()/
Fact.not_collected() take an optional *diagnostics the same way, for a
producer that wants to record why (e.g. which depth was requested)
without it becoming a smuggled value.
FactStatus has exactly six members (PRESENT, PARTIAL,
NOT_COLLECTED, UNSUPPORTED, FAILED, NOT_APPLICABLE — see that
module's own docstring) and deliberately has no seventh "confirmed
absent" member: per PRESENT's own documented meaning ("the producer
ran, covered the requested scope, and established the facts — including
establishing that a collection is legitimately empty"), a confirmed
absence is PRESENT carrying an empty/None value, not a distinct
status. Fact[T]'s constructors are therefore Fact.present(value)
(value may legitimately be None/[] — that is confirmed absence),
Fact.not_collected(), Fact.unsupported(), Fact.failed(reason),
Fact.not_applicable(), and Fact.partial(value). There is no
Fact.absent_confirmed() — a draft of this plan proposed one and it was
corrected during review for contradicting the vocabulary it claims to
reuse unchanged; a caller wanting to assert absence calls
Fact.present(None) (or Fact.present(())/Fact.present([]) for a
collection) explicitly, so the payload contract's only rule is that this
is the one legitimate way to spell "present, empty" — never a bare
sentinel construction readers could mistake for "not collected."
Fact.value_or(default) and Fact.is_present exist, but value_or is
not a detector-safe way to read one — a first draft of this phase
offered it as one of "the only two ways to read one without a full
match," which a reviewer correctly rejected: old.vtable.value_or([])
!= new.vtable.value_or([]) collapses NOT_COLLECTED/FAILED/
UNSUPPORTED back into the same default as a confirmed empty value,
reintroducing by a different spelling the exact ambiguity this phase
exists to make unrepresentable. value_or is reserved for non-semantic
presentation code (a report renderer choosing a display fallback, where
collapsing "not collected" and "confirmed absent" to the same rendered
text is an acceptable UI simplification, not a detection decision);
every detector reads a Fact[...]-typed field only by inspecting
.status/pattern-matching the full FactStatus space. The
check_ai_readiness.py rule in this phase's acceptance criteria enforces
this distinction, not merely "was there a bare attribute access" —
value_or called from diff_types.py/diff_layout.py or any other
detector module is flagged exactly the same as a bare attribute read; the
rule's allowed callers are presentation modules only, not "anywhere
outside model/fact.py" as this phase's first draft stated it.
Fact.__bool__ is explicitly defined to raise TypeError("Fact[T]
has no truth value — read .is_present or .value_or(...)") — plain
absence of __bool__ leaves ordinary Python object truthiness in effect
(every Fact[T] instance would be truthy regardless of status), so the
no-implicit-truthiness invariant needs the raise, not silence.
A second override was attempted for the same reason and reverted — a
first draft of this phase defined Fact.__eq__/__ne__ to raise the
identical TypeError, and a later review round correctly rejected it.
Fact[T] is itself a field on RecordType/Function/every other
fact-bearing dataclass, and a raising __eq__ on a field poisons the
containing dataclass's own generated __eq__ the instant comparison
reaches that field: two otherwise-identical RecordType instances (an
ordinary test assertion, a list/snapshot comparison, Phase 6's own
CanonicalEntity equality) would raise instead of comparing, which is a
far more disruptive failure than the narrow one this override was meant
to close. Fact.__eq__ stays the plain dataclass-generated structural
comparison (status/value/diagnostics together) — correct for a
containing object's own equality, and exactly what old.default !=
new.default gets instead of a raise. The actual guard against that
specific misuse (comparing two Fact[...] values directly inside
detector logic, instead of unwrapping first) is the same mechanism
already enforcing the .value_or() rule: the check_ai_readiness.py
static AST check, widened to also flag a bare Fact[...]-typed field on
either side of ==/!= inside a detector module — same file scope, same
enforcement layer, not a second runtime mechanism underneath the first.
Scope for this phase (deliberately narrow). Convert the fields
AGENTS.md's "Known gaps" names as actively causing fabricated
findings from absent evidence: RecordType.vtable/vptr_offset_bits
(the type_vtable_changed guard), RecordType.bases (the accepted-gap
type_base_changed entry — converting its representation first makes a
future evidence-based guard additive instead of another reinterpretation
of None), and Param.is_va_list (the reliability-flag entry) — plus
RecordType.virtual_bases, converted in this same Phase 0 PR alongside
bases rather than deferred (see the identical-producer/identical-
availability-conditions reasoning a few sections down), for five fields
total across four owning dataclasses. Every
other model field stays as-is in this phase — a blanket conversion is
Phase 5's job, after D7's registry exists to drive it mechanically.
Where the Fact[...] value actually comes from (both directions —
fresh extraction and loading a legacy persisted snapshot). This phase is
incomplete without both halves; a detector switched to read Fact[...]
with nothing populating it correctly would either suppress every existing
finding (if unpopulated defaults to not_collected()) or silently
recreate the exact ambiguity this phase exists to remove (if derived
naively from the existing raw value with no producer-aware distinction).
Concretely, this repository already has the mechanism this phase
generalizes, in the form of AbiSnapshot's per-field, per-producer
reliability flags (clang_vtable_facts_reliable, clang_va_list_facts_
reliable, and their siblings for other fields not converted in this
phase) — each already encodes, in careful hand-written prose, exactly the
producer/schema-version distinction Fact[...] generalizes into a typed
value instead of a side boolean:
- Fresh extraction (dumper_castxml.py, dumper_clang.py/
dumper_clang_vtable.py, dwarf_snapshot.py): each producer now
constructs the field's value directly as a Fact[...] at parse time —
Fact.present(vtable_list) when it actually reconstructed a vtable,
Fact.unsupported() for a producer that has never populated this fact
at all (castxml for is_va_list, per that field's own existing
docstring), Fact.not_collected() when the run's evidence depth never
reached that extractor. No snapshot-level reliability flag is needed for
a freshly-built snapshot, since the per-field Fact[...] states it
directly — this is the generalization's actual payoff, not an
afterthought.
"Constructs the field's value directly at parse time" is not quite
true for vptr_offset_bits on the DWARF backend, and a first draft of
this phase missed the gap. dwarf_snapshot.py runs a fixed-point
resolution pass after every RecordType already exists
(rec.vptr_offset_bits = resolved, at the sites resolving an inherited
vptr offset through a virtual-primary-base fallback — confirmed by
reading the real code, not assumed) — this has to run post-construction
because it needs cross-references between already-built records a
single object's own __post_init__ cannot see. A vptr_offset_bits_fact
constructed only inside __post_init__, at the point each RecordType
is first built, would freeze at that record's pre-resolution state —
typically Fact.not_collected(), since vptr_offset_bits is None is
exactly the condition that put the record on this pass's own worklist —
while the legacy vptr_offset_bits field goes on to hold the correctly
resolved value a moment later. A migrated detector reading the Fact
field would then see "not collected" for a record the legacy field
(and, for a caller that didn't migrate, every existing behavior)
correctly resolved — silently losing exactly the fact this conversion
exists to make visible, for the one DWARF-specific case that resolves
in two passes instead of one. Fixed by updating both representations at
each of these fixed-point call sites, not by deferring Fact
construction (which would mean every other, single-pass record
waiting on a cross-record pass that in practice never touches it):
rec.vptr_offset_bits_fact = Fact.present(resolved) alongside
rec.vptr_offset_bits = resolved wherever this pass resolves a value,
the same "both representations move together" discipline this phase's
legacy-field-resync fix already establishes for the explicit-constructor
direction, just applied to a producer-internal write instead of a
caller-supplied one.
- Loading a legacy, pre-Fact[...] persisted snapshot
(serialization.py): the existing reliability flag is read once, at
load time, to reconstruct the correct Fact[...] value for that
snapshot's schema version and producer — clang_vtable_facts_reliable ==
True backfills Fact.present(raw_vtable); == False backfills
Fact.not_collected() (never Fact.present([]) — the old field's
"blanket empty" value on an unreliable snapshot is not a confirmed
absence, exactly the "real but WRONG data" distinction that flag's own
docstring already draws). The reliability flags themselves become
write-only after this phase, but are not deleted at Phase 10 — an
earlier draft of this plan said they were, and that contradicts Phase
8's own commitment. ADR-062 Phase 1's v1-v25 import adapter is
explicitly a permanent capability (ProjectSnapshot must always be
able to import any snapshot version this project ever shipped, not only
versions newer than some cutoff), and a pre-Fact[...] snapshot's
reliability flags are the only evidence that lets the importer tell a
trustworthy empty value apart from one that was never collected — once
deleted, that information is gone from the input entirely, and no
later code can reconstruct it. So clang_vtable_facts_reliable/
clang_va_list_facts_reliable (and every sibling this pattern applies
to) stay in serialization.py's read path, and in the wire format, for
exactly as long as a pre-Fact[...] schema version remains importable
— which, per Phase 8's own commitment, is indefinitely. What does
go away is the domain-side boolean field (AbiSnapshot.clang_vtable_
facts_reliable as a live, queryable attribute on a freshly-built
snapshot) — nothing in the current codebase reads it once every
consumer reads Fact[...] instead, so the attribute itself is the one
piece of this that is genuinely removable, not the wire-level decode
logic that still has to run for a historical input.
RecordType.bases has no equivalent flag to read, and "every sibling
this pattern applies to" above does not actually include it — a first
draft of this phase implied it did, by backfilling bases the identical
conditional way vtable/is_va_list backfill from their own reliability
flags, and review correctly caught that bases has no such flag to
condition on at all (confirmed by grep: no *_bases_reliable-shaped
field exists anywhere in this codebase). This is not an omission this
phase introduces — AGENTS.md's own type_base_changed "Known gaps" entry
already documents, at length, that no independent evidence signal exists
for this field, and that the existing, live detector's accepted policy is
to always treat a captured bases list as real ("the alternative —
suppressing a real hierarchy change ... is strictly worse"). Backfilling
a legacy snapshot's raw bases to Fact.not_collected() — the False-
flag branch's behavior for vtable/is_va_list — would be a real,
new regression here specifically: it would silently suppress every
type_base_changed finding the existing, unconverted detector already
produces today against every pre-Fact[...] snapshot, for a field whose
status quo never suppressed on capture-gap grounds in the first place.
Backfilling unconditionally to Fact.present(raw_bases) is therefore the
only correct choice for this one field's legacy-loading path — not a
weaker substitute for the flag-conditioned mechanism, but the literal
zero-behavior-change preservation of what bases already does today,
known limitation included. This is deliberately asymmetric with
vtable/is_va_list's own legacy-loading bullet above: those two
fields have a real signal to condition on and use it; bases does not,
and pretending otherwise by reusing the same conditional shape would
fabricate a confidence neither the flag nor the field's own history
supports. Closing the underlying gap — giving bases a real reliability
signal, the way vtable already has one — is exactly the kind of
cross-cutting data-model work AGENTS.md's own entry already named as a
needed, not-yet-attempted follow-up; this phase converts the
representation, which is what makes that follow-up additive instead of
another reinterpretation of a raw None/empty value, and does not
attempt the evidence-signal design itself.
Writing a freshly-extracted snapshot back out needs its own fix, not
just a reader-side backfill. serialization.snapshot_to_dict() calls
asdict(snap) on the whole AbiSnapshot — dataclasses.asdict()
recurses into every nested dataclass field, including a Fact[...]
instance, which it flattens into {"status": FactStatus.PRESENT, "value":
..., "diagnostics": (...)} with the status key holding the raw
FactStatus enum member, not a JSON-safe value — json.dump() raises
on an unrecognized type. This is not a new problem Fact[...] invents:
snapshot_to_dict() already has to do exactly this conversion for
ElfMetadata's own enums today (its "Serialize ElfMetadata enums to
strings for JSON compatibility" post-asdict() pass, right below the
asdict() call itself) — this phase extends that same, already-
established pattern to every Fact[...]-typed field's status, writing
status.value (the plain string, e.g. "present") instead of the enum
member, with the reverse conversion added to serialization.
snapshot_from_dict()'s per-field loaders. This is a genuine new key in
the serialized document for every converted field, so it is a real schema
change: serialization.SCHEMA_VERSION is bumped by one, following the
identical precedent each of the clang_*_facts_reliable flags already
used when it was introduced (schema v21 for vtable reliability, v23 for
is_va_list, ...) — this is not a new kind of schema decision, it is the
same kind this codebase already makes routinely for exactly this class of
field addition.
Files. abicheck/model/availability.py (new — the relocated
FactStatus/Confidence/order-tuple vocabulary); abicheck/storage/
availability_status.py (trimmed to a re-export shim); abicheck/model/
fact.py (new — Fact[T]); abicheck/model/entities.py's RecordType
and abicheck/model/declarations.py's Param dataclasses — not
model/snapshot.py, which only imports both from their real owning
modules and defines neither; a first draft of this phase named the wrong
file, the same way Phase 5's Files section already correctly names the
two (new Fact[...]-typed fields alongside the existing
ones, old field deprecated-but-present for one release to keep
asdict-based external consumers working). The old field is not a
live @property deriving from the new one — dataclasses.asdict()
only serializes declared dataclass fields, never properties, so making
the old field a property would silently remove it from every
asdict-based consumer's output instead of keeping it populated, the
opposite of this compatibility goal. Instead, every producer derives the
old field from the new Fact[...] value at the same construction call
— vtable=fact.value_or([]) right next to vtable_fact=fact — so
there is exactly one write, not two independently-maintained ones that
could drift; the old field is never independently assigned raw producer
output again after this phase — with one named exception, below.
"Removed in Phase 5's registry-driven sweep" names the wrong phase, and
a review round correctly found nothing anywhere actually does this
removal as stated — Phase 5's own Scope section converts a different,
disjoint set of fields (RecordType.is_final, Function.
contract_attributes, Variable.alignment_bits, and siblings Phase 0
left alone) and never touches vtable/bases/vptr_offset_bits/
is_va_list at all, and Phase 10's checklist has no row for this phase
either — only for the narrower domain-side clang_*_facts_reliable
boolean attributes. Scheduled instead where it belongs: Phase 10's
checklist below gains its own row for this phase, removing the four
retained legacy attributes once the widened, repository-wide
legacy-attribute-read check this phase's own Acceptance criteria adds
(see below) reports zero remaining readers outside the compatibility
bridge's own __post_init__ and serialization — the same "accounting
pass, not new design" bar every other Phase 10 row already uses, and the
same one-release retention window this phase's own "kept... to keep
asdict-based external consumers working" commitment already implies
rather than leaving open-ended. dumper_castxml.py/
dumper_clang.py/dumper_clang_vtable.py/dwarf_snapshot.py (each
producer constructs the Fact[...] value directly, per the design above);
dumper_layout_backfill.py's _backfilled_record() — a post-parse
path, not a producer construction call, that dataclasses.replace()s an
already-built RecordType to overwrite vtable/vptr_offset_bits with
corroborating DWARF evidence after either header parser has already run.
The single-write rule above does not hold across this call: it must build
a new Fact[...] from the DWARF value first and derive the replaced
legacy fields from that Fact, the same order every producer uses, or the
backfilled RecordType ends up with a legacy field holding the
DWARF-corroborated value while its Fact[...] field still holds the
header parser's pre-backfill one — exactly the split-source drift this
phase exists to prevent, reached through a second call site rather than a
different producer.
A second named exception: RecordType/Param are public API
dataclasses — AGENTS.md's own convention on this file's public types
means an external Python-API caller can construct one directly
(RecordType(..., bases=["Base"]), Param(..., is_va_list=True)),
bypassing every producer call site named above entirely, and a first
draft of this phase didn't account for that. If the new Fact[...]
sibling field defaulted to Fact.not_collected() the way an ordinary
dataclass field default would, a direct caller supplying only the legacy
field gets a migrated detector reading "not collected" for a value the
caller explicitly gave it — silently discarding caller-supplied data this
phase must not break. Making the sibling field required instead breaks
every existing direct-construction call site outright, which is the
opposite failure. The fix: the new field's real default is None (not
Fact.not_collected() — a caller can still explicitly pass
Fact.not_collected() and have that honored, since None is reserved
purely as "nothing supplied," distinct from an explicit not-collected
claim), and __post_init__ backfills it from the legacy field when still
None — vtable_fact = Fact.present(self.vtable) if self.vtable_fact is
None else self.vtable_fact, mirroring the shape Param's own
__post_init__-based validation already uses elsewhere in this codebase
for exactly this kind of defaulting. Precedence is explicit and
one-directional for which value wins, but — a later review round caught
this too — "wins" has to mean the legacy field is resynchronized from it,
not merely that the Fact[...] field ends up correct while the legacy
field is left stale. RecordType(vtable=["old"], vtable_fact=Fact.
present(["new"])) is a real, constructible case (not a hypothetical): if
__post_init__ only ever reads the legacy field to backfill the Fact
and never writes the legacy field back, a migrated detector reading
vtable_fact sees ["new"] while dataclasses.asdict() and every
existing, unmigrated Python consumer reading rec.vtable directly still
sees ["old"] — two disagreeing representations on the same object,
which is the exact defect this whole phase exists to eliminate,
reintroduced by the one compatibility path meant to prevent it. The
actual rule, corrected: whichever value is authoritative for a given
construction — the explicit Fact[...] when supplied, the backfilled one
derived from the legacy field otherwise — is written to both fields
before __post_init__ returns, the same single-source-of-truth guarantee
every producer's own construction call already gives for free (Design
section, above) now given to a direct caller combining both forms too.
RecordType(vtable=["old"], vtable_fact=Fact.present(["new"])) ends
construction with self.vtable == ["new"], not ["old"] — the explicit
Fact[...] value also overwrites the legacy field, not only the other
way around.
This bridge as just described is still wrong for the common case, and a
later review round caught it: RecordType.bases/vtable already default
to [] and Param.is_va_list already defaults to False — identical to
an explicitly-supplied confirmed-empty value. vtable_fact = Fact.
present(self.vtable) if self.vtable_fact is None else self.vtable_fact
cannot tell "caller explicitly wrote bases=[]" apart from "caller wrote
RecordType(...) and never touched bases at all" — both leave
self.bases == [] by the time __post_init__ runs, and both would
backfill to Fact.present([]), falsely claiming "collected, confirmed
empty" for the ordinary case of a caller (most of this codebase's own
existing test fixtures, for a start) that never asserted anything about
the field at all. That is the identical unavailable-vs-empty collapse
Phase 0 exists to eliminate, reintroduced through the one compatibility
path meant to preserve callers, not create a new instance of the bug
for them. The fix needs an omission sentinel, not a truthiness check, and
the two field shapes (list-typed, bool-typed) need two different
mechanisms — a first draft of this fix proposed one uniform mechanism
for both, and it is unimplementable for the boolean field: Python has
exactly two bool instances, True and False; there is no third,
distinct bool-typed object a sentinel could be, so self.is_va_list is
_OMITTED_IS_VA_LIST can never be true for any caller-supplied value —
whichever of True/False the sentinel is defined to equal, that
identity check collides with a caller legitimately passing the same
value. For RecordType.bases/vtable (list-typed), the mechanism
this section previously described cannot actually be implemented as a
direct field default — a dataclass field may not take a mutable object (a
list, dict, or set instance) as its own direct default at all;
Python's dataclasses module raises ValueError: mutable default <class
'list'> for field ... is not allowed the moment the class body executes,
before any instance is ever constructed. A singleton list used as
bases' own direct default is exactly such an object, so bases:
list[str] = _OMITTED_BASES never reaches __post_init__ at all — the
class itself fails to define.
Two mechanisms were tried and rejected for both field shapes before
landing on the one that actually works, and the reasoning for rejecting
each is worth keeping, since a later round otherwise re-proposes one of
them. (1) A bare field(default_factory=...) look-alike for the list
case does not repair the identity check: a default_factory runs fresh on
every omitted construction, so each omitted instance gets its own,
distinct empty-list object — never the one singleton
self.bases is _OMITTED_BASES needs to match. (2) Widening the declared
type to bool | None/list[str] | None (an earlier revision of this
section) makes the field constructible, but a later review round
correctly flagged it as a real breaking change for this bridge's own
stated purpose — "every reader still sees a plain bool/list[str]"
is true only after __post_init__ runs, and AGENTS.md is explicit that
"changing [a public dataclass's] public surface is a breaking change to
the Python API — coordinate it": a type-checked external caller reading
Param.is_va_list/RecordType.bases now has to handle None at the
static-type level for a value that can never actually be None at
runtime, which is exactly the kind of "the representation disagrees with
itself" defect the Governing Invariant singles out, just relocated from
the dataclass body to its type annotation.
The mechanism that actually satisfies every constraint at once —
dataclass-constructible, identity-checkable, and never widens the
declared field type — wraps the existing private-sentinel idea in
typing.cast() at the point the sentinel is built, not at the point it is
compared. A module-level singleton of a dedicated, non-bool/non-list
marker class (_Omitted, one instance) is constructed once and then
cast() to the field's own real type — _OMITTED_IS_VA_LIST: bool =
cast(bool, _Omitted()), _OMITTED_BASES: list[str] = cast("list[str]",
_Omitted()) — which tells a type checker the sentinel is a bool/
list[str] (so the field's own declared type needs no union, no None,
no widening of any kind) while its actual runtime identity is a distinct,
non-bool/non-list object no caller-supplied value can ever equal by
identity. For the list-typed fields specifically, the mutable-default
ValueError is avoided the same way field(default_factory=list) already
avoids it today, just returning the existing singleton instead of a
fresh list each time — field(default_factory=lambda: _OMITTED_BASES) —
which is a legal default_factory (dataclasses only forbids a direct
mutable-typed default, not what a factory function returns) and, unlike
mechanism (1) above, returns the identical object on every omitted
construction, since the factory itself holds no state and always returns
the same module-level reference. __post_init__ checks self.bases is
_OMITTED_BASES/self.is_va_list is _OMITTED_IS_VA_LIST (a # type:
ignore[comparison-overlap] is expected and correct here, since the
comparison is exactly the one case cast() told the type checker could
never be true — confirmed against this repository's own mypy --strict,
which accepts the construction with zero errors and reports the field's
type as exactly bool/list[str], never a union), backfills
Fact.not_collected() for the true-omission case and Fact.present(...)
for an explicit value (including an explicit []/False, still
distinguishable from omission since it is not the sentinel by identity),
then normalizes the field to an ordinary False/[] before returning —
so after construction every reader, asdict()-based or otherwise, sees
exactly the type the field has always declared, with no accepted
trade-off left to state. Verified directly (not merely reasoned about,
given how many rounds this exact design question has already gone
through): a minimal repro of this construction passes mypy --strict
with zero errors, reports dataclasses.fields(...)'s type as the
unwidened annotation, and round-trips correctly through dataclasses.
asdict() for both the omitted and explicit-value cases, including the
explicit-empty-list case staying distinct from omission.
This choice has a real, named consequence for every existing
direct-construction call site in this codebase's own test suite, and a
later review round asked this plan to actually own that consequence
rather than only justify the choice that causes it. Once a migrated
detector reads .status and skips rather than reports when it sees
Fact.not_collected() (the whole point of the migration), a pre-existing
test fixture built as RecordType(name="Foo") with no bases/
bases_fact — because that field didn't exist before this phase, not
because the test intended "unknown evidence" — now feeds that detector a
not_collected signal it never meant to assert, and a test that expects
a TYPE_BASE_CHANGED/TYPE_VTABLE_CHANGED/PARAM_BECAME_VA_LIST/
PARAM_LOST_VA_LIST-family finding from such a fixture can start failing
the moment its detector migrates — not the moment this phase lands the
sentinel itself, which is a no-op until a detector actually reads
.status. That ordering is what keeps this self-auditing within this
codebase: each detector's own migration PR (already its own Files/Tests
entry elsewhere in this plan) runs the existing suite and any fixture
whose real intent was "confirmed empty/non-variadic" surfaces as a loud,
specific test failure at that PR, which is fixed by making the fixture
say what it actually means — RecordType(name="Foo", bases_fact=Fact.
present([])) — not by reverting the detector's new, correct behavior.
Each detector-migration task in this plan's own Files sections gains this
as an explicit sub-task: audit and update every fixture the newly-migrated
detector's tests touch, in the same PR, rather than leaving a fixed
fixture as an unplanned follow-up discovered by a later, unrelated CI run.
This is not self-auditing for an external Python-API caller who builds
an AbiSnapshot by direct construction outside this repo's own test
suite, though — that caller has no test of this codebase's own to fail,
and their comparison now silently reads "no evidence" where it used to
read "confirmed empty," a genuine behavioral change with nothing to force
them to notice it. That is disclosed, not silently shipped: this phase's
changelog fragment (this repo's own scriv create convention, already
required for any change touching abicheck/**/*.py) states the behavior
change explicitly — direct construction of RecordType/Param without
the new sibling Fact field now represents unrecorded evidence, not a
confirmed empty/non-variadic value, to any Fact-aware consumer; a caller
that wants the old, confirmed-empty semantics passes the sibling field
explicitly (bases_fact=Fact.present([])) — which is exactly the
compatibility bridge already documented above, just invoked deliberately
instead of relied on implicitly.
A third field shape needs a third mechanism, and a later review round
found it missing: RecordType.vptr_offset_bits is also converted by
this phase (named in the Scope section above) and is int | None,
already defaulting to None today — where None is already a real,
meaningful value ("no vptr observed"), not an unused slot the way bool's
two values are both already spoken for. Unlike is_va_list, None
cannot double as this field's omission marker — RecordType() (omitted)
and RecordType(vptr_offset_bits=None) (explicit: confirmed no vptr) must
backfill differently (Fact.not_collected() vs. Fact.present(None)),
but both already leave self.vptr_offset_bits is None with no way to
tell them apart, the identical ambiguity the bases/is_va_list fixes
above each close for their own field. int does not have bool's
two-instance problem, though — it has the opposite problem from list
this field shares the fix with: a fresh private sentinel object (not a
literal value, and not reusing None) works exactly like the list case,
since nothing short of that exact object will ever compare identical to
it. The field's actual dataclass default becomes the identical
cast()-sentinel construction bases/is_va_list already use above —
_OMITTED_VPTR_OFFSET_BITS = cast("int | None", _Omitted()), never
exported — rather than the literal None. This field's declared type
does not widen at all, because it has nothing to widen to — int | None
was already its type before this phase, for the field's own, legitimate,
pre-existing reason (a real "no vptr observed" value), not introduced by
this mechanism the way a bare bool/list[str] field would otherwise
have needed one; cast("int | None", ...) merely tells the type checker
the sentinel already belongs to the union that was always there.
__post_init__ checks self.vptr_offset_bits is
_OMITTED_VPTR_OFFSET_BITS (identity) to tell omission from an explicit,
confirmed-None, backfills Fact.not_collected() only for the
true-omission case and Fact.present(self.vptr_offset_bits) (None
included) for an explicit value, then normalizes the field to a real
int | None (None if it was the sentinel) before __post_init__
returns — the same post-condition the other two fields reach, by the
identical "a genuinely distinct object, made to type-check as the field's
own real type via cast(), can't collide with anything a caller passes"
mechanism, just applied to a field whose natural resting value happens to
coincide with Python's only singleton None the way bool's natural
resting values coincide with its only two.
All three mechanisms end at the
identical post-condition (the legacy field is a plain, fully-populated
value after __post_init__, the sentinel never leaks to a reader, and the
field's own declared type never widens) — they differ only in how the
sentinel is shaped to type-check as the field's own real type: cast()
to the field's already-Optional type for vptr_offset_bits (nothing to
widen, the union predates this phase), and cast() to the field's
otherwise-unwidened bool/list[str] type for is_va_list/bases/
vtable, with the list-typed fields additionally routed through a
default_factory that returns the one singleton rather than a fresh
object, since a direct mutable-typed default is rejected outright by
dataclasses regardless of what mechanism the value itself uses.
Continuing the Files list: serialization.py
(snapshot_to_dict()'s Fact[...]-status-to-string encoding, extending
its existing ElfMetadata-enum-encoding pattern; snapshot_from_dict()'s
matching decode; SCHEMA_VERSION bump; and the legacy-schema backfill
path, reading the existing reliability flags exactly once on load);
diff_layout.py/diff_types.py's vtable/base-list detectors, and every
other semantic reader of the three converted fields, not only the two
primary detectors, and not only the ones living under diff_*.py —
diff_param_qualifiers._diff_param_va_list
(p_old.is_va_list/p_new.is_va_list), diff_vtable_layout.
_is_polymorphic (rec.vtable/rec.virtual_bases), and diff_cxx_rules's
base-walk helpers (start.bases/rec.bases) all read the raw field
directly today and were missing from a first draft of this file list —
each retains the exact unavailable-vs-empty ambiguity this phase exists
to close until it is migrated too.
internal_leak.py::_enqueue_record_children() is a fourth such reader,
and a first draft of this paragraph's own closing sentence — "the AI-
readiness gate... checks every module under diff_*.py" — is exactly
why it was missed: internal_leak.py is not a diff_*.py module at
all, so a gate scoped to that glob would silently never see it.
_enqueue_record_children() walks rec.bases (and, on the following
line, rec.virtual_bases) directly to decide whether
an internal type is reachable from a public one through inheritance, for
INTERNAL_TYPE_LEAKS_VIA_PUBLIC_API; reading the unconverted raw fields
here has the identical failure mode every other unmigrated reader has —
if bases_fact is Fact.not_collected() but the legacy field was
normalized to [] for backward compatibility, this walk sees no bases at
all, can miss the one public-inheritance path that makes the leak real,
and silently suppresses or demotes the finding. Fixed two ways, not one:
internal_leak.py is added to this phase's migration file list alongside
the diff_*.py modules above, and the AI-readiness gate's module scope
widens from "every module under diff_*.py" to an explicit allowlist of
every module this phase identifies as a semantic reader of the three
converted fields — a glob that happens to match today's known readers is
not the same invariant as "every known reader is checked," and the next
non-diff_*.py reader this plan's own drafting process misses should fail
the gate, not silently bypass it the way this one did.
A further repo-wide check found the allowlist-by-hand-enumeration
itself is exactly the failure mode the previous paragraph just diagnosed,
reproduced across two more passes — and a review round correctly found
the prose narrating each pass separately had drifted into inconsistent
counts ("four more"/"five more" mixing module counts with call-site
counts) and inconsistent per-reader field lists (an earlier mention of
internal_leak.py/surface_graph.py named only bases, while both
actually read virtual_bases too, just on a separate line rather than
in one combined expression) and miscategorized two genuinely diff_*.py
modules as "outside" that scope. Replaced with one explicit table,
checked against the real source directly rather than re-described a
third time in prose:
| Reader | Function | Fields read | In diff_*.py? |
|---|---|---|---|
contract_evidence_collect.py |
build_type_graph() |
bases, virtual_bases |
no |
diff_time64.py |
_fold_record_tokens() |
bases, virtual_bases |
yes |
diff_stdlib_impl.py |
_public_by_value_type_closure() |
bases, virtual_bases |
yes |
surface_graph.py |
_build_type_refs() |
bases, virtual_bases (separate lines) |
no |
internal_leak.py |
_enqueue_record_children() |
bases, virtual_bases (separate lines) |
no |
export_surface.py |
line 1167's unresolved-type scan | bases, virtual_bases |
no |
surface.py |
two public-closure walks (lines 632, 782) | bases, virtual_bases |
no |
type_reachability.py |
stdlib-reference text scan (lines 873-874) | bases, virtual_bases |
no |
dumper_scoping.py |
dependency-retention text scan (lines 342-343) | bases, virtual_bases |
no |
Nine distinct modules, ten call sites (surface.py contributes two).
diff_time64.py/diff_stdlib_impl.py are diff_*.py modules — the
original AI-readiness gate (scoped to that glob) already covers their
call sites; they needed no new gate-scope widening, only the same
Fact-aware migration every other row gets. The seven remaining rows are
outside diff_*.py and are exactly why the gate itself stops being a
glob at all (below). Every row's consequence follows one of two shapes:
a missing inheritance edge can make export_surface.py's exclusion_
is_provable gate wrongly treat a type as out-of-contract instead of
failing closed (contract_evidence_collect.py/export_surface.py), or
it can silently suppress/omit a derived finding or shrink a resolved
surface the same way the vtable/base-list detectors already can
(every other row, including surface.py's own reachability closure
feeding compute_public_surface() directly). All nine modules are added
to this phase's migration file list, and — since a fifth, then a tenth,
hand-missed call site is now a demonstrated, repeating pattern, not a
one-off — the
AI-readiness check itself stops being a hand-maintained module allowlist:
it becomes a real static scan for any direct attribute access naming
bases/vtable/is_va_list/virtual_bases (the fourth field sharing
this exact ambiguity, scheduled for conversion in this same phase below
— not itself one of the three fields the Design section above already
covers) on a value whose declared type resolves to RecordType/Param,
repository-wide, with all nine named modules above
becoming the check's own initial known-failures baseline (mirroring
check_ai_readiness.py's existing MYPY_ERROR_BASELINE/
LARGE_FILE_ALLOWLIST pattern: a reviewed, shrinking allowlist of
known violations, not a permanent exemption list a new violation can
quietly join). RecordType.virtual_bases was left out of this phase's
Design section with no concrete phase scheduled to pick it up, and a
review round correctly rejected that as a dead end: Phase 5's own
eligibility mechanism is keyed to an existing backend-reliability flag,
and this field has none, so leaving it for Phase 5 would mean it never
gets picked up by anything. Checking the real producers settles where
it actually belongs: dumper_clang.py's _parse_bases() and
dwarf_snapshot.py's base-classification walk both populate bases/
virtual_bases from the same call, in the same pass, under the same
availability conditions — there is no separate collection step and no
separate reliability signal for one versus the other, so treating them as
two different phases' work would be artificial. virtual_bases is
therefore converted in this same Phase 0 PR, using the identical
sentinel-based Fact[list[str]] mechanism already fully specified for
bases above (same list-typed omission-sentinel construction, same
__post_init__ bridge, same legacy-schema backfill reading whatever
signal bases_fact itself backfills from, since the two are produced
together) — not a new mechanism, the same one with a second field name.
Every one of the nine readers named above reads bases/
virtual_bases together in one loop body (contract_evidence_collect.py,
diff_time64.py, diff_stdlib_impl.py, diff_cxx_rules's base-walk
helpers, internal_leak.py, export_surface.py, both surface.py
walks, type_reachability.py, dumper_scoping.py) and migrates both
fields in the same pass, not
bases now and virtual_bases in a still-later phase — the provenance
propagation this finding asked for is exactly "every semantic reader of
virtual_bases is already on this phase's own migration list for
bases," not a second, separate reader audit.
Tests. A direct test on Fact[T]'s actual comparison contract, added
before any detector migration depends on it: if fact: raises
TypeError; fact_a == fact_b/fact_a != fact_b do not raise —
they perform the plain structural comparison (status/value/
diagnostics), the same as any other dataclass — pinning the reverted
design directly (confirmed to fail against a version of Fact that
defines a raising __eq__/__ne__, which breaks this test as well as
every containing RecordType/Function comparison). A second test
confirms the containing-object guarantee this reversion exists to
protect: two separately-constructed but field-identical RecordType
instances (including equal Fact[...] sibling fields) compare equal via
the dataclass-generated RecordType.__eq__, without raising — confirmed
to fail against the raising design, which is the actual counterexample
that caused the reversion. A third test covers the real enforcement
point instead: a static AST check flags fact_a == fact_b/fact_a !=
fact_b (or against a literal) written directly inside a diff_*.py
detector module, the same mechanism and file scope as the existing
.value_or() rule.
Port the existing tests/test_vtable_evidence_guard.py
Hypothesis properties to assert over Fact[...] states directly, not only
derived booleans; add a property asserting no detector in diff_types.py/
diff_layout.py pattern-matches a Fact[...]-typed field without handling
every FactStatus variant (a static AST check, mirroring
check_ai_readiness.py's own style, is preferable to a runtime check here
since the failure mode is a missing case, not a bad value). Add a direct
serialization.py round-trip test per converted field pinning the
backfill rule itself: a pre-conversion fixture snapshot with the
reliability flag True loads as Fact.present(...), one with the flag
False loads as Fact.not_collected() — not Fact.present([])/
Fact.present(False) — since that exact confusion (a placeholder value
read as a confirmed fact) is the bug this phase exists to make
unrepresentable; a freshly-extracted snapshot round-trips through every
backend's real parser and never consults the legacy flag at all. Add a
second, direct test asserting snapshot_to_dict() on a freshly-built
snapshot never emits a raw FactStatus enum member anywhere in the
resulting dict (walk the tree and assert every value is a JSON-primitive
type) — confirmed to fail against the pre-fix asdict()-only path, which
is exactly the failure mode a reviewer caught in this design. A third test
covers the direct-construction compatibility bridge: RecordType(...,
vtable=["f"]) with no vtable_fact given backfills to
Fact.present(["f"]); RecordType(..., vtable=["f"],
vtable_fact=Fact.not_collected()) keeps the explicit Fact value rather
than the backfilled one, pinning the stated precedence directly; and a
bare Param(is_va_list=True) with no is_va_list_fact given backfills
the same way — each confirmed to fail against a version of the dataclass
that defaults the new field to Fact.not_collected() instead of None.
A fourth test pins the omission-marker fix directly, the exact
counterexample that caught the gap in the first design, across both
mechanisms: RecordType()
(nothing touched — the common case, not the nonempty/True cases the
third test above already covers) backfills vtable_fact/bases_fact to
Fact.not_collected(), never Fact.present([]); RecordType(bases=[])
(an explicit, confirmed-empty base list) backfills to Fact.present([]),
distinct from the previous case despite both leaving self.bases == [];
and a bare Param() backfills is_va_list_fact to Fact.not_collected(),
never Fact.present(False), while Param(is_va_list=False) (an explicit,
confirmed-not-variadic value) backfills to Fact.present(False) — each
confirmed to fail against the truthiness-based (non-sentinel, non-None)
version of the bridge, which cannot tell the two cases apart for either
field shape. A fifth test pins the type itself, both statically and at
runtime: Param.is_va_list's/RecordType.bases's declared annotation
stays exactly bool/list[str] — no union, confirmed by running mypy
--strict against a fixture module using the compatibility bridge and
asserting zero errors, which also confirms dataclasses.fields(...)'s
recorded type is unwidened — and after construction (with or without
the argument) self.is_va_list/self.bases is always a plain bool/
list[str], never the sentinel, confirming the cast sentinel normalizes
away before any reader, including asdict()-based serialization, can
observe it. A
sixth test pins vptr_offset_bits's own third mechanism, the exact
counterexample that caught this field shape missing entirely: a bare
RecordType() backfills vptr_offset_bits_fact to Fact.not_collected(),
while RecordType(vptr_offset_bits=None) (an explicit, confirmed-no-vptr
value) backfills to Fact.present(None) — distinct from the previous
case despite both leaving self.vptr_offset_bits is None — confirmed to
fail against a version of the bridge that reuses None itself as both
the field's natural value and its omission marker. A seventh test pins
the legacy-field-resync fix: RecordType(vtable=["old"],
vtable_fact=Fact.present(["new"])) ends construction with self.vtable ==
["new"], not ["old"] — confirmed to fail against a version of the
bridge that lets the explicit Fact[...] value win for vtable_fact
itself while leaving self.vtable unsynchronized, which is the exact
two-disagreeing-representations counterexample this fix closes. An
eighth test pins the DWARF fixed-point-resolution fix directly, through
the real dwarf_snapshot.py resolution pass rather than a hand-built
RecordType: a base class with a real vtable and a derived class
inheriting its vptr through the virtual-primary-base fallback (the exact
shape that pass exists to resolve), asserting the derived record's
vptr_offset_bits_fact reads Fact.present(resolved) — not
Fact.not_collected() — after the pass runs, matching its own
now-resolved vptr_offset_bits value; confirmed to fail against a
version of the fix that updates only the legacy field at the resolution
site and leaves the Fact field frozen at its pre-resolution,
construction-time state.
Acceptance criteria. The three converted fields cannot be read by any
detector without explicit availability handling — enforced by a new
check_ai_readiness.py check flagging, inside diff_*.py/any detector
module — idioms.py named explicitly here, since a first draft of this
criterion scoped itself to diff_*.py and missed it: it is semantic
detector logic reached by pattern_verdicts.py (ADR-027 D2.2's single-
snapshot anti-pattern recognition), reading rec.vtable/rec.bases
directly to emit polymorphic-type anti-pattern findings, not a
presentation or bridge module — the check's module-scope list is every
module producing a Change/finding from model data, not only the ones
named diff_*.py — either a bare attribute read of a Fact[...]-typed field or a
.value_or(...) call on one (both collapse the status space the same
way); .status/pattern-match access is the only permitted form there.
.value_or(...) itself is not banned repository-wide — it stays legal in
presentation-only modules (reporter.py/html_report.py/sarif.py and
siblings), which is a real, narrower allowlist, not "anywhere outside
model/fact.py."
The check as stated above cannot actually close this gap by itself — it
recognizes only reads of the new, Fact[...]-typed field names
(vtable_fact/bases_fact/vptr_offset_bits_fact/is_va_list_fact),
never a detector that keeps reading the retained legacy attribute
(rec.vtable, rec.bases, rec.vptr_offset_bits, param.is_va_list)
directly. This phase's own compatibility bridge keeps those legacy
fields populated and normalized specifically so existing, unmigrated
callers keep working — but that same retention means a detector can
continue reading rec.vtable (unchanged, still a plain list[str], still
passes every existing type check) and never touch the Fact[...] field or
the new check at all, bypassing availability handling entirely while
looking, to both a type checker and this AST check, exactly like a
correctly-migrated detector.
Two different checks are in play here, and a first draft of this
criterion conflated their scopes — narrowing the legacy-attribute
widening to "only inside a detector module" silently abandoned the
repository-wide scope the reader-migration check above (lines 668–682)
already commits to, which is exactly the contradiction review caught:
surface.py/export_surface.py/dumper_scoping.py/contract_evidence_
collect.py/internal_leak.py/type_reachability.py are real,
documented semantic readers of bases/vtable/virtual_bases/
is_va_list — the whole reason they're on this phase's own migration
list and initial known-failures baseline — but none of them is a
diff_*.py-shaped detector module, so narrowing enforcement to
"detector module" would let any of them reintroduce a direct legacy-field
read after this phase ships with nothing catching it, recreating the
exact unavailable-vs-empty confusion the migration exists to close.
The legacy-attribute-name widening (rec.vtable/rec.bases/
rec.virtual_bases/rec.vptr_offset_bits/param.is_va_list, on a value
whose declared type
resolves to RecordType/Param) therefore stays the repository-wide
scan already specified above — covering every one of the nine-plus named
semantic-reader modules, not only detector modules — with the
compatibility bridge's own __post_init__ (which legitimately reads and
writes the legacy field to backfill/resync it) as the one named exemption,
and serialization/asdict()-based external consumers likewise exempt
since they read the dataclass generically, not by naming the field.
Only the other half of the check — a bare Fact[...]-typed field read
or a .value_or(...) call on one — keeps the narrower, detector-module-
only file-scope restriction, because that half's exemption (presentation
modules legitimately calling .value_or(...) to render a display string)
has no equivalent for a direct legacy-field read: nothing about rendering
a value for display justifies bypassing the availability-aware field
entirely when the legacy attribute is sitting right there with the same
information, unconverted. Full test suite green; FP-rate/
tier-accuracy gates unchanged (this phase changes representation, not
detector logic).
Landed (first slice), not the whole phase — read this before assuming
the Design section above is fully implemented. abicheck/model/
availability.py (relocated FactStatus/Confidence), abicheck/model/
fact.py (Fact[T], the cast()-sentinel omission mechanism for all
three field shapes), RecordType.bases_fact/virtual_bases_fact/
vtable_fact/vptr_offset_bits_fact, and Param.is_va_list_fact are
real and tested (tests/test_model_fact.py,
tests/test_serialization_roundtrip.py::TestFactFieldRoundTrip).
serialization.py encodes/decodes the new fields and bumps
SCHEMA_VERSION to 26, backfilling a legacy snapshot correctly from the
existing clang_vtable_facts_reliable/clang_va_list_facts_reliable
flags (split into storage/fact_codec.py/storage/enum_codec.py — both
ADR-061 storage-layer leaf modules, depending on nothing beyond model —
to stay under the 2000-line file-size cap).
Producer-side Fact[...] construction (second slice) — landed.
dumper_castxml.py, dumper_clang.py, and dwarf_snapshot.py now each
construct RecordType.bases_fact/virtual_bases_fact/vtable_fact/
vptr_offset_bits_fact explicitly at parse time via a new shared helper,
model.entities.record_layout_facts() (spread into the RecordType(...)
call alongside the matching legacy keyword arguments) — every one of these
three producers genuinely resolves bases/virtual_bases/vtable/
vptr_offset_bits itself (real semantic analysis for castxml, an AST walk
for direct-clang, DWARF DIE traversal for the binary backend), opaque
records included, so this is a pure "state what was already true
explicitly" change with zero observable behavior difference on its own —
the omission bridge already derived the identical Fact.present(...) from
the always-supplied legacy value. DWARF's vptr_offset_bits is the one
field that may still be None pending _finalize_vptr_offsets's own
later fixed-point pass; that pass already resyncs both representations via
resolve_vptr_offset_bits() from the prior slice, untouched here.
The one real, deliberate behavior change: Param.is_va_list_fact is now
Fact.unsupported() on both dumper_castxml.py and dwarf_snapshot.py —
neither producer can ever determine va_list-ness, on any run, which
UNSUPPORTED states plainly where the omission bridge's own
NOT_COLLECTED ("not collected this time") could not. dumper_clang.py's
is_va_list_fact is Fact.partial(...), not Fact.present(...) (Codex
review, second round): that backend genuinely evaluates
_clang_param_is_va_list() per parameter, but the check only covers
x86-64 System V and conservatively answers False — not a confirmed
negative — on any other target (the function's own documented
target-scoping residual — no per-snapshot record of which target ABI a
clang parse actually ran against — is unchanged; closing it still needs
the toolchain-identity probe named elsewhere in this repo's AGENTS.md).
vptr_offset_bits_fact is Fact.partial(...) on castxml/clang too (a
third finding, same review): both backends derive it from the same
0-if-polymorphic Itanium primary-base heuristic that the legacy
vptr_offset_bits field's own capability-matrix row already documents as
PARTIAL — it does not track a secondary vtable's placement under
multiple inheritance, so the Fact[...] sibling must not overclaim FULL
either. scripts/backend_capabilities.py's FACT_ROWS (and its
_is_placeholder_literal scanner, taught to recognize a
Fact.unsupported()/not_collected()/not_applicable()/failed() call
as a placeholder rather than extraction, and to recurse into
Fact.present()/partial()'s own wrapped argument) were corrected to
match. New tests: tests/test_castxml_fact_construction.py,
tests/test_dumper_clang_fact_construction.py,
tests/test_dwarf_fact_construction.py — the is_va_list_fact change is
confirmed to fail against the pre-change code (git stash); the
representational-only assertions pass either way, since they pin
already-correct, now-explicit behavior rather than a regression.
The widened, non-glob AI-readiness check — landed (third slice).
scripts/fact_field_readers.py (registered by check_ai_readiness.py as
fact-field-readers) is the real, repo-wide static scan this Design
section names: an AST walk for a direct attribute read (ast.Load) of
bases/virtual_bases/vtable/vptr_offset_bits/is_va_list anywhere
under abicheck/, not a diff_*.py glob — because a glob is exactly what let this section's
own hand-enumeration repeatedly miss a real reader across several review
rounds (the fifth, then the tenth call site each surfaced only in a later
pass). Auditing the real tree while building this check (rather than
trusting that hand list) found three more genuine semantic readers the
plan's own nine-reader table doesn't name:
buildsource/header_graph.py's inheritance-edge emitter (walks rt.bases
to build the L5 source graph's INHERITS edges), buildsource/
source_extractors/base.py's entity_from_record() (folds rec.bases/
rec.vtable into the L4/L5 source-ABI-replay entity-identity fingerprint),
and idioms.py's factory/non-virtual-destructor anti-pattern detectors
(_recognise_factory/_is_virtual_dtor_present/_collect_base_targets/
_detect_non_virtual_dtor, all reading rec.vtable/rec.bases directly)
— confirming this Design section's own conclusion in the most direct way
possible: a hand-maintained list is not the same invariant as "every known
reader is checked," and it takes exactly this kind of real scan to find
the readers a hand audit keeps missing.
The check's baseline (KNOWN_UNMIGRATED_READERS, allowlist-and-shrink,
IMPORT_CYCLE_ALLOWLIST's own convention) currently records every known
reader site across the modules named above and throughout this Design
section (including vptr_offset_bits and every dynamic-read form later
review rounds added -- getattr/its aliases, operator.attrgetter,
__getattribute__, MatchClass keyword and positional patterns -- each
described in its own round below) — the exact count is deliberately not
restated here as a number (Codex review, fresh evidence: an earlier
revision of this paragraph copied the count by hand and went stale the
moment a later round changed it, the same drift risk every "stays at N"
review-note copy below carried too -- those now read "the baseline count
is unchanged" instead of a literal figure, for the identical reason).
len(KNOWN_UNMIGRATED_READERS) in the source is this baseline's own fact
owner, and the count is explicitly expected to shrink as readers migrate.
Each site is keyed "<rel>::<qualname>::<attr>::
<outer-expr-text>::<expr-text>::<occurrence>" -- qualname the enclosing
function (<module> for module-level code, Class.method for a method),
outer-expr-text the read's own outermost containing expression
(_outermost_containing_expr, climbing every enclosing ast.expr up to
the first statement boundary), expr-text the read's own exact bare
source text (ast.get_source_segment), occurrence a rank among reads
sharing all four of those. This is the final key shape, landed by the
"fingerprint the containing expression" round further below -- the count
and key shape in this paragraph previously described only the check's
first landed slice, silently contradicting the append-only review notes
below it once those notes moved past it (Codex review, fresh evidence);
readers auditing the gate should treat the review notes' own final state
as authoritative, and this paragraph is now kept in sync with it rather
than left as a stale first draft. That first draft keyed only
"<rel>::<attr>::<occurrence>"; a
Codex review round caught the real gap in that: an existing read migrated
or deleted and a different, unrelated read of the same attribute later
added to the same file would silently inherit the vacated occurrence
number and read as an already-reviewed site — scoping the counter to
(qualname, attr) needs the new read to land in the exact same function at
the exact same rank to collide, a real coincidence rather than a routine
edit's side effect.
A second Codex round found the function-scoped key still collides.
diff_param_qualifiers.py's if not p_old.is_va_list and p_new.is_va_list:
has two DIFFERENT reads (p_old.is_va_list, p_new.is_va_list) on one
line, in the same function — a purely positional rank can't tell them
apart, or protect a future, unrelated third read from inheriting one's
rank once it's migrated away. Fixed by folding the read's own source text
into the key (ast.get_source_segment, scoping the occurrence counter to
(qualname, attr, text)) — a collision now needs the new read to be the
textually identical expression, not merely occupy the same rank in the
same function. Every key in the baseline below carries this shape.
EXEMPT_FUNCTIONS is the separate, non-shrinking set of specific
functions this check never flags: RecordType.__post_init__/
Param.__post_init__ (the fields' own __post_init__ omission-bridge
implementation) and two specific DWARF functions,
dwarf_snapshot._DwarfSnapshotBuilder._finalize_vptr_offsets and
dumper_layout_backfill._backfilled_record, which compute or combine the
raw legacy value itself rather than making a compatibility decision from
it. Function-scoped, not module-scoped — a first draft exempted the
whole dwarf_snapshot.py/dumper_layout_backfill.py modules, and a
Codex review round found that was wrong: both files hold a second kind
of function that genuinely decides something from these fields rather
than merely computing them --
_DwarfSnapshotBuilder._filter_types_by_reachability reads bases/
virtual_bases to decide which types survive into the exported snapshot
(dropping one on a false "no bases" reading is a real, silent
correctness loss, not a cosmetic one), and dumper_layout_backfill.
_fields_corroborate reads bases/virtual_bases/vtable to decide
whether two records structurally match across the header/DWARF backfill.
A whole-module exemption hid both from the scan entirely; narrowing to
function scope surfaced them as two more genuine, previously-invisible
readers, now real KNOWN_UNMIGRATED_READERS entries themselves (bringing
the total from the originally-reported 91 to 99).
Two more real gaps found in the same review round, both fixed. (1)
RecordType.vptr_offset_bits was originally left out of
FACT_BRIDGED_ATTRS on the theory that it was "already meaningfully
None... never itself ambiguous" before this phase — false: it carries
the identical _OMITTED_VPTR_OFFSET_BITS sentinel-based omission bridge
the other four fields use (RecordType() backfills Fact.not_collected();
RecordType(vptr_offset_bits=None) backfills Fact.present(None) — two
different Facts for the same None legacy value), so a direct read has
the identical ambiguity. Adding it surfaced four more real reader sites
(diff_layout.py's _check_vptr_introduced/_has_layout_descriptor).
(2) The scan matched only ast.Attribute nodes, missing a dynamic
getattr(obj, "vtable", ...) read with the attribute name as a literal
string constant — diff_cpp_patterns._is_empty_record reads vtable
exactly this way, invisible to the original scan. Fixed by also matching
a getattr() call whose second argument is a string-literal ast.Constant
naming one of the bridged attributes (a non-literal name stays out of
scope, the same no-type-inference limit already stated for the attribute
case). No type inference is attempted — verified empirically, by running
the scan against the whole package before writing the baseline, that
every hit recorded there is genuinely a RecordType/Param access
(attribute or getattr) with zero cross-class collisions. Tests:
tests/test_fact_field_readers.py (the real repository has zero unlisted
violations; every baseline/exempt entry still names something real; the
AST primitive's own contract — attrs detected including vptr_offset_bits,
getattr calls detected and a non-matching/dynamic name ignored, Store
ignored, per-function occurrence numbering, module/class qualname
derivation — pinned directly; and end-to-end cases against a throwaway
abicheck/-shaped tree confirming the check function itself, not only the
primitive, fires on a new violation, stays silent on a baselined one, and
respects a function-scoped exemption without leaking to a sibling function
in the same file).
A fifth Codex review round found two more real gaps, both fixed. (1)
The scan matched an ast.Attribute read and a getattr() call, but not a
structural-pattern-matching read: case RecordType(bases=[]): reads
bases via ast.MatchClass.kwd_attrs (a list[str], paired positionally
with kwd_patterns), a node shape neither branch recognized — invisible
to the check even though it collapses unavailable and confirmed-empty the
same way a direct attribute read does. Fixed by matching a MatchClass
node's kwd_attrs against FACT_BRIDGED_ATTRS too, keyed by the matched
keyword pattern's own location and the whole class pattern's source text
(RecordType(bases=[])) as expr-text, since a MatchClass node carries
no location for a single keyword on its own. Verified empirically to have
zero existing hits — no match/case statement in the repository
currently patterns on any of these five fields, so the baseline count is unchanged. (2) The root AGENTS.md's AI-readiness gate table (the canonical
verification contract every other check is listed in) had no row for
fact-field-readers, leaving it undiscoverable from that table — added
alongside engine-cli-boundary's own row. New test:
test_detects_a_structural_pattern_match_naming_a_bridged_attr.
A sixth Codex review round found two more real gaps in the same area,
both fixed. (1) Positional class patterns: case RecordType(_, _, _,
_, _, []): reads a field via cls.__match_args__ positionally, which
MatchClass.kwd_attrs (the previous fix's whole mechanism) doesn't cover
at all — an empty kwd_attrs list for a purely positional pattern, so
the loop finds nothing. Resolving which position maps to which field
would need real __match_args__ introspection (the dataclass's own
declared field order) this pure-AST scan can't do — instead of leaving
this silently invisible, ANY non-empty positional pattern on a
MatchClass whose cls resolves by bare name to RecordType/Param
(FACT_BRIDGED_CLASS_NAMES, a new two-name constant) is reported with a
synthetic <positional> attr, on this module's own established
false-positive-over-false-negative principle. (2) A deeper, third-round
collision in the baseline key itself: two DIFFERENT call sites sharing
identical bare attribute text — old_decision(rec.bases) and, elsewhere
in the same function, keep(rec.bases) — still differed only by
occurrence ordinal even after the previous two rounds' fixes (function
scope, then bare-expression text), because the key never looked past the
attribute node itself to the expression containing it. Migrating
old_decision away and adding an unrelated third read anywhere in the
same function re-numbers the survivors in encounter order — keep
silently drops to rank 1 (harmless, it was already reviewed), but the new
read then lands on rank 2, silently inheriting keep's own baseline
approval. Fixed with _outermost_containing_expr(): a one-pass parent
map, then climbing every enclosing ast.expr up to (but never past) the
first statement boundary. Deliberately the containing expression, not
the containing statement — a first attempt at this same fix climbed to
the nearest ast.stmt instead, which for if not p_old.is_va_list and
p_new.is_va_list: changes.append(make_change(...)) pulled the entire
if-body into the key (a ~970-character entry that would silently break
on any unrelated edit inside that body) — caught before landing by
checking the actual generated key lengths, not just that the fix
compiled. Stopping at the expression boundary gives the whole boolean
test (not p_old.is_va_list and p_new.is_va_list, shared correctly by
both reads, still disambiguated between them by the existing bare-text
component) without the body. Key format is now "<rel>::<qualname>::
<attr>::<outer-expr>::<expr-text>::<occurrence>" — five components before
the ordinal, not three. Baseline regenerated directly from the real scan
(still the same number of entries, only the key shape changed) and re-verified at zero
unlisted violations. New tests: two different call sites with identical
bare text now getting distinct keys with no ordinal needed, a compound
statement's body confirmed excluded from the key, a positional pattern on
a bridged class detected, and one on an unrelated class ignored.
A seventh Codex review round found the positional-pattern fix itself
was defeated by an import alias, fixed in the same PR. from
abicheck.model import RecordType as RT then case RT(_, _, _, _, _,
[]): names the identical class, but a bare node.cls.id in
FACT_BRIDGED_CLASS_NAMES check rejects "RT" outright — invisible to
the positional-pattern fix despite being exactly the shape it exists to
catch. Fixed with _imported_class_aliases(): maps every local name an
import/from ... import statement binds one of FACT_BRIDGED_CLASS_
NAMES under (via as) back to its real name, whole-tree rather than
function-scoped (an import is visible for its entire enclosing scope
regardless of where a later pattern uses it) — the positional-pattern
check resolves through this map before testing membership. Verified
empirically: still zero existing hits, baseline count is unchanged. New tests:
a positional pattern through an import alias detected, one on an alias
of an unrelated class ignored.
An eighth Codex review round found two more real gaps, both fixed in
the same PR -- the same two mechanisms, probed one alias mechanism
further each. (1) RT = RecordType (a plain module-level assignment,
not an import ... as) then case RT(_, _, _, _, _, []): -- the
import-alias fix from the previous round only visited Import/
ImportFrom nodes, so a simple name assignment was invisible to it.
Fixed by extending _imported_class_aliases() to also collect a
single-target name = OtherName assignment as a candidate, resolved to
FACT_BRIDGED_CLASS_NAMES (directly, or transitively through an
already-resolved alias -- RT2 = RT chains too) the same fixed-point
way fact_detector_misuse._fact_aliases already chains local aliases.
(2) import builtins; builtins.getattr(rec, "bases") and from builtins
import getattr as read_attr are both the identical read as a bare
getattr(rec, "bases") call, but the existing check only matched an
ast.Call whose func was literally the bare ast.Name "getattr".
Fixed with _builtins_getattr_aliases(): collects every local name bound
to the real getattr builtin (the bare name itself, plus any from
builtins import getattr as X) and every local name bound to the
builtins module itself (import builtins, import builtins as b),
and the getattr-detection branch now also matches a qualified call
(<name>.getattr(...)) whose base resolves to one of those module
names. Verified empirically: still zero existing hits, baseline count is unchanged. New tests: a local class-name alias detected and a negative control
for an unrelated class, a qualified builtins.getattr call detected, an
aliased getattr import detected, and a negative control for a call
qualified through an unrelated module.
A ninth Codex review round found two more real gaps, both fixed in the
same PR -- the same two mechanisms probed one further indirection layer
each, matching the eighth round's own framing. (1) read_attr = getattr
then read_attr(rec, "bases") is the identical dynamic read as a bare
getattr(rec, "bases") call, but _builtins_getattr_aliases() only ever
collected the bare name and a from builtins import getattr as X import
-- a plain assignment chain to the builtin callable was invisible, the
identical gap the eighth round's fix (1) already closed for a class
alias but left open for getattr itself. Fixed by extending
_builtins_getattr_aliases() with the same fixed-point assignment-chaining
_imported_class_aliases already does (RT = RecordType, RT2 = RT),
just for the builtin callable instead of a class name. (2) rec.bases +=
inherited updates a bridged field, but Python marks the target Attribute
node ast.Store even though the operation reads the field's existing
value first, to combine it with the right-hand side, before writing the
result back -- an ordinary Store/Del (a plain overwrite) genuinely
never reads, but an AugAssign target always does, and the existing
Load-only restriction missed this distinction entirely: the target
Attribute node is still visited independently by ast.walk (it's a child
of the AugAssign), but its Store context skipped the ordinary
attribute branch too. Fixed with a dedicated ast.AugAssign branch
matching the target's own attribute name, keyed on the target Attribute
node itself (not the whole AugAssign statement) so its site/text line up
with an ordinary attribute read at the same position. Verified
empirically: still zero existing hits, baseline count is unchanged. New tests: a
local getattr alias detected, a chained getattr alias detected, an
augmented assignment detected, and a negative control confirming an
ordinary Store overwrite is still not flagged.
A tenth Codex review round found two more real gaps, both small,
bounded extensions of already-built mechanisms -- fixed, matching the
review-convergence status already posted on this PR (further exotic
indirection is documented as a known gap; a bounded combination of two
already-supported mechanisms is not). (1) RT: type = RecordType then
case RT(_, _, _, _, _, []): -- an ordinary annotated assignment
(ast.AnnAssign), not the ast.Assign the class-alias collector already
matched. The same gap the eighth round's fix (1) closed for a plain
RT = RecordType, reached through a differently-typed AST node the
collector never visited. Fixed by adding an ast.AnnAssign branch to
_imported_class_aliases() alongside the existing ast.Assign one,
feeding the identical fixed-point chain. (2) read_attr = builtins.
getattr then read_attr(rec, "bases") -- combining two mechanisms this
file already supports independently (the qualified-call recognition from
round eight and the plain-assignment chaining from round nine) in a way
neither alone catches: the assignment's own value is builtins.getattr
(an ast.Attribute), not a bare ast.Name, so the existing candidate
collector -- which only ever matched an ast.Name value -- never added
it. Fixed with a second candidate list resolved once, after the walk
finishes collecting every import builtins occurrence (needed since,
unlike the plain-name candidates, this resolution needs the complete
builtins_names set before it can tell whether the qualifying name is
really the builtins module). Verified empirically: still zero existing
hits, baseline count is unchanged. New tests: an annotated class alias detected
through a positional pattern, and a qualified assignment to the
getattr builtin detected.
An eleventh Codex review round found one more real gap, the same
small, bounded shape as round ten's fix (1) -- fixed. read_attr:
Callable[..., object] = getattr then read_attr(rec, "bases") -- the
annotated-assignment spelling of the getattr-alias assignment, an
ast.AnnAssign the candidate collector never matched (it only ever
walked ast.Assign). The same gap round ten's fix (1) already closed
for _imported_class_aliases(), now closed for _builtins_getattr_
aliases() too, for both the plain-name and qualified-attribute value
shapes -- factored into one shared _add_candidate() helper so the
ast.Assign and ast.AnnAssign branches can't independently drift on
which value shapes each recognizes. Verified empirically: still zero
existing hits, baseline count is unchanged. New tests: an annotated getattr
alias detected, and the annotated form of the qualified spelling
(read_attr: object = builtins.getattr) detected too.
A twelfth Codex review round found one more real gap, the class-alias
mirror of round eleven's getattr fix -- fixed. import abicheck.model
as model; RT = model.RecordType then case RT(_, _, _, _, _, []): --
combining two already-supported forms (a qualified class reference,
already recognized when used directly as model.RecordType(...)-shaped
construction elsewhere in this file's own reasoning, and an assignment
alias) in a way neither alone catches: the assignment's own value is
model.RecordType, an ast.Attribute, not a bare ast.Name, so
_imported_class_aliases()'s candidate collector never resolved it.
Fixed by resolving a qualified-attribute RHS immediately whenever its own
.attr is one of FACT_BRIDGED_CLASS_NAMES -- matching the identical
name-only stance the import ... as branch already takes for its own
qualifying source module (never checked either way) -- factored into one
shared _register_assign() helper (mirroring round eleven's
_add_candidate() for _builtins_getattr_aliases()) so the ast.Assign
and ast.AnnAssign branches can't independently drift on which RHS
shapes each recognizes. Verified empirically: still zero existing hits,
baseline count is unchanged. New test: a qualified class alias detected through
a positional pattern.
A thirteenth Codex review round found one more real gap, applying to
both alias resolvers at once -- fixed. RT = Alias = RecordType (an
ordinary chained assignment) then case RT(_, _, _, _, _, []):, and the
identical shape for _builtins_getattr_aliases(): read1 = read2 =
getattr then read1(rec, "bases") -- both resolvers' ast.Assign
branches were still restricted to len(node.targets) == 1, so a chained
assignment (every target receiving the identical RHS, unlike a
tuple-unpacking target, which has no single value to attribute) was
excluded the same way the fifth-Codex-round-in-fact_detector_misuse.py
finding was for that module's own candidate collector. Fixed by looping
over every target in node.targets and registering each plain-Name
one, in both _imported_class_aliases() and _builtins_getattr_
aliases(). Verified empirically: still zero existing hits, baseline
count is unchanged. New tests: a chained class alias detected through a
positional pattern, and a chained getattr alias detected.
A fourteenth Codex review round found one more real gap -- fixed.
import builtins; b = builtins then b.getattr(rec, "bases") --
builtins_names was only ever populated from a real import statement
(import builtins/import builtins as b), never from a plain assignment
alias of an already-known one. Fixed by resolving builtins_names to a
fixed point too, reusing the identical assign_candidates list the
getattr alias chain already builds (it already captures every simple
Name-valued assignment, not just getattr-related ones) -- resolved
before the existing qualified_candidates step, so b.getattr(...)
is recognized through the now-expanded builtins_names as well.
Verified empirically: still zero existing hits, baseline count is unchanged.
New test: a call through an assigned alias of the builtins module.
A fifteenth Codex review round found the scan still missed two more
standard dynamic-attribute-reading forms, distinct from getattr's own
alias family the previous several rounds closed -- both fixed.
operator.attrgetter("bases")(rec) and object.__getattribute__(rec,
"bases")/rec.__getattribute__("bases") read the identical legacy value
through different standard-library entry points -- getattr() is itself
defined in terms of __getattribute__, and attrgetter is the
callable-returning equivalent -- so unavailable evidence could again read
as confirmed-empty while the required gate passed. Fixed with a new
_is_attrgetter_constructor_call() helper (recognizing both the
qualified operator.attrgetter(...) spelling and the bare one reached
via from operator import attrgetter) plus two new branches matching a
__getattribute__ call in both its bound (rec.__getattribute__
("bases")) and unbound (object.__getattribute__(rec, "bases")/type.
__getattribute__(rec, "bases")) forms. Both stay scoped to the same "no
type inference" limit as every other branch here: only a single,
literal-string field name is recognized -- attrgetter("a.b")'s dotted
chain, and a two-step getter = attrgetter(...); getter(rec) indirection
through an intermediate variable, are both out of scope, the identical
limit a non-literal getattr() default already accepts. Deliberately no
local-alias resolution for attrgetter itself, unlike getattr's own
now-elaborate alias-chain tracking -- attrgetter returns a callable
assigned once and invoked later, which is exactly the two-step
indirection this scan already excludes for every other form, so building
the same machinery for it would recognize a shape this scan otherwise
deliberately doesn't. Verified empirically: still zero existing hits in
the real repository, baseline count is unchanged. Eight new tests: the
qualified and bare attrgetter forms, a dotted-name negative control, the
variable-indirection negative control, the bound and unbound
__getattribute__ forms, and a non-matching-name negative control for
the latter.
A sixteenth Codex review round found two more real gaps in the new
attrgetter recognition itself, both fixed. (1) An ordinary import
alias of either operator or attrgetter bypassed the gate. import
operator as op; op.attrgetter("bases")(rec) and from operator import
attrgetter as ag; ag("bases")(rec) both read the identical legacy field
as the unaliased spellings, but the fifteenth round's matching checked
only the exact literal names "operator"/"attrgetter". Fixed with a
new _operator_attrgetter_aliases() -- the identical import-alias
mechanism _builtins_getattr_aliases() already provides for getattr/
builtins, applied to this pair -- resolving import operator as X and
from operator import attrgetter as X into the two name sets _is_
attrgetter_constructor_call() now checks against, instead of two hard-
coded literals. Deliberately narrower than _builtins_getattr_aliases()'s
own multi-round evolution: no assignment-chain resolution (op2 = op,
ag2 = ag) was added, since neither finding asked for it and this stays
consistent with attrgetter's own existing "no local-alias resolution"
stance for a constructed getter value -- only the two import-form
aliases this finding named are covered. (2) Only the first argument to
attrgetter was inspected. attrgetter accepts any number of
positional field names and reads every one of them --
attrgetter("size_bits", "bases")(rec) reads bases too, but the
fifteenth round's own code checked only node.func.args[0], so a bridged
field named anywhere but first was invisible. Fixed by moving the whole
attrgetter case out of the single-attribute elif chain into its own
top-level handler (mirroring how MatchClass is already handled
separately, since it too can produce more than one match per node),
inspecting every literal, string-constant positional argument
independently and reporting each bridged one as its own site. Verified
empirically: still zero existing hits in the real repository, baseline
count is unchanged. Five new tests: the import ... as/from ... import ... as
alias forms, a multi-argument call with the bridged field in the second
position, a multi-argument call with two independently-bridged fields
(both reported), and a multi-argument negative control naming no bridged
field at all. A sixth, separate finding from the same round (a stale
"landed" paragraph in this Design section still describing the check's
first slice -- 99 sites, the pre-fingerprint key shape -- contradicting
the append-only review notes once they moved past it) is fixed directly
in that paragraph itself, not narrated as its own round here.
A seventeenth Codex review round found two more real gaps, one in the
new attrgetter alias resolution and one a genuine false positive in the
existing getattr recognition -- both fixed, the second with a real
mid-fix correction. (1) A plain-assignment alias of operator/
attrgetter was still out of scope. import operator as op; op2 = op;
op2.attrgetter("bases")(rec) and from operator import attrgetter as ag;
ag2 = ag; ag2("bases")(rec) read the identical legacy field as the
unaliased/singly-import-aliased spellings, but the previous round's own
docstring claimed "a Call-typed value has no simple assignment shape to
chain through" -- true, but irrelevant: op/ag are ordinary references
(a module, a builtin callable) before being called, chaining through a
plain assignment exactly the way _builtins_getattr_aliases()'s own
getattr/builtins resolution already does. Fixed by giving
_operator_attrgetter_aliases() the identical fixed-point assignment-
chaining pattern that function already uses. (2) A local binding that
shadows the bare getattr/builtins names was never excluded from the
builtin match at all. def f(getattr, rec): return getattr(rec,
"bases") -- an ordinary, unrelated function parameter reusing the name
getattr -- was unconditionally treated as the real builtin, blocking a
valid, unrelated change. Fixed with a new _locally_bound_names(),
consulted at each getattr/builtins match site. The first revision of
that helper also covered ordinary ast.Assign/ast.AnnAssign targets,
not just parameters, and immediately broke six existing tests --
read_attr = getattr; read_attr(rec, "bases") IS a genuine local
assignment target of the identical shape, but treating that assignment as
"shadowing" is backwards: it's how the existing alias-resolution
mechanism makes read_attr trustworthy in the first place, not a reason
to distrust it. Telling a real shadow (getattr = some_unrelated_value)
apart from a real alias assignment (read_attr = getattr) needs
per-assignment tracing of what each target's own value resolves to --
information _builtins_getattr_aliases()'s own internal assign_
candidates already computes but doesn't expose, and doing so scoped
per-function is a real, separate follow-up, not attempted here. Since a
parameter can never be an alias source in that same sense (nothing in a
function signature assigns FROM getattr), _locally_bound_names() was
narrowed to parameters only, closing the reported false positive with no
risk of this conflict -- and def f(rec): getattr = some_other_thing;
getattr(rec, "bases") (an ordinary reassignment shadow, not reported by
this round) is left as a documented, accepted residual false positive
rather than a silently reintroduced one, pinned by its own test. Verified
empirically: still zero existing hits in the real repository, baseline
count is unchanged, and mypy/ruff both stayed clean. Five new tests: the
two chained-alias attrgetter/operator forms, the shadowed-parameter
negative control, its sibling-function negative control (shadowing in one
function must not leak into another), and the residual-gap documentation
test.
An eighteenth review round found one more real gap and one real test-
suite performance problem, both fixed. (1) The attrgetter
recognition never consulted the shadowing check the getattr branch
already got. def f(operator, rec): return operator.attrgetter("bases")
(rec) -- an ordinary, unrelated parameter reusing the name operator --
was unconditionally treated as the real module, the identical false
positive the getattr-shadowing round fixed for that branch specifically,
left open here since this branch was added afterward and never wired to
_locally_bound_names()/_shadowed() at all. Fixed with a new
_attrgetter_matched_name() helper (re-deriving which of _is_attrgetter_
constructor_call()'s two recognized shapes actually matched, narrowly
typed so the call site doesn't need its own unchecked ast.Attribute/
ast.Name assumption -- mypy can't narrow node.func.func's type through
a boolean and-chain the way an explicit isinstance inside a dedicated
function can) and a not _shadowed(...) guard on the same branch. (2)
Three of this file's own tests each independently re-parsed and
re-scanned every abicheck/**/*.py file from scratch. test_no_
unlisted_violation_in_real_repo/test_baseline_entries_are_real_sites/
test_exempt_functions_are_real_sites each ran their own full
PKG.rglob("*.py") walk -- ~14s apiece on a real measurement, ~42s total,
consuming close to this whole file's share of the documented 43-second
fast-suite budget by itself. Fixed with a new module-scoped _repo_
reader_scan pytest fixture running the walk exactly once; all three
assertions are pure derivations of the identical underlying scan data.
test_no_unlisted_violation_in_real_repo's own derivation no longer
calls check_fact_field_readers() directly, but is provably equivalent
to it -- that function's own filtering is exactly the same two-step check
(EXEMPT_FUNCTIONS, then KNOWN_UNMIGRATED_READERS membership) applied
to the identical scan, confirmed by reading its source rather than
assumed; check_fact_field_readers()'s own Findings/message-building glue
stays independently, directly tested by this file's existing synthetic-
fixture tests (test_check_reports_a_new_violation/test_check_is_
silent_for_clean_code), so nothing lost real coverage. Verified
empirically: still zero existing hits, baseline count is unchanged, mypy/
ruff both stayed clean, and this file's own runtime dropped from ~26s
to ~9s for its (now 60, previously 57) tests. Three new tests: the
shadowed-operator-parameter and shadowed-bare-attrgetter-parameter
negative controls, and a sibling-function negative control (shadowing in
one function must not leak into another's genuine attrgetter() call).
A nineteenth review round found the identical shadowing gap in the
unbound __getattribute__ branch's sibling -- the bound form
(rec.__getattribute__("bases")) has no builtin name to shadow (its
receiver is an arbitrary object, exactly like rec.bases_fact itself), but
the unbound form (object.__getattribute__(rec, "bases")) names the
builtin object/type the same way the getattr/attrgetter branches
name theirs, and it was the one branch of the four dynamic-reader forms
never wired to _shadowed(): def f(object, rec): return object.
__getattribute__(rec, "bases") -- an ordinary, unrelated parameter reusing
the name object -- was unconditionally treated as the builtin, the
identical false positive already fixed for getattr and (eighteenth round)
attrgetter. Fixed with a not _shadowed(node, node.func.value.id) guard
on the same condition, mirroring the other three branches exactly -- no new
helper needed here, since node.func.value.id is already narrowed to
ast.Name by the existing isinstance checks in the same and-chain, one
level shallower than the attrgetter branch's node.func.func access that
needed its own dedicated helper in the eighteenth round. Verified
empirically: still zero existing hits in the real repository, baseline
count is unchanged, mypy/ruff both stayed clean. Three new tests: the
shadowed-object-parameter and shadowed-type-parameter negative
controls, and a sibling-function negative control (shadowing in one
function must not leak into another's genuine unbound
object.__getattribute__() call).
A twentieth review round found two more real gaps, both fixed. (1)
The unbound __getattribute__ receiver check matched only the two
literal spellings "object"/"type", missing an ordinary import
alias. from builtins import object as O; O.__getattribute__(rec,
"bases") is the identical dynamic read as the unaliased spelling, but
was invisible to the scan entirely. Fixed with a new
_unbound_getattribute_receiver_aliases() helper (always includes the
bare "object"/"type", plus any from builtins import object as O/
from builtins import type as T, plus a plain-assignment alias chain,
mirroring _builtins_getattr_aliases()'s own getattr-alias chaining
exactly), consulted in place of the literal two-element tuple; the
existing _shadowed() guard from the nineteenth round applies unchanged,
since it already checks whatever name was actually matched. (2)
_shadowed() consulted only a call's own innermost qualname, never an
enclosing function's binding. def outer(getattr): def inner(rec):
return getattr(rec, "bases") -- inner binds no parameter of its own
named getattr, but Python's ordinary closure rule still captures the
arbitrary callable outer's own parameter holds, so this was
unconditionally treated as the real builtin despite being shadowed one
scope up. Fixed with a new _lexical_function_parents() helper -- a
standalone copy of fact_detector_misuse.py's identical-purpose helper,
simplified to this module's own coarser, dot-joined qualname scheme (no
#lineno disambiguator, an existing, accepted characteristic of this
module's qualnames already) -- and widened _shadowed() to walk the
call's entire lexical scope chain, testing membership in each ancestor's
own locally_bound set in turn rather than only the innermost one.
Verified empirically: still zero existing hits in the real repository,
baseline count is unchanged, mypy/ruff both stayed clean. Four new tests:
the aliased-unbound-__getattribute__ positive case and its
unrelated-name negative control, and the enclosing-parameter-shadow
exclusion and its sibling-function negative control.
A further Codex review round found the attrgetter branch's own
baseline key never actually fingerprinted the read's containing
expression, unlike every other reader form. outer_text = ast.
get_source_segment(source, node) ... used the attrgetter call's own
bare text for both the outer-expression and expr-text key slots, instead
of climbing to the read's real outermost containing expression via
_expr_text() the way the plain-attribute/getattr/__getattribute__
branches already do. old_decision(attrgetter("bases")(rec)) and
keep(attrgetter("bases")(rec)) -- two attrgetter reads with an
identical bare call but different containing expressions -- collapsed to
the exact same key, ending in occurrence 1 for both once sorted by
position; migrating the first reader while adding an unrelated new one
at the same rank would silently reuse the vacated key, the exact
collision _outermost_containing_expr() exists to close for every other
form (the very finding that motivated that helper's own creation
several rounds above -- this branch was simply never wired to it). Fixed
by computing outer_text = _expr_text(node) (the containing expression)
separately from text (the call's own bare source), matching the
six-part key's established (qualname, attr, outer_text, text,
occurrence) shape exactly. Verified empirically: still zero existing
hits in the real repository (no attrgetter read exists anywhere in
abicheck/ today, so the baseline itself is unaffected by the key-shape
fix), baseline count is unchanged, mypy/ruff both stayed clean. New
test: two attrgetter reads sharing identical bare-call text inside
different containing expressions, pinning that they now get distinct
keys.
A further Codex review round confirmed, with a concrete repro, a gap
_locally_bound_names()'s own docstring had already predicted but left
unattempted -- recorded here as an accepted known gap rather than a bug
fix, since a correct fix needs real per-position dataflow this module
does not have. def f(getattr, rec): getattr = builtins.getattr; return
getattr(rec, "bases") -- a shadowing parameter later rebound to a
genuine alias source -- currently reports no site at all, even though the
call genuinely reads a bridged field through that rebound name (the
sibling attrgetter/operator shape has the identical gap). Correctly
distinguishing this from a genuine, still-shadowing parameter (getattr =
some_unrelated_value) needs order-aware tracing of which assignment is
actually in effect at the call's own position -- an order-blind "was
this name ever reassigned to a recognized alias anywhere in the scope"
check is unsound in the other direction, since def f(getattr, rec):
result = getattr(rec, "bases"); getattr = builtins.getattr calls
getattr before the rebind, while it still holds the arbitrary
parameter value, and would be wrongly excluded by that simpler check.
This module's presence/absence-only shadowing model has no notion of
"which binding is in effect here" at all; building one is a materially
larger change than every guard condition landed incrementally so far --
it took fact_detector_misuse.py's own alias machinery upwards of twenty
review rounds to reach exactly this kind of order-sensitivity for a
structurally similar problem. Recorded directly in _locally_bound_
names()'s own docstring (extending the paragraph that already predicted
this shape) rather than fixed under review pressure: this is a false
negative, the direction this module's own "a false positive is far
cheaper than the false negative it closes" trade-off argues hardest
against silently accepting, but an incorrect, order-blind attempt risks
trading it for a new false positive on a genuine shadow -- not obviously
an improvement, and not the kind of bounded, single-condition extension
every other round in this section landed. Revisit with real per-position
tracing if this shape is found in practice.
Two further Codex review rounds found two more real gaps, both fixed,
in the same area the previous several rounds had already been hardening.
(1) _operator_attrgetter_aliases() seeded "attrgetter"/"operator"
into their own alias sets unconditionally, mirroring how
_builtins_getattr_aliases() correctly seeds the real builtin
"getattr" -- but attrgetter/operator are not builtins; nothing makes
either name mean the stdlib module/function without a real import
operator/from operator import attrgetter somewhere in the file. An
ordinary, unrelated local function or variable sharing either name --
def attrgetter(rec, name): ... with no import anywhere -- was therefore
wrongly recognized as the real constructor. Fixed by seeding both sets
empty and relying entirely on the module's own existing real-import scan
(the same one every getattr/builtins qualified-alias branch already
required) -- an asymmetry the fix's own docstring now states explicitly,
since a future reader could otherwise "fix" the asymmetry the wrong way
by re-adding the unconditional seed. New tests: an unrelated local
attrgetter function with no import, and its dotted-access sibling (an
unrelated operator-named local variable with its own attrgetter
method), both in tests/test_fact_field_readers_wrapper_scoping.py. Two
of this area's own pre-existing shadowing tests
(test_ignores_attrgetter_shadowed_by_an_operator_parameter/
test_ignores_attrgetter_shadowed_by_a_bare_attrgetter_parameter) had no
real import in their fixtures either, so under this fix they would have
passed for the wrong reason (no import at all, rather than the shadowing
guard) -- both were corrected to include a real module-level import, so
they keep testing the shadowing check specifically rather than silently
degrading into a second copy of the new import-requirement test.
(2) _outermost_containing_expr() stopped climbing one level too early
at a keyword-argument value or a comprehension's own for ... in ...
clause, since neither ast.keyword nor ast.comprehension is itself an
ast.expr -- the climb's own guard condition (isinstance(parent,
ast.expr)) correctly stops at any ordinary non-expression boundary
(a statement), but these two wrapper node types are not statements
either; they are transparent syntactic scaffolding a real containing
expression still continues through. Two real examples surfaced by
regenerating the baseline against the fix: make_change(...,
old_value=str(t_old.bases), ...) (a keyword argument) and
{_topmost_scope_suffix(b) for b in header.bases + header.virtual_bases}
(a comprehension's own iterable clause) both had their previously-recorded
outer-expr-text narrower than the real surrounding expression, silently
under-recording how much context a reader site's own key actually
carries. Fixed with a new _TRANSPARENT_EXPR_WRAPPER_TYPES = (ast.keyword,
ast.comprehension) tuple, checked alongside ast.expr in the climb's own
condition. This legitimately reshaped 19 pre-existing, already-reviewed
KNOWN_UNMIGRATED_READERS entries (same underlying sites, wider
outer-expr-text) -- the baseline was regenerated from a fresh real-repo
scan rather than hand-edited, with the count confirmed unchanged (104
before and after) and every reshaped key confirmed a 1:1 rename of an
existing site, not a new or missing violation. New tests, in
tests/test_fact_field_readers_wrapper_scoping.py: a keyword-argument
read climbing to its enclosing call, a comprehension read climbing to its
enclosing display, and two reads sharing identical attribute/expression
text but wrapped in different keyword arguments of different calls,
pinning that each climbs to its own enclosing call rather than the two
collapsing onto a shared inner boundary.
Both fixes' new tests were split into a new sibling file,
tests/test_fact_field_readers_wrapper_scoping.py, rather than appended
to test_fact_field_readers.py -- which was already within a few lines of
the architecture gate's 1200-line test-file cap once the two test_
ignores_an_unrelated_*_with_no_import and three climbing tests were
added -- mirroring how tests/test_fact_detector_misuse_def_time_scope.py
was split out of test_fact_detector_misuse.py for the identical reason.
Verified empirically: still zero existing attrgetter/operator false
positives introduced against the real repository (the baseline
regeneration's own 104-in/104-out count is the check), mypy/ruff both
stayed clean, and python scripts/check_architecture.py reports 0 errors
with both test files under the cap.
A further Codex review round found a fresh repro for a residual this
module's own docstring had already named, explicitly, as an accepted,
narrow gap -- attempted a fix, and reverted it after the fix's own blast
radius and residual coverage didn't justify it. unmigrated_fact_reader_
sites's docstring already states: "A genuinely duplicated expression (the
identical containing expression appearing twice, with identical bare-read
text, in one function) still falls back to the ordinal, an accepted,
narrow residual." Codex's repro sharpens why that's a real risk, not
just a curiosity: decide(rec.bases) under if cond: and, separately,
under else: gets two baseline entries differing only by occurrence rank
(...::1, ...::2); deleting the if branch's call and adding a
different, genuinely new decide(rec.bases) call somewhere else in the
function re-numbers the survivors from scratch, so the new call can land
on the vacated rank and silently inherit an unrelated site's approval.
Attempted a fix: _branch_path(), walking a read's ancestors up to its
enclosing function and recording which alternative of each if/try/
match/loop-else it sits in, added as a new key segment. A first cut
used a bare structural label ("if"/"else") -- verified against the
exact repro and found insufficient: two separate, sibling if
statements in the same function both mark as plain "if" regardless of
which one, so Codex's own repro (the read moving from a deleted else:
to a different if statement elsewhere) still collided. Revised to
carry the branch's own governing text (f"if:{test text}",
f"except:{i}:{type text}", f"case:{i}:{pattern text}") rather than a
bare label -- deliberately still not the body text
_outermost_containing_expr's own docstring already rejected including
(a change anywhere inside the branch's own body would then reshape the
key for no reason); only the governing condition/type/pattern, which
changes only when the branch's own identity does. This version correctly
closed Codex's exact repro and a hand-built adversarial variant reusing
the identical condition-variable name across separate if statements,
verified empirically both ways.
Reverted anyway, after actually running the full existing suite --
not because the mechanism was wrong, but because its true cost wasn't
visible until measured. The new key segment is a real structural
change to the key shape for every entry, not just branching ones: a
non-branching read's branch_marker is "", but the key is still built
by "::"-joining a fixed number of components, so every one of the 40+
existing hand-written key-literal tests in this file failed -- not
because their logic was wrong, but because every expected key string now
needs an inserted empty "::" segment before the occurrence rank, and
the 104-entry KNOWN_UNMIGRATED_READERS baseline needs a full
regeneration for the same reason. That is a large, mechanical, but
genuinely invasive change for a gap this module's own docstring had
already scoped and accepted as narrow -- and even after paying that
cost, a residual remains (two sibling branches with byte-identical
governing text, e.g. two separately-written if cond: blocks with the
identical condition spelling and an identical duplicated call inside
each) still falls back to the ordinal, unchanged from before the
attempt. Weighed against this module's own established discipline
(documented elsewhere in this same file and in AGENTS.md's "attempted
twice, reverted twice" pattern): a fix that trades a large, invasive
change and a full baseline regeneration for a narrower, not
eliminated, version of an already-accepted, already-documented residual
is not a clear improvement, so it was reverted rather than shipped.
git diff after the revert is clean (scripts/fact_field_readers.py
byte-identical to before the attempt); the full existing test suite
(73 tests across this file and its wrapper-scoping sibling) passes
unchanged. The residual stays exactly as this module's own docstring
already described it before this round: an accepted, narrow gap, now
with a concretely verified (not merely hypothetical) repro on record here
and in the PR's own review thread.
A further Codex review round found two more real gaps, both fixed.
(1) The attrgetter branch required the constructor call to be
immediately invoked (attrgetter("bases")(rec), matched via
isinstance(node, ast.Call) and isinstance(node.func, ast.Call) and
_is_attrgetter_constructor_call(node.func, ...)), missing the equally
common callback spelling entirely: sorted(records,
key=operator.attrgetter("bases")) and map(attrgetter("bases"),
records) both construct the identical getter, just hand it to another
function instead of calling it themselves -- the read still happens, on
whatever sorted/map eventually calls it with. Fixed by matching the
constructor call directly (_is_attrgetter_constructor_call(node, ...),
no outer-call requirement at all) -- the same conservative-by-design
principle every other branch here already follows: a false positive
(a constructed-but-never-called getter) costs a reviewed baseline entry,
a false negative would be silent. This also closes, as a side effect
rather than a targeted fix, the previously-documented two-step
local-variable-indirection gap (getter = attrgetter("bases");
getter(rec)) -- _is_attrgetter_constructor_call()'s own docstring
previously stated this was deliberately out of scope, "the same
x = attrgetter equivalent of _builtins_getattr_aliases()" that
attrgetter doesn't get; both that docstring and unmigrated_fact_
reader_sites()'s own attrgetter-branch comment were corrected to
describe the new, broader match instead of the narrower one they no
longer describe. One consequence worth naming: text (the read's own
bare source, previously the whole double-call attrgetter("bases")
(rec)) is now just the constructor call's own text (attrgetter(
"bases")), since that call is what's actually matched -- outer_text
is unaffected (_outermost_containing_expr() still climbs through the
immediate outer call when there is one, since ast.Call is itself an
ast.expr). Every existing attrgetter test asserting an exact key
literal for the immediate-double-call shape needed updating to the
narrower text value; the real repository baseline is unaffected (still
zero existing attrgetter reads anywhere in abicheck/).
(2) _locally_bound_names() only ever tracked a parameter as a
locally-bound name, never a nested def/class statement's own
name. def getattr(obj, name): return None followed by getattr(rec,
"bases") -- an ordinary, unrelated function definition sharing the
builtin-looking name getattr, at module scope or nested inside the
calling function itself -- was still unconditionally treated as the real
builtin, since a def/class statement's own binding target (Python's
ordinary STORE_NAME/STORE_FAST rule, the identical rule
fact_detector_misuse.py's own _def_containing_qualnames already
models for the sibling module) was invisible to this module's shadowing
check. Fixed by threading a nearest_func parameter through _locally_
bound_names()'s walk (mirroring _lexical_function_parents's identical
concept -- class bodies stay transparent, the same simplified model this
whole module already uses) and recording each def/class's own name
against whichever scope directly contains it, alongside its existing
parameter tracking.
Both fixes' new tests, plus the attrgetter-indirection test repurposed
from a negative to a positive control (its old premise -- "no local-alias
resolution for attrgetter" -- no longer holds), went into tests/
test_fact_field_readers_wrapper_scoping.py (room remained under the
architecture gate's 1200-line cap; one existing test was moved there from
test_fact_field_readers.py to keep the main file under the cap after
its own docstring updates grew it past 1200). Verified empirically: still
zero existing hits in the real repository, mypy/ruff both stayed
clean, and python scripts/check_architecture.py reports 0 errors with
both test files under the cap.
A further Codex review round found a real regression in finding 4's own fix, plus one more real gap in the module's line-based qualname model, both fixed.
(1) The class-body-transparency fix used the wrong scope concept.
Finding 4's fix threaded a nearest_func parameter, skipping class
layers the same way _lexical_function_parents deliberately does for
closure purposes -- but a method's own name does not bind into its
enclosing function/module namespace at all; it becomes a class attribute
(C.getattr), invisible to an ordinary bare-name lookup anywhere outside
the class body. Recording it against the skip-class nearest_func meant
class C: def getattr(self, name): ... made an unrelated function
elsewhere in the same module -- with no textual relationship to C at
all -- read as if it had a local getattr binding, silently excluding
its own genuine getattr(rec, "bases") call. Fixed by replacing
nearest_func with binding_scope: str | None -- the scope a bare name
actually binds into, as opposed to "the scope a closure looks up
through" -- None while directly inside a class body (nothing recorded
there at all, matching how _shadowed() never queries a class-body scope
either, since none of this module's qualname machinery models one), and
the function's own qualname once recursed into a function body (an
ordinary nested function's own name genuinely does bind into its
immediately enclosing function, unlike a method's into its class). New
tests: the reported case, and the identical class-body-transparency rule
for a nested class (not just a nested def) shadowing attrgetter.
(2) A parameter default value/annotation was attributed to the
function's own body scope, not the enclosing scope it actually evaluates
in. def f(getattr, x=getattr(rec, "bases")): ... -- Python evaluates
the default before f's own parameters exist, so this call genuinely
reads the real builtin, but _enclosing_qualnames()'s [child.lineno,
end] range covers the function's own signature line too, so _shadowed()
saw f's own (not-yet-bound) parameter getattr and wrongly excluded a
real read. A decorator needs no equivalent fix -- it sits on a line
strictly before child.lineno, already outside the function's own
range by construction. Fixed by registering each default/annotation's own
[lineno, end_lineno] range under the current (enclosing, pre-function)
qualname -- narrower than the function's own range in the ordinary case,
so the existing smallest-range-wins tie-break lets it correctly override
the function's own broader range for just those lines. A default/
annotation sharing a line with genuine function-body code (a one-liner
def f(x=getattr(rec, "bases")): return x) is a real, accepted residual
this line-based model can't distinguish further -- the same granularity
limit this function's own docstring already accepts throughout. New
tests: the reported case, a negative control confirming an ordinary body
call is still correctly shadowed, and a negative control confirming a
default nested inside a function that genuinely declares the shadowing
parameter is still correctly excluded (this fix only corrects the
function's own signature line being wrongly attributed to itself, not
shadowing in general).
Both fixes' new tests went into tests/test_fact_field_readers_wrapper_
scoping.py (room remained under the architecture gate's 1200-line cap).
Verified empirically: still zero existing hits in the real repository,
mypy/ruff both stayed clean.
A further Codex review round found two more real gaps, both fixed.
(3) No branch recognized a mapping-based field read at all.
vars(rec)["bases"]/rec.__dict__["bases"] both read the exact same
normalized legacy value rec.bases does, through the instance's own
__dict__ mapping rather than attribute-lookup machinery -- invisible
to every existing branch (ast.Attribute, getattr(), attrgetter(),
__getattribute__()), none of which is an ast.Subscript. Added as a
new branch matching a Subscript in ast.Load context with a literal
string key in FACT_BRIDGED_ATTRS, whose .value is either a vars(...)
call (gated on _shadowed() for the vars spelling, an ordinary
bare-name call the same shadowing guard every other dynamic form already
gets) or a .__dict__ attribute access (matched unconditionally -- an
attribute access, unlike a bare name, has nothing for a local binding to
shadow). A non-literal key (vars(rec)[name]) stays out of scope, the
same "no type inference" limit every other dynamic-read form here
already accepts. New tests: both positive forms, a shadowed-vars-
parameter negative control, a non-literal-key negative control, and an
unrelated-key negative control.
(4) _locally_bound_names() never visited ast.Import/ast.
ImportFrom at all. from helper import getattr then getattr(rec,
"bases") -- an ordinary import of an unrelated module's own getattr
symbol, reusing the builtin-looking bare name -- was still treated as
the real builtin, since an import statement's own binding was invisible
the same way a bare parameter/def/class name once was (findings 4-5
above, before those were fixed). Fixed by recording each imported name
(alias.asname if given, else the plain name, with the identical
first-.-segment split fact_detector_misuse.py's own import branch
already applies for an unaliased dotted import a.b.c) against
whichever scope directly contains the import statement -- with a
deliberate carve-out for the specific imports this module already
recognizes elsewhere as a genuine alias source for a real builtin/
operator symbol: from builtins import getattr/object/type, from
operator import attrgetter, and a bare import builtins/operator.
Recording one of those as a local binding here too would have made
_shadowed() see it as shadowing itself, silently breaking the very
recognition it exists to enable -- e.g. from operator import
attrgetter; attrgetter("bases")(rec) would have stopped being
recognized at all, a real regression rather than an incomplete fix. Every
other import, including an aliased from builtins import getattr as g
(recognized under the alias g, excluded the identical way), still binds
and shadows normally. New tests: an unrelated module-scope import shadow,
its aliased-import variant, the identical shadow established inside a
function body instead of at module scope, a sibling-function negative
control (the shadow in one function must not suppress detection in an
unrelated one), and four positive controls confirming every one of the
five recognized-import carve-outs (from builtins import getattr, its
aliased spelling, from operator import attrgetter, and a bare import
operator) still resolves correctly.
Both fixes verified empirically: still zero existing hits in the real
repository, mypy/ruff both stayed clean. New tests in
TestMappingBasedFieldReads and TestImportedNamesShadowBuiltinRecognition
(tests/test_fact_field_readers_wrapper_scoping.py).
A further Codex review round found one more real gap in the mapping-
subscript fix above. vars(rec).get("bases")/rec.__dict__.get(
"bases") -- the dict.get() spelling of the identical mapping read --
were both still invisible, since neither is an ast.Subscript, the only
shape the previous fix's branch matched. Fixed by factoring the shared
"is this a mapping over the instance's own __dict__" check (vars(
...)/.__dict__) out into a new _is_mapping_receiver() helper, reused
by both the existing subscript branch and a new .get()-call branch --
so the two forms can't independently drift on what counts as a
recognized mapping receiver, the same lesson this file's own earlier
entries (the attrgetter-vs-__getattribute__ alias sets, the vars/
__dict__ shadowing rules) keep re-learning about duplicated conditions.
An optional second .get() argument (the default) is accepted but not
inspected, matching how getattr()'s own third argument is treated
elsewhere in this module. New tests: both mapping-receiver forms with
.get(), a call carrying an explicit default, a shadowed-vars-
parameter negative control, an unrelated-key negative control, and a
negative control confirming an ordinary .get() call on some unrelated
object (not vars(...)/.__dict__) is never flagged merely because its
argument happens to spell a bridged field name.
Verified empirically: still zero existing hits in the real repository,
mypy/ruff both stayed clean. New tests in TestMappingGetFieldReads
(tests/test_fact_field_readers_wrapper_scoping.py).
A further Codex review round found one more real gap: _is_mapping_
receiver() only ever matched the bare literal spelling vars, not a
real alias of it. import builtins; builtins.vars(rec)["bases"] (a
qualified call through a real builtins alias) and read_map = vars;
read_map(rec).get("bases") (a plain assignment alias) were both
invisible -- the identical gap _builtins_getattr_aliases() already
closed for getattr specifically, never extended to vars. Fixed with
a new, generalized _builtins_symbol_aliases(tree, symbol,
builtins_names): the symbol-specific half of that same alias-
resolution mechanism (import-from, a plain-assignment chain resolved to
a fixed point, a qualified X.symbol chain), taking the caller's
already-resolved builtins_names as a parameter rather than re-deriving
it, so vars doesn't need a third hand-duplicated copy of the shared
import builtins/module-alias collection _builtins_getattr_aliases()
itself already owns. _builtins_getattr_aliases() itself was left
unchanged rather than refactored to share this helper -- it is already
hardened across five prior review rounds (see its own docstring), and
generalizing it risks reopening one of them for no benefit, since it
already returns exactly the builtins_names this function needs.
A real regression caught before the tests even ran, by the tests
themselves rather than a fresh review round: an aliased import of vars
(from builtins import vars as V) was still wrongly excluded, for a
different reason than the one this fix targets. _locally_bound_names's
own recognized-import carve-out (added two rounds earlier for getattr/
object/type/attrgetter) had no entry for vars at all, so from
builtins import vars as V was itself recorded as an ordinary local
binding of V at module scope — making _shadowed() see V as shadowed
by its own import statement, the exact inversion the carve-out exists
to prevent. Fixed by adding "vars" to that carve-out's recognized-name
set, alongside the new alias-resolution fix above (not a separate,
unrelated bug — the same "this import is a recognized alias source,
not a shadow" principle, just missing one more entry).
Verified against the reported qualified-call and assigned-alias repros,
an imported-alias positive control, and three negative controls (a
builtins-shadowing parameter, an aliased-import self-shadow, and an
unrelated object's own .vars() method). Zero existing hits, mypy/
ruff both stayed clean. New tests in TestVarsAliasesInMappingReads
(tests/test_fact_field_readers_wrapper_scoping.py).
A further Codex review round found one more real gap: _operator_
attrgetter_aliases()'s own assignment-chain resolution never recognized
a qualified RHS. import operator as op; ag = op.attrgetter;
ag("bases")(rec) was invisible, since _add_candidate() only ever
matched a plain ast.Name value -- the identical gap _builtins_getattr_
aliases()'s own qualified_candidates mechanism already closes for
read_attr = builtins.getattr. Fixed the same way: a qualified
X.attrgetter assignment is collected into a new qualified_candidates
list during the same walk, resolved once the walk (and therefore
operator_names) is complete -- since, unlike the plain-name chain, this
needs the complete set to know whether X really is a resolved
operator alias. The resolution order matters: operator_names's own
fixed point runs first, then the qualified resolution folds into
attrgetter_names, then attrgetter_names's own plain-name fixed point
runs last -- so a name assigned from a qualified alias can itself be
chained further (ag = op.attrgetter; ag2 = ag).
Verified against the reported repro, a further-chained variant
(confirming the qualified resolution feeds back into the existing
plain-name chain), and a negative control for an unrelated object's own
.attrgetter attribute. Zero existing hits, mypy/ruff both stayed
clean, fact_field_readers.py at 1741 lines (well under the 2000-line
hard cap). New tests in TestQualifiedAttrgetterAssignmentAliases
(tests/test_fact_field_readers_wrapper_scoping.py).
One more finding, from the next review round: the unbound-method
object.__getattribute__(rec, "bases") call branch only ever matched a
call made directly off object/type/an alias of either receiver -- it
had no notion of the method itself being lifted out to a plain name
first. read_attr = object.__getattribute__; read_attr(rec, "bases")
performs the identical unbound-method read, but the call-matching branch
requires node.func to still be an ast.Attribute (X.__getattribute__
(...)) -- once the method is assigned to read_attr, node.func is an
ast.Name, and neither this branch nor _unbound_getattribute_receiver_
aliases() (which tracks aliases of the receiver object/type, not of
the method) had anything to recognize it. Fixed with a new
_unbound_getattribute_method_aliases(tree, object_type_names), mirroring
_builtins_symbol_aliases()'s own qualified-candidate mechanism: a plain
assignment whose RHS is X.__getattribute__ for some X already in the
caller's resolved object_type_names seeds the set, then an ordinary
plain-name chain resolves further aliases of that alias to a fixed point
-- so a receiver alias (from builtins import object as O) and a method
alias (read_attr = O.__getattribute__) compose correctly together. A new
call-matching branch was added alongside the existing unbound-receiver
branch, requiring the same not _shadowed(...)/two-args/literal-string-
argument/FACT_BRIDGED_ATTRS shape the existing branches already enforce.
Verified against the reported repro, a chained-alias variant, a
receiver-alias-composed-with-method-alias variant, the type.
__getattribute__ sibling spelling, and two negative controls (a
shadowing parameter; a non-bridged attribute argument) via direct AST
reproduction before writing tests. Zero existing hits, mypy/ruff both
stayed clean, fact_field_readers.py at 1821 lines (well under the
2000-line hard cap, though headroom is narrowing -- the next finding in
this area should reassess whether a sibling module split is due before
adding more). New tests in TestUnboundGetattributeMethodAliases
(tests/test_fact_field_readers_wrapper_scoping.py).
One more finding, from the next review round: the mapping-subscript
branch's ast.Load-only restriction missed the augmented-assignment
shape entirely, the identical gap the dedicated ast.Attribute-target
AugAssign branch already exists to close for the plain-attribute
form. rec.__dict__["bases"] += values / vars(rec)["bases"] += values
both read the field's existing value before combining it with the
right-hand side, but Python marks an AugAssign target ast.Store
regardless of its shape -- the target Subscript node is still visited
independently by ast.walk (it's a child of the AugAssign), but its
Store context skips the ordinary, Load-only Subscript branch too, so
nothing caught it. Fixed with a new ast.AugAssign branch matching a
Subscript target with a literal string key naming a bridged attribute,
gated on _is_mapping_receiver(node.target.value) -- mirroring the
existing attribute-target AugAssign branch exactly, including keying
the finding on the target Subscript node itself (not the whole
AugAssign statement) so its site/text line up with an ordinary
subscript read at the same position.
Verified against both mapping spellings (rec.__dict__[...]/
vars(rec)[...]) and three negative controls (a non-bridged key; a
plain, non-augmented overwrite, which genuinely never reads and correctly
stays unflagged; a non-mapping receiver) via direct AST reproduction
before writing tests. Zero existing hits, mypy/ruff both stayed clean,
fact_field_readers.py at 1845 lines -- still under the 2000-line hard
cap, but headroom has narrowed enough (155 lines) that the next finding
in this area should split a sibling module before adding more, per the
prior round's own note. New tests in
TestAugmentedAssignmentThroughMappingReceivers (tests/
test_fact_field_readers_wrapper_scoping.py).
Two more findings from the next review round, plus the sibling-module
split the prior round's own note said was due. (1) _locally_bound_
names()/_enclosing_qualnames() deliberately don't model an ast.Lambda
as its own scope at all (see either function's own docstring on why this
module's design stays coarser than fact_detector_misuse.py's) -- a
lambda's body shares its enclosing function's qualname, so a lambda's
own parameter was never recorded as bound anywhere _shadowed()'s
qualname-based check could see. lambda getattr, rec: getattr(rec,
"bases") -- an ordinary, unrelated lambda parameter reusing the
builtin-looking name -- was still treated as the real builtin. Rather
than widening the qualname/scope machinery itself (a materially larger
change touching three functions' worth of established, narrower-by-design
modeling), _shadowed() now also walks the call's own true AST ancestry
(via the already-available parents map) checking every enclosing
ast.Lambda's own parameters directly -- exact by construction, so it
can never misattribute a shadow to a call genuinely outside the lambda,
even one sharing the same line/qualname the coarser qualname model would
conflate them under (verified with a dedicated negative control: a
genuine getattr call textually outside the lambda, in the same
function, still gets caught). _shadowed()'s own parameter type widened
from ast.Call to ast.expr, since it only ever consults .lineno and
walks parents -- neither Call-specific -- and the new bare-ast.Name
use site from finding (2) needs the wider type too. (2)
_is_mapping_receiver() only ever matched vars(rec)/X.__dict__
directly at the point it inspects an expression -- fields = vars(rec);
fields["bases"] / fields = rec.__dict__; fields.get("bases") were both
invisible once the mapping was stored in an intermediate variable first,
the same "no alias tracking" gap this module's other alias helpers
already close for getattr/vars/attrgetter themselves, unclosed here
for their own result. Fixed with a new _mapping_receiver_aliases()
(the identical name-only, fixed-point assignment-chain pattern every
other alias helper in this module already uses), consulted by
_is_mapping_receiver()'s new final branch, gated on _shadowed() the
same way the direct vars(rec) form already is.
The sibling-module split (Codex/CodeRabbit finding notwithstanding --
this one was self-imposed, per the prior round's own note, once the two
fixes above pushed the file to 1975 of 2000 lines, only 25 short of the
hard cap). scripts/fact_field_readers_scope.py now holds
_enclosing_qualnames, _parent_map, _TRANSPARENT_EXPR_WRAPPER_TYPES,
_outermost_containing_expr, _locally_bound_names, and _lexical_
function_parents -- a mechanical extraction, not a redesign, mirroring
fact_detector_misuse_scope.py's own identical split from the sibling
gate: every function moved unchanged, as one contiguous block, with a
matching sys.path guard (fact_field_readers.py importing the sibling
module whether run directly or loaded as scripts.fact_field_readers by
a test that never imports check_ai_readiness.py first) verified by
running tests/test_fact_field_readers_wrapper_scoping.py in isolation,
not just as part of the full suite -- the exact scenario a missing guard
would silently pass in a full run and fail only in isolation. Registered
in scripts/CLAUDE.md's Inventory table.
Verified against both reported repros, a positive control for each (a
genuinely unrelated lambda/an unrelated dict alias must still be
caught/stay unflagged), a chained-mapping-alias variant, and negative
controls for shadowing and a non-bridged key, via direct AST reproduction
before writing tests -- including confirming the closure-shadow case
(def outer(getattr): return (lambda rec: getattr(rec, "bases"))(None))
and a nested-lambda shadow both still resolve correctly with the new
ancestor-walk check running ahead of the pre-existing qualname-based one.
Zero existing hits, mypy/ruff both stayed clean,
fact_field_readers.py back down to 1615 lines after the split (well
under the 2000-line hard cap, real headroom restored) plus the new 410-line
fact_field_readers_scope.py. New tests:
TestLambdaParametersShadowDynamicReaders and
TestMappingReceiverAliasesResolveThroughLocalNames in tests/
test_fact_field_readers_wrapper_scoping.py.
Two more findings from the next review round, plus a second test-file
split once the second push crossed the 1200-line cap outright. (1)
Neither the mapping-.get() branch nor the subscript branch matches
vars(rec).__getitem__("bases") (the explicit dunder-method spelling of
the identical subscript read) or operator.getitem(vars(rec), "bases")
(the standard-library callable spelling) -- both read the exact same
normalized legacy value already caught elsewhere. Fixed with two new
sibling branches: a __getitem__ call on a mapping receiver, and a
getitem call through a real operator module alias (operator_names,
already resolved by _operator_attrgetter_aliases() for attrgetter's
own qualified form -- reused as-is, not re-derived), gated on _shadowed()
the same way every other module-qualified call in this file already is.
(2) Every alias-collection helper in this module (_builtins_getattr_
aliases, _operator_attrgetter_aliases, _unbound_getattribute_receiver_
aliases, _builtins_symbol_aliases, _mapping_receiver_aliases) is
deliberately name-only and whole-tree, with no notion of which scope a
given alias was actually recognized in -- and _locally_bound_names()'s
own recognized-import carve-out (the mechanism that keeps a genuine alias
import like from operator import attrgetter as ag from being wrongly
treated as a shadow of itself) simply omits such an import from its
bound dict entirely, rather than recording anywhere that it was
recognized. _shadowed()'s outward closure walk had nothing to stop it
at the scope the alias was actually resolved in, so it kept walking past
that scope to search an enclosing one -- and if that enclosing scope
happened to bind the same bare name to something completely unrelated
(from helper import ag at module scope, an ordinary, unrecognized
import), the walk wrongly treated that unrelated binding as a shadow of
the genuinely-resolved inner alias. from helper import ag at module
scope, from operator import attrgetter as ag inside f, ag("bases")
(rec) inside f -- a real field read -- was invisible. Rather than
threading real per-scope alias resolution through all five of those
name-collection helpers (the kind of redesign fact_detector_misuse.py's
own _imported_fact_aliases() docstring already declined for an
analogous reason -- see its "known gap" entry earlier in this doc),
_locally_bound_names() now returns a second dict alongside its
existing one: which names were recognized as a genuine alias source, per
scope, rather than discarding that information. _shadowed()'s walk now
checks this second dict at each scope it passes through and returns
False (unshadowed) the moment it finds the name recognized there,
before ever reaching an enclosing scope's own (potentially unrelated)
binding. This is a real, bounded fix rather than the declined redesign,
because it reuses the exact closure-walk mechanism _shadowed() already
has -- the only change is giving it one more signal to stop on, not
teaching every alias helper to understand scope.
Verified against both reported repros, an aliased-import variant of the
operator.getitem fix, and negative controls for each (a non-mapping
__getitem__ receiver; a shadowed operator parameter; an unrelated
outer import with no inner recognized re-import, confirming the fix
doesn't widen recognition, only stops the walk early once a real alias is
found; a genuine parameter shadow, confirming ordinary shadowing is
unaffected; a closure through a real recognized alias from an enclosing
scope with no re-import of its own, confirming the pre-existing
closure-walk behavior survives) via direct AST reproduction before
writing tests. Adding these tests pushed tests/
test_fact_field_readers_wrapper_scoping.py to 1219 of its own 1200-line
cap -- over it outright -- so its tail (the five most recent test
classes: augmented assignment through mapping receivers, lambda parameter
shadowing, mapping-receiver aliases, explicit mapping-item readers,
per-scope dynamic-reader alias resolution) was split into a new sibling
file, tests/test_fact_field_readers_later_fixes.py, mirroring the
test_fact_detector_misuse_alias_edge_cases.py precedent on the sibling
gate -- mechanical extraction, every class moved unchanged, verified both
in combination and in isolation. Zero existing hits, mypy/ruff both
stayed clean, fact_field_readers.py at 1662 lines,
fact_field_readers_scope.py at 439 lines, the wrapper-scoping test file
back down to 936 lines, the new later-fixes test file at 315 lines -- all
well under their respective caps. New tests:
TestExplicitMappingItemReaders and TestDynamicReaderAliasesResolvePer
LexicalScope, both now in tests/test_fact_field_readers_later_fixes.py.
One more finding from the next review round, on the same operator.
getitem shape just landed. The new getitem call-matching branch
required an ast.Attribute callee (operator.getitem(...) through a
resolved operator module alias) -- _operator_attrgetter_aliases()
resolves getitem's own bare-name import alias family nowhere at all,
unlike attrgetter, which already gets the identical import-seeded/
chained/qualified resolution via attrgetter_names. from operator
import getitem as gi; gi(vars(rec), "bases") was invisible. Fixed by
widening _operator_attrgetter_aliases() to also return a third set,
getitem_names, resolved by literally duplicating attrgetter_names's
own three-stage mechanism (import-seeded from from operator import
getitem [as X], a plain-assignment chain, and a qualified X.getitem
assignment once operator_names is known) against a separate getitem-
tagged qualified-candidate list, sharing this same function's
operator_names/assign_candidates collection so the two families can't
independently drift on what counts as a resolved operator alias. A new
sibling call-matching branch (bare ast.Name callee, node.func.id in
getitem_names) was added alongside the existing qualified-form branch.
A second, self-found gap surfaced while first verifying this fix
empirically, before any external review flagged it: _locally_bound_
names()'s recognized-import carve-out had no entry for from operator
import getitem, unlike its sibling attrgetter. Without that entry,
the new alias import was recorded as an ordinary local binding rather
than a recognized alias source -- making _shadowed() see the import
itself as shadowing its own later use, the exact inversion that carve-out
exists to prevent (the same class of bug the vars alias round earlier
in this file's own history already hit and fixed for an identical
reason). Caught before this even reached review, since the very first
empirical check of the direct, unaliased import form returned no site at
all -- confirming the value of reproducing every case via python3 -c
before writing tests, not just the one the review comment names. Fixed
by adding "getitem" alongside "attrgetter" in that carve-out's
recognized-name set.
Verified against the reported repro, an unaliased variant, a chained-
alias variant, and the qualified-assignment form (gi = operator.
getitem), plus negative controls (an unrelated local function with no
import; a shadowing parameter) -- all via direct AST reproduction before
writing tests, and re-verified after the carve-out fix confirmed both the
previously-broken direct-import forms and the previously-working chained/
qualified forms all resolve correctly together. Still zero existing hits,
mypy/ruff both stayed clean, fact_field_readers.py at 1720 lines
(well under the 2000-line hard cap). New tests:
TestGetitemImportAliasesResolveTheBareCallableForm in tests/
test_fact_field_readers_later_fixes.py (403 lines, well under its own
1200-line cap).
Two more findings from the next review round. (1) None of the alias
collectors ever recognized ast.NamedExpr -- only ast.Assign/
ast.AnnAssign -- so (read := getattr)(rec, "bases") and (fields :=
vars(rec))["bases"] were both invisible. Fixing this needed two separate
changes, not one, since the walrus repro packs two distinct gaps into one
expression: the alias itself (read, fields, real bindings a later,
ordinary reference should resolve through) needed a NamedExpr branch in
every alias-collecting function's own ast.Assign/ast.AnnAssign walk
(_imported_class_aliases, _builtins_getattr_aliases,
_builtins_symbol_aliases, _unbound_getattribute_receiver_aliases,
_unbound_getattribute_method_aliases, _mapping_receiver_aliases,
_operator_attrgetter_aliases -- all seven, applied mechanically since
every one of them shares the identical structural gap, not just the two
functions the finding itself named) -- and, separately, the walrus used
directly as the call's own callee or mapping receiver (as both reported
repros actually are) reads the field right there, in the very expression
that introduces the alias, which no amount of alias-table lookup can
catch since there is no later reference to look up. Fixed with a small,
targeted unwrap at the two sites the finding actually named: _is_mapping_
receiver() now unwraps a NamedExpr to its own .value before any of
its existing checks run (composing for free with every shape it already
recognizes), and the getattr-call-matching branch gained a third
alternative recognizing node.func as a NamedExpr whose own .value is
a known getattr name. (2) _locally_bound_names() modeled only a
parameter, a def/class name, and an import as a real local binding --
a for target, a with ... as target, an except ... as name, a
comprehension's own for target, and a match capture were all invisible
to it, so for getattr in funcs: return getattr(rec, "bases") (an
ordinary, unrelated loop variable reusing the builtin-looking name) was
still unconditionally flagged as a real read -- a false positive on
genuinely valid code, not a missed misuse. Fixed as one generalized
shadowing class, per the finding's own suggestion, rather than five
hand-rolled special cases: _target_bound_names() extracts every plain
name a for/with target binds (recursing through Tuple/List/
Starred nesting), and _match_pattern_captures() walks a match
pattern's own subtree for every MatchAs/MatchStar/MatchMapping
capture regardless of nesting depth. None of these five forms introduces
its own new scope (a for/with/except/match binds directly into
whatever function already contains it, and this module's own line-based
scope model already resolves a comprehension's elt to its enclosing
function, matching its established coarser granularity) -- so every
binding is recorded against the current binding_scope directly, the
same target a parameter already uses.
Verified against both reported repros for finding (1) plus a later-use
(non-inline) variant, a shadowed-parameter negative control, and an
unrelated-builtin negative control; and all five reported binding shapes
for finding (2) (for, tuple-unpacking for, with ... as,
except ... as, a comprehension target, a bare match capture, an
as-pattern match capture, and a capture nested inside a class pattern),
plus a positive control confirming an unrelated for loop doesn't
suppress a real read elsewhere in the same function -- all via direct AST
reproduction before writing tests. Still zero existing hits, mypy/
ruff both stayed clean, fact_field_readers.py at 1765 lines and
fact_field_readers_scope.py at 538 lines (both well under the 2000-line
hard cap). New tests: TestNamedExprAliasesAndInlineCallsAreRecognized
and TestLexicalBindingFormsAreTreatedAsShadows, both appended to tests/
test_fact_field_readers_later_fixes.py (558 lines, well under its own
1200-line cap).
Two more findings from the next review round, plus one declined as a
re-raise of an already-decided, documented tradeoff. (1) The lexical-
binding-forms fix's own comprehension-target handling (previous round)
was a real regression, caught by review: it recorded a comprehension's
for target against _locally_bound_names()'s coarser, function-only
qualname model -- correct for for/with/except/match (none of
which are block-scoped in Python), but a comprehension genuinely does
introduce its own new scope, so that recording shadowed every call
anywhere later in the whole enclosing function, not just calls
genuinely inside the comprehension. [x for getattr in funcs] followed
by an unrelated, later getattr(rec, "bases") in the same function was
wrongly suppressed. Fixed by reverting that one binding form out of
_locally_bound_names() entirely and handling it instead in
_shadowed()'s own real AST-ancestor walk -- the identical mechanism
already used for a lambda parameter's identical "not a scope this
module's coarser qualname model tracks" shape -- so a comprehension
target shadows exactly the calls nested inside it (elt, filters, later
generators), never anything outside it. The outermost generator's own
iterable is the one exception (mirroring fact_detector_misuse_scope.py's
identical carve-out for the same construct): it evaluates in the scope
enclosing the comprehension, before the comprehension's own target
exists, so [x for getattr in getattr(rec, "bases")] must still be
flagged. Implementing this exactly needed one more structural fact about
the AST than the analogous fix in the sibling module: a comprehension's
generators[0] holds the ast.comprehension clause object, not its
.iter directly, so the ancestor walk tracks which clause object it just
ascended through (outermost_iter_clause) across the next hop, rather
than comparing the immediate child against .iter at the point the
ListComp/etc. itself is reached -- a first attempt compared against
generators[0].iter directly and silently never matched, since the
ancestor walk's immediate child at that point is always the clause
object. (2) A lambda's own default value expression evaluates at lambda-
creation time, in the enclosing scope, before the lambda's own
parameters exist at all -- the identical def-time-vs-body-time
distinction _enclosing_qualnames's own default/annotation handling
already draws for a named def -- but _shadowed()'s lambda-ancestor
check unconditionally treated any call reached through a Lambda
ancestor as shadowed by its parameters, regardless of whether the call
came from the lambda's body or its args (default values). lambda
getattr=getattr(rec, "bases"): getattr read the real builtin in its
default, but was wrongly treated as shadowed. Fixed by checking child is
node.body at the point the walk reaches the Lambda ancestor -- exact
by construction (true only on the hop ascending directly out of the body
subtree, however deeply nested; false for every hop coming from args
instead) -- and only checking the lambda's own parameters when that holds.
(3) Declined, citing an already-documented rationale rather than
re-implementing. _builtins_getattr_aliases()'s own plain-assignment
alias resolution (read = getattr) is deliberately whole-tree/module-
wide, not per-function -- its own docstring already states this
explicitly ("Whole-tree, matching _imported_class_aliases's own scope
for the identical reason"), and _imported_class_aliases()'s own
docstring gives the reason: "every mechanism here is almost always module
level, and scanning the whole tree is the same over-approximating-is-safe
stance this module already takes elsewhere." The finding reproduces
exactly the false-positive shape that stance already, deliberately
accepts: two unrelated functions each assigning the identical local name
read to two different values (read = getattr in one, read = helper
in another) both have their own read(rec, "bases") call flagged, since
the alias name is resolved once, globally. This is the identical class of
question already decided (and declined, twice, with the identical
citation) earlier in this same PR's review history for
_imported_fact_aliases's own module-wide import-alias resolution (see
that thread's own "convergence note") -- a correct fix needs the same
kind of per-scope threading through the fixed-point alias resolution that
was already judged "a real, if narrow, redesign... not a follow-up to the
last one" there, not a bounded extension of an established mechanism.
Documented as a known, pre-existing, deliberately-accepted tradeoff
rather than re-litigated.
Verified against both reported repros for findings (1)/(2), a positive
control confirming each fix's intended shadowing case still works
(comprehension elt/filter/non-outermost-generator shadowing; a lambda
body genuinely shadowed by its own parameter), and the outermost-
generator-iterable exception itself for both a list comprehension and a
set comprehension, all via direct AST reproduction before writing tests.
Still zero existing hits, mypy/ruff both stayed clean,
fact_field_readers.py at 1829 lines and fact_field_readers_scope.py
at 556 lines (both well under the 2000-line hard cap). New tests:
TestComprehensionTargetShadowsOnlyWithinTheComprehension and
TestLambdaDefaultsEvaluateBeforeParameterShadows, both appended to
tests/test_fact_field_readers_later_fixes.py (660 lines, well under its
own 1200-line cap).
Two more findings from the next review round: one fixed, a bounded
extension of an established mechanism; one declined as needing a real
redesign rather than a bounded fix. (1) dict.__getitem__(vars(rec),
"bases") -- the unbound-method spelling of the bound
vars(rec).__getitem__("bases") form already recognized, the identical
relationship object.__getattribute__(rec, "bases") already has to
rec.__getattribute__("bases") elsewhere in this module -- was invisible.
Fixed by resolving dict's own alias family through _builtins_symbol_
aliases()'s already-generic mechanism (the same one vars itself already
uses -- dict_names = _builtins_symbol_aliases(tree, "dict",
builtins_names), no new collector needed, since dict is a real,
always-in-scope builtin in the identical "no import required" category
getattr itself is in) and adding one new call-matching branch mirroring
the bound form's own shape, with the receiver check against dict_names
instead of _is_mapping_receiver(). Proactively verified every sibling
alias form before writing tests, per this file's own established
discipline (a prior round's own self-found regression) rather than only
the one reported repro -- the aliased-import form (from builtins import
dict as D; D.__getitem__(...)) initially produced no site at all, traced
to the identical class of bug caught earlier in this same file's history:
_locally_bound_names()'s recognized-import carve-out had no entry for
"dict" alongside its existing "getattr"/"object"/"type"/"vars"
tuple, so the import itself was recorded as an ordinary local binding
rather than a recognized alias source, making _shadowed() see the
import as shadowing its own later use. Fixed by adding "dict" to that
tuple, then re-verified every alias shape together (bare import, aliased
import, plain-assignment chain, qualified builtins.dict) rather than
re-testing only the one that had been broken.
(2) Declined, documented as a known gap needing real redesign rather
than attempted under review pressure. class C: def getattr(self,
name): ...; value = getattr(rec, "bases") -- a class-body-level call
(not inside a method) genuinely does see an earlier same-class-body
binding via ordinary sequential class-body execution, but
_locally_bound_names() passes binding_scope=None when descending into
a ClassDef, discarding every binding made anywhere in the class body,
not just a method's own name -- so this reads as an unshadowed real
getattr call and is wrongly flagged. Investigated a same-round fix and
found it structurally unsound, not merely inconvenient: this module's
_enclosing_qualnames() gives class-body-level code no distinct qualname
of its own at all (it inherits whichever qualname encloses the class
statement, confirmed directly -- the reported repro's own call resolved
to qualname "<module>"), which is exactly why binding_scope=None was
chosen for class bodies in the first place (a documented, deliberate
fix, see _locally_bound_names()'s own "That 'directly, syntactically
contains it' scope is NOT simply the nearest enclosing function" section):
recording a class-body binding (in particular a method's own name)
against the same qualname that also covers code genuinely outside and
after the class statement would leak it there too -- class C: def
getattr(self, name): ... followed by a real, unrelated
getattr(rec, "bases") after the class definition, at the same outer
scope, would then be wrongly excluded, reintroducing a worse version of
the exact bug that None was chosen to prevent. A correct fix needs a
genuinely distinct, position-based class-body scope (mirroring how a
comprehension's own scope was just handled above via _shadowed()'s
AST-ancestor walk rather than the qualname model) -- and, unlike the
comprehension case, get it order-sensitive too, since class-body
execution is sequential top-to-bottom, ordinary code (a call before the
shadowing def must still see the real builtin) -- exactly the
"materially larger change than the guard conditions this module has
added incrementally so far" _locally_bound_names()'s own docstring
already names as a known, deliberately-unattempted gap for a
structurally identical reordering problem. Recorded here rather than
attempted as a reactive same-round patch.
Verified against the reported repro, the aliased/bare-import/plain-
assignment/qualified alias forms, and negative controls (a dict
parameter shadow; an unrelated mapping-receiver argument) for finding
(1), via direct AST reproduction before writing tests. Still zero
existing hits, mypy/ruff both stayed clean, fact_field_readers.py
at 1859 lines and fact_field_readers_scope.py at 558 lines (both well
under the 2000-line hard cap). New tests:
TestUnboundDictGetitemMappingReaders, appended to tests/
test_fact_field_readers_later_fixes.py (728 lines, well under its own
1200-line cap).
Still not landed: no detector (diff_layout.py/diff_types.py/
diff_param_qualifiers.py/the reader set the check above now tracks
precisely) has actually been migrated to read .status — the check above
only guards the existing sites against a new, unreviewed one
joining them; it does not change what any of them do.
Migrating a detector now would add real complexity for zero behavior
change until every producer's own construction is at least this explicit
— landed as of the second slice for the five fields this phase scoped —
deferred deliberately, not silently, per this plan's own "vertical slice,
not flag day" discipline: each slice is a primitive the rest of Phase 0
builds on, landed and tested on its own rather than held until every
consumer migrates too.
Three more fact-field-readers findings (same review round), all
bounded extensions of already-shipped alias/constructor-recognition
mechanisms — fixed, one after chasing a real self-inflicted regression
this same class of bug had already produced twice. (1) dict.get(vars
(rec), "bases") — the unbound-method spelling of the already-recognized
bound vars(rec).get("bases") form, the identical relationship the
already-shipped unbound dict.__getitem__ branch has to its own bound
sibling. A straightforward mirror of that branch, reusing dict_names
unchanged. (2) operator.attrgetter("bases.foo") — a dotted attrgetter
path. The existing docstring's blanket "no type inference" framing for a
dotted argument was overbroad: the first dotted component is read
directly off the literal string (field.partition(".")), no inference
needed, while a later component genuinely would need to know the
runtime type of what the first component reads — so only the first
component is matched, keeping the "no type inference" limit exactly
where it actually applies. Verified a multi-argument call
(attrgetter("size_bits", "bases.foo")) still reports only the real
FACT_BRIDGED_ATTRS hit, and that a bridged name appearing in a
non-first component (attrgetter("foo.bases")) stays correctly out of
scope. (3) operator.itemgetter("bases")(vars(rec)) — the attrgetter-
shaped constructor spelling of a subscript read, previously entirely
untracked (no itemgetter_names alias family existed at all). Added by
widening _operator_attrgetter_aliases()'s return to a 4-tuple
(itemgetter_names resolved the identical import-seeded/chained/
qualified way attrgetter_names/getitem_names already are) and a new
_is_itemgetter_constructor_call()/_itemgetter_matched_name() pair
mirroring the attrgetter versions — but with one deliberate difference:
unlike attrgetter's "match wherever constructed, regardless of how the
getter is later used" stance, itemgetter is matched only at the
outer, immediate call, gated on _is_mapping_receiver() the same way
every other subscript-reading form here already is. The reason the two
forms can't share one stance: attrgetter("bases")(x) reads x.bases
for any x, so there's no narrower receiver shape to require, while
itemgetter("bases")(x) reads x["bases"], exactly as legitimate for an
arbitrary unrelated mapping as for an instance's own vars()/
__dict__ — an ungated constructor-wide match would have been far
noisier than its dict.get/operator.getitem siblings.
A real self-inflicted regression, caught by proactively verifying every
sibling alias form before writing tests rather than only the one repro —
the third instance of this exact bug class this session (a new alias
family missing its own entry in _locally_bound_names()'s recognized-
import carve-out tuple, fact_field_readers_scope.py). The bare-name
forms (from operator import itemgetter, and its own as alias) both
returned [] — a genuine false negative, not merely an incomplete
positive check — because the carve-out tuple gating which imports are
treated as a real alias source rather than an ordinary shadowable local
binding still only listed ("attrgetter", "getitem") for the operator
module; itemgetter was never added alongside them. Without the
carve-out entry, _locally_bound_names() recorded the bare itemgetter
import as an ordinary local binding, and _shadowed() then read it back
as shadowing itself — every bare-name itemgetter call was wrongly
excluded. Root-caused (not guessed) by adding temporary trace prints at
each condition in the new matching branch, isolating the exact point
where _shadowed(...) returned True for an unshadowed name, rather
than assuming which of the several new conditions was at fault. Fixed by
adding itemgetter to the tuple (now ("attrgetter", "getitem",
"itemgetter")) and updating the docstring paragraph that enumerates the
carved-out spellings to match. Re-verified every sibling form afterward
(bare, aliased, operator-qualified, an aliased operator module, a
plain-assignment alias of the operator module, a qualified-assignment
alias) rather than stopping at the one repro that first surfaced the gap.
A file-size consequence, handled the established way rather than
reactively. Adding the itemgetter machinery pushed fact_field_
readers.py to 2019 lines — over the AI-readiness gate's 2000-line hard
cap. Rather than trim content, the _operator_attrgetter_aliases()/
_is_attrgetter_constructor_call()/_attrgetter_matched_name()/
_is_itemgetter_constructor_call()/_itemgetter_matched_name() block
(self-contained — no dependency on anything else in the module besides
bare ast) was moved to fact_field_readers_scope.py as a second,
later block, exactly mirroring how that sibling module's first block
was split out originally; the module's own docstring was extended to
record the second move. fact_field_readers.py dropped to 1701 lines,
fact_field_readers_scope.py grew to 896 — both comfortably under the
cap. Verified with the full existing 176-test suite plus 33 new
positive/negative-control tests (three new classes —
TestUnboundDictGetMappingReaders, TestAttrgetterDottedPaths,
TestItemgetterMappingReaders — appended to tests/test_fact_field_
readers_later_fixes.py, 974 lines, well under its own 1200-line cap),
mypy/ruff format/ruff check all clean on both touched scripts,
check_architecture.py and check_ai_readiness.py both at 0 errors, and
check_docs_contract.py unchanged at its two pre-existing warnings.
One more finding on the same round's itemgetter fix: only a single
constructor argument was ever inspected. operator.itemgetter("foo",
"bases")(vars(rec)) returns a getter that reads both requested keys as
a tuple — real, documented itemgetter behavior, and the identical
multi-key shape attrgetter's own recognition already handles — but the
new itemgetter branch's own len(node.func.args) == 1 guard (and
_is_itemgetter_constructor_call()'s matching len(node.args) == 1)
silently missed a bridged key riding alongside an unrelated one. Fixed by
widening _is_itemgetter_constructor_call() to len(node.args) >= 1
(mirroring _is_attrgetter_constructor_call()'s identical bound) and
restructuring the itemgetter branch from the single-attribute elif chain
into its own top-level case — the same reason the attrgetter branch
itself is a top-level case rather than folded into that chain: it now
iterates every constructor argument and reports each bridged, literal
string-constant key independently, rather than fitting into a chain
shaped for exactly one attr per matched node.
Verified against the reported multi-key repro, both keys bridged
(reporting both independently), the bare (non-qualified) spelling,
a no-bridged-keys negative control, a non-mapping-receiver negative
control, and a shadowed-operator-parameter negative control, via direct
AST reproduction before writing tests. The full existing single-key
suite (dict.get, dotted attrgetter, single-key itemgetter, and every
other already-shipped reader form) re-verified unaffected. Still zero
existing hits, mypy/ruff both stayed clean, fact_field_readers.py
at 1733 lines and fact_field_readers_scope.py at 901 lines (both well
under the 2000-line hard cap). New tests:
test_inspects_every_key_in_a_multi_key_getter,
test_reports_each_bridged_key_independently,
test_multi_key_bare_spelling_still_recognized,
test_ignores_a_multi_key_getter_with_no_bridged_keys, appended to
TestItemgetterMappingReaders in tests/test_fact_field_readers_later_
fixes.py (1014 lines, well under its own 1200-line cap).
Two more fact-field-readers findings (next review round), both real,
both bounded extensions of already-shipped mechanisms. (1) _shadowed()'s
comprehension-generator handling blanket-checked every generator's own
target against a call reached through any generator's iterable -- correct
for the outermost generator (already excluded entirely, since it evaluates
before any target exists) but wrong for a non-outermost one: [x for x in
xs for getattr in getattr(rec, "bases")] -- the second generator's own
iterable evaluates before that same generator's own target is bound, the
identical binding-order rule the outermost generator's iterable already
gets, just one level less special-cased. Fixed by generalizing the single
outermost_iter_clause tracked during the ascent into
comprehension_gen_clause/comprehension_via_iter, identifying which
generator (by index, via node.generators.index(...)) and which part of
it (.iter, evaluating before that generator's own target; .ifs,
evaluating after) the call was reached through, then shadowing only
against the generators that are actually already bound at that point: none
for the outermost iterable, generators strictly before the current one for
a non-outermost iterable, the current generator inclusive for a filter, and
every generator for the comprehension's own final elt/key/value
(reached with no intervening generator clause at all). (2) operator.
itemgetter("bases")(vars(rec)) was recognized only at the point of
immediate construction-and-call -- get = operator.itemgetter("bases");
get(vars(rec)), storing the constructed getter in a variable first before
calling it, is ordinary, common Python that was silently missed. Fixed
with a new _itemgetter_alias_keys(), tracking every local name bound via
a plain ast.Assign to the result of an itemgetter-constructor call,
mapped to that call's own literal keys -- deliberately not chased through
a further plain-name alias (get2 = get, the same "no type inference
beyond one hop" limit already accepted elsewhere in this module) and
dropped entirely for a name assigned more than once anywhere in the file
(ambiguous by the second assignment, so guessing which one a later call
used would risk fabricating a false positive rather than merely missing a
true one).
Both pre-existing tests that pinned the exact gaps these fixes close were
corrected rather than left pinning a now-fixed bug -- the fourth and fifth
instances of this exact pattern this plan has now recorded.
test_still_shadows_within_a_non_outermost_generators_iterable used a
repro where the shadowing target was the call's own generator's target
(self-shadowing, the bug), not an earlier generator's target (the
positive control its own docstring actually claimed to be testing) --
corrected to the earlier-target repro, with a new sibling test class,
TestComprehensionGeneratorBindingOrder, exhaustively covering all four
binding-order cases (a generator's own iterable, its own filter, the final
element expression, and a third generator confirming the rule generalizes
past two). test_does_not_match_a_getter_constructed_but_not_immediately_
called asserted == [] for exactly the aliased-itemgetter repro this
round's second fix now correctly detects -- renamed and rewritten to
assert the real, single hit, with a new sibling test pinning the
one-hop-only chased-no-further limit explicitly rather than leaving it
implicit.
Verified against both reported repros, every binding-order permutation
listed above, the reassigned-variable and chained-second-name negative
controls for the itemgetter alias case, and every existing sibling form
(dict.get, dotted attrgetter, direct-call itemgetter, multi-key itemgetter)
re-verified unaffected, all via direct AST reproduction before writing
tests. Still zero existing hits, mypy/ruff both stayed clean,
fact_field_readers.py at 1809 lines and fact_field_readers_scope.py at
966 lines (both well under the 2000-line hard cap). New tests:
TestComprehensionGeneratorBindingOrder (4 tests) and
test_detects_a_getter_assigned_to_a_variable_before_being_called/
test_does_not_chase_an_aliased_getter_through_a_second_name/
test_ignores_a_reassigned_getter_variable in
TestItemgetterMappingReaders, all in tests/test_fact_field_readers_
later_fixes.py (1113 lines, still under its own 1200-line cap).
A follow-up Codex review on the same commit found _itemgetter_alias_keys()
walked only ast.Assign -- get: object = operator.itemgetter("bases")
(an annotated assignment) and (get := operator.itemgetter("bases"))
(a named expression/walrus, bound and referenced later, not the immediate-
self-call shape) construct and bind the identical getter through a
different Python binding statement, and both were silently missed:
unmigrated_fact_reader_sites() returned no site for either form, so the
ERROR-level gate could be bypassed by simply spelling the same alias
through AnnAssign/NamedExpr instead of Assign. Fixed by generalizing
_itemgetter_alias_keys()'s walk to the identical three-branch shape
_mapping_receiver_aliases() (fact_field_readers.py) already uses for
this exact purpose -- a shared _record(target, value) helper called from
ast.Assign, ast.AnnAssign, and ast.NamedExpr branches, preserving the
existing ambiguity rule (a name assigned more than once anywhere in the
tree is dropped entirely, regardless of which binding forms produced the
multiple assignments).
Verified against both reported repros (AnnAssign, NamedExpr-then-later-call),
a multi-key variant of each, the bare (non-qualified) itemgetter spelling
combined with each, the original plain-ast.Assign case as a regression
check, and the reassigned-variable and non-mapping-receiver negative
controls for both new forms -- all via direct AST reproduction before
writing tests. One further shape was checked and found out of scope for
this finding: (get := operator.itemgetter("bases"))(vars(rec)), an
immediate self-call on the walrus expression itself (no later reference to
get at all), stays undetected -- distinct from the reported "store then
call later" pattern the review actually named, and from the already-handled
operator.itemgetter("bases")(vars(rec)) immediate-construction-and-call
form (whose func is the Call node itself, not a NamedExpr wrapping
one); left as an unreported, narrower residual rather than folded into this
fix's scope. Still zero existing hits, mypy/ruff both stayed clean,
fact_field_readers_scope.py at 989 lines (well under the 2000-line hard
cap). New tests split into a dedicated sibling file,
tests/test_fact_field_readers_itemgetter_binding_forms.py (9 tests, two
classes: TestItemgetterConstructorAliasedThroughAnnAssign/
TestItemgetterConstructorAliasedThroughNamedExpr), rather than appended to
test_fact_field_readers_later_fixes.py, which had only ~87 lines of
headroom left under its own 1200-line cap.
The "left as an unreported, narrower residual" shape above turned out not
to stay a residual: a follow-up Codex review round named it directly and it
turned out to be a bounded, mechanical extension of an already-established
sibling mechanism, not exotic indirection worth declining.
(get := operator.itemgetter("bases"))(vars(rec)) -- the outer call's own
func is the ast.NamedExpr itself, not a Call, so neither the
immediate-construction-and-call branch (which requires node.func to
literally be the constructor Call) nor the plain-Name alias branch (which
requires node.func to be a bare Name) ever matched, and
unmigrated_fact_reader_sites() returned no site -- letting a bridged-field
read bypass the ERROR-level gate. The review pointed at the exact right
precedent to mirror: this module already handles the identical shape for
getattr ((read := getattr)(rec, "bases"), checked against the walrus's
own .value -- what is actually being called -- not its .target). Fixed
with a new branch matching isinstance(node.func, ast.NamedExpr) and
isinstance(node.func.value, ast.Call) and _is_itemgetter_constructor_call
(node.func.value, ...), inspecting every constructor argument the same
multi-key way the immediate-construction-and-call branch already does, and
gated by _shadowed() against _itemgetter_matched_name(node.func.value)
-- the same "is the itemgetter/operator name itself locally shadowed at
this point" check the immediate-call branch already applies, just against
the walrus's wrapped call rather than node.func directly.
Verified against the reported repro, a multi-key variant, the bare
(non-qualified) itemgetter spelling, a no-bridged-keys negative control, a
non-mapping-receiver negative control, a shadowed-operator-parameter
negative control, and every existing sibling form re-verified unaffected
(the walrus-then-later-call form, the plain-ast.Assign/AnnAssign forms,
the immediate-construction-and-call form with no assignment at all, the
sibling getattr walrus-callee mechanism, and the still-undetected
two-hop-alias-chain residual staying correctly out of scope), all via direct
AST reproduction before writing tests. Still zero existing hits, mypy/
ruff both stayed clean, fact_field_readers.py at 1858 lines (well under
the 2000-line hard cap). New tests: TestItemgetterConstructedAndCalledThrough
AWalrusCallee (6 tests), appended to
tests/test_fact_field_readers_itemgetter_binding_forms.py (235 lines,
plenty of headroom under its own 1200-line cap -- the natural home for this
fix, unlike the two files already tight on headroom).
Separately, this same round brought PR #921's branch up to date with
main, closing a CI architecture step failure (abicheck/contract_
evidence.py:86: model -> policy is forbidden, a compare -> model ->
policy -> compare dependency cycle) that was never this PR's own doing.
The branch was 53 commits behind main; main had already merged a fix
for the identical cycle (PR #931,
fix/contract-evidence-model-policy-cycle) before this round started, so
merging main in (a clean merge, no conflicts) was sufficient -- no
independent architecture fix was needed on this branch. Verified with
python scripts/check_architecture.py reporting 0 error(s) both on the
merge commit and, separately, on main itself before merging.
A further Codex review round read the gate's own error message as
promising a recognition mechanism that does not exist, and was declined
as a detection-logic change while accepted as a wording fix. The
finding: "when migrated code first checks rec.bases_fact.status and
then reads rec.bases, unmigrated_fact_reader_sites() still returns
the legacy read... the gate needs to recognize an applicable
sibling-status guard before reporting the legacy read (or change the
contract and diagnostic to require reading the Fact value instead)."
Investigated both halves of that either/or separately. Recognizing "a
preceding, applicable sibling-status guard" correctly needs genuine
control-flow analysis this scan deliberately doesn't have (a plain,
position-blind AST walk, per this module's own docstring) --
what counts as "preceding" (same branch? any earlier line in the same
function? does it need to dominate every path to the read?), what
counts as an "applicable" check (equality against a specific
FactStatus member? a truthiness test? membership?), and whether the
guard's branch is even the one containing the read, are all genuine
design questions with no existing precedent to model after -- and
grep -rn "_fact\.status\b" abicheck/ confirms zero real call sites in
the codebase exercise this pattern today, matching the plan doc's own
already-recorded "Still not landed: no detector... has actually been
migrated to read .status" note two entries above. Building real
control-flow recognition for a pattern nothing in the tree uses yet,
under review pressure, risks exactly the "attractive nuisance" this
plan's own established discipline warns against -- a heuristic that
could produce false negatives (masking a genuinely unguarded read that
merely happens to share a function with an unrelated status check
elsewhere) for a benefit no real caller currently needs.
The second half -- "change the contract and diagnostic" -- was the real,
actionable gap: the message's previous wording ("either migrate this
reader to check .status first, or add its stable key to
KNOWN_UNMIGRATED_READERS") is genuinely ambiguous between "replace the
legacy-field read with a Fact[...]-sibling read" (the actual intended
migration path, which naturally stops matching this scan's own
attribute-read pattern once done, needing no new recognition logic at
all) and "add a status check immediately before the still-present
legacy read" (the reading Codex's finding took, which this scan can
never honor without the unbuilt control-flow analysis above). Tightened
the message to state the "no control-flow analysis, a preceding check
does not exempt the read" contract explicitly, so a developer reading
it cannot come away expecting a recognition mechanism the scan doesn't
have. Verified the new wording against a real preceding-.status-check
repro (confirming it still, correctly, fires) and against the existing
end-to-end violation-reporting test, and confirmed no existing test
pinned the old wording (grep-checked across every test_fact_field_
readers*.py file) before changing it. Still zero existing hits,
mypy/ruff both stayed clean, fact_field_readers.py at 1863 lines
(well under the 2000-line hard cap). New tests split into a dedicated
small sibling file, tests/test_fact_field_readers_status_check_
diagnostic.py (2 tests), rather than appended to
test_fact_field_readers.py, which had only ~24 lines of headroom left
under its own 1200-line cap.
A second static gate, closing a gap this plan's own generated code left
open — landed. abicheck/model/fact.py's own Fact class docstring
states, in the present tense: "A detector reads a Fact[...]-typed field
only by inspecting .status ... The guard against comparing two
Fact[...] values directly inside detector logic (rather than unwrapping
first) is a static check, not a runtime one — see
scripts/check_ai_readiness.py's fact-detector-misuse check." No such
check existed when that sentence was written in the first slice — a
docstring describing a gate the codebase doesn't have yet is exactly the
kind of drift AGENTS.md's adr-status-sync/generated-file-ownership
checks exist to catch for other artifact kinds, just not one either of
them scans for here. scripts/fact_detector_misuse.py (registered as
fact-detector-misuse, mirroring fact_field_readers.py's own
extraction) is that check: an AST scan for an ==/!= comparison where
either side is a <attr>_fact field access or a
Fact(...)/Fact.<classmethod>(...) constructor call — Fact[T]
deliberately doesn't override __eq__ (poisoning the containing
dataclass's own generated equality would be worse), so this comparison
doesn't raise; it silently falls back to structural dataclass equality
over status/value/diagnostics together, which can answer True or
False without ever checking whether either side is PRESENT — exactly
the ambiguity Fact[T] exists to make unrepresentable, reintroduced by a
different spelling. Verified empirically (by running the real scan
against the whole package) to have zero existing hits under abicheck/
today — the only real matches for this pattern anywhere in the repo are
in tests/, asserting a constructed Fact equals an expected one, which
is legitimate assertion code outside this check's abicheck/-only scope
— so this check ships with no baseline at all, unlike
fact-field-readers's allowlist-and-shrink KNOWN_UNMIGRATED_READERS:
any match is an unconditional error. Tests: tests/
test_fact_detector_misuse.py (the real repository has zero violations;
the AST primitive's own contract — both attribute-pair and
constructor-call shapes detected for == and !=, a chained comparison's
each adjacent pair caught independently, is/is not and an unrelated
attribute/method name ignored; and end-to-end cases confirming the check
function fires on a new violation and stays silent on .status-based
unwrapping).
A Codex review round found a real gap in the name-only matching, fixed
in the same PR. The scan recognized x.bases_fact == y.bases_fact
directly but not the same misuse laundered through an ordinary local
variable: old_fact = old.bases_fact followed by old_fact == new_fact
has two bare ast.Name operands, and a single node can't answer whether
a name is Fact-typed without knowing which function it belongs to — the
identical scope question fact_field_readers.py's own qualname-keyed
baseline already had to solve. Fixed with _fact_aliases(): a
per-function dict[qualname, set[name]] built from two name-only
sources, a simple single-target assignment from a recognized Fact-typed
expression, and a function parameter whose own annotation is Fact[...]
(or bare Fact) — both scoped to their own enclosing function so an
alias in one function can't leak into an unrelated same-named local in a
sibling. Deliberately conservative in the over-approximating direction
(an aliased name is trusted for the whole function, not narrowed to the
lines after its assignment) — a false positive here is far cheaper than
the false negative it closes, and this check has no control-flow
analysis to narrow it correctly anyway. Verified empirically to have zero
existing hits, so the check still has no baseline. New tests: aliasing
through a local assignment, two Fact[...]-annotated parameters compared
directly, an alias not leaking across sibling functions, and an ordinary
non-Fact local left alone.
A second Codex round found the alias tracking itself stopped one hop
too early, fixed in the same PR. first = rec.bases_fact; second =
first; second == other launders the misuse through a second ordinary
assignment: second's own RHS is the bare ast.Name first, which
_is_fact_typed_expr doesn't recognize (only an attribute access or a
constructor call), so a single pass over assignments stopped at first
and never learned second was an alias too. Fixed by resolving the
per-function alias set to a fixed point: collect every simple
single-target assignment as a (name, value) candidate first, then
repeatedly add any candidate whose value is either directly Fact-typed
or is itself a Name already known as an alias in that same function,
until a pass adds nothing new — bounded by construction (finitely many
candidates, each pass adds at least one or the loop stops). New test:
test_detects_a_comparison_through_a_chained_alias.
A third Codex round found two more real gaps, both fixed in the same
PR. (1) old_fact: Fact[list[str]] = old.bases_fact is an
ast.AnnAssign, a distinct node type the candidate collection (ast.
Assign-only) never matched at all — the ordinary annotated-assignment
spelling bypassed the gate entirely. Fixed by collecting AnnAssign too:
its own annotation is an unconditional signal on its own (mirroring the
function-parameter case — a bare old_fact: Fact[...] with no value is
still Fact-typed), and when it isn't Fact-typed but a value is present,
the assignment still joins the ordinary fixed-point candidate pool. (2)
fact = rec.bases_fact in an outer function, then def inner(): return
fact == other — inner's own qualname has no assignment establishing
fact, but it's a real, visible closure variable there; a lookup scoped
strictly to the exact qualname missed it. Fixed by processing qualnames
in shallowest-first order (by dot-count — a real approximation of "outer
scopes before inner ones", not exact Python scoping since a class body
isn't actually a closure scope for its methods, but the same
over-approximating-is-safe direction this whole module already takes)
and seeding each qualname's known-alias set with its lexical parent's
already-resolved set before running the fixed point over its own
candidates. A real bug surfaced while implementing this: a qualname with
no candidates or annotations of its own (like inner in that example)
was never added to the dict, so its own lookup silently saw nothing
rather than its parent's set — fixed by unioning in every qualname
_enclosing_qualnames actually produced, not just ones with a candidate.
Re-verified the existing sibling-non-leakage guarantee still holds (an
alias in one function must not leak into an unrelated, non-nested sibling
sharing a parameter name) with a dedicated test alongside the two new
ones. Verified empirically: still zero existing hits under abicheck/
today. New tests: an annotated local assignment, a bare annotated local
with no value, a closure over an outer alias, and the sibling-leakage
negative control restated for this fix.
A fourth Codex round found the closure-inheritance fix itself was
wrong for one real shape, fixed in the same PR. That fix derived a
qualname's "lexical parent" by string-splitting the dotted qualname
(rsplit(".", 1)) — correct for a plain nested function, but wrong for a
class nested inside a function: fact = rec.bases_fact in an outer
function, then class C: def method(self): return fact == other still
closes method over fact in real Python (a class body isn't a closure
scope, but the function wrapping it still is), yet the dotted qualname
"f.C.method" splits to "f.C" — a synthetic scope no real function
owns, so it was never itself processed or seeded, silently breaking the
chain there. Fixed with _lexical_function_parents(): walks the tree
directly, tracking each function's nearest enclosing function
separately from the dotted-name prefix (skipping over any intervening
class layer), giving the real Python closure-scope chain instead of one
reconstructed from a string that conflates class and function nesting.
Qualnames are now processed in order of true scope-nesting depth (walking
this parent map) rather than dot-count, so a parent is still always
resolved before a child consults it. Verified empirically: still zero
existing hits. New test:
test_detects_a_comparison_through_a_closure_over_a_class_nested_method.
A CodeRabbit finding on the same round: an import alias of Fact
itself, not just the fields on RecordType/Param, was invisible too --
fixed in the same PR. from abicheck.model.fact import Fact as F then
F.present(a) == F.present(b) is the identical misuse as Fact.
present(a) == Fact.present(b), but _is_fact_typed_expr/_is_fact_
typed_annotation both hard-coded the literal bare name "Fact". Fixed
with _imported_fact_aliases() -- collects every local name bound to
Fact via from ... import Fact as F (matched by imported name alone,
not by source module, the same name-only stance fact_field_readers.
py's own import-alias helpers already take) -- threaded as a new
fact_names parameter through both functions and every call site.
Verified empirically: still zero existing hits. New tests: an aliased
constructor-call comparison, an aliased annotation comparison, and a
negative control for an unrelated import merely sharing the alias name.
A fifth Codex round found the closure-inheritance fix itself produces a
real false positive, not a missed detection -- fixed in the same PR.
Every prior round in this section closed a false-negative gap (misuse
that should have been flagged but wasn't); this one is the opposite
direction, which this module's usual "a false positive here is cheaper
than a false negative" stance does not cover -- a false positive means
rejecting valid code as a hard CI error. fact = rec.bases_fact in an
outer function, then def inner(fact, other): return fact == other --
inner's own fact parameter is an ordinary, unrelated local that merely
reuses the name; Python's scoping makes it local to the whole inner
function (shadowing the outer alias throughout, not just after some
reassignment point), but the closure-inheritance fix's known |=
aliases.get(parent, set()) unconditionally unioned the parent's alias set
into every child scope, with no way to exclude a name the child rebinds
itself. Fixed by tracking locally_bound: every name a function binds on
its own -- every parameter (Fact-typed or not) and every simple assignment
target, collected in the same walk that already builds aliases and
candidates -- and subtracting it from the inherited set before seeding:
known |= aliases.get(parent, set()) - locally_bound.get(qualname,
set()). Verified empirically: the shadowed-parameter case now correctly
reports no misuse, the real (non-shadowed) closure case from the fourth
round is unaffected, and the real repository still reports zero
violations. New tests:
test_ignores_a_parameter_that_shadows_an_outer_fact_alias and
test_ignores_a_reassigned_local_that_shadows_an_outer_fact_alias (the
latter pinning that the shadowing rule applies to a plain reassignment,
not just a parameter, and that it covers the whole function body, not
only the lines after the reassignment).
Review convergence reached on this PR; two further findings documented
as known gaps rather than fixed (Codex review, both after the above
fix). Posted as a PR comment: six review rounds on this file have each
closed a real, bounded gap in the same "name-only, no type inference"
alias/closure-resolution machinery; going forward, a further round
finding yet another indirection on the same "is this expression
Fact-typed" question is the same inherent no-type-inference residual
fact_field_readers.py's own docstring already accepts, not a specific
miss worth chasing indefinitely -- documented in the module rather than
fixed, per this repo's own review-convergence guidance. Two findings
landed exactly in that category and are recorded in
_imported_fact_aliases's own docstring rather than fixed here: (1) the
import-alias set this function builds is module-wide rather than scoped
to the function the ImportFrom sits inside, so a (highly unusual)
per-function from ... import Fact as F could leak its alias name into
an unrelated sibling binding of the same short name; (2) only a bare from
... import Fact as F is recognized -- a module-qualified constructor call
(fact_model.Fact.present(...)) is invisible, since the constructor-call
match assumes func.value is a bare ast.Name, not an arbitrary
ast.Attribute chain. Both would need real per-scope import tracking or
arbitrary-attribute-chain resolution respectively -- their own scoped
redesigns, not follow-ups to the fixes already in this file.
A further Codex review round found two more real false positives on
the shadowing fix's own machinery -- both fixed, since a false positive
here (rejecting valid code) is never treated as a convergence-eligible
finding regardless of how many rounds precede it. (1) locally_bound
only ever recorded a bare ast.Name assignment target and a function's
own parameters, so fact, other = pair (ordinary tuple-unpacking) inside
a nested function shadowing an outer fact = rec.bases_fact alias was
never recorded as a local binding -- the nested function's own fact
still read as the outer alias, flagging a valid fact == other. Fixed
with _bound_names(), a small recursive helper unpacking ast.Tuple/
ast.List/ast.Starred targets, wired into locally_bound for
ast.Assign (all targets, not just the single-Name case candidates
stays restricted to), plus the other real Python local-binding forms the
same review comment named: a for/async for loop target, a with/
async with as target, and an except ... as name: handler name.
(2) _enclosing_qualnames/_lexical_function_parents both keyed a
function purely by its dotted name, so two functions sharing one bare
name -- the shape a @typing.overload-decorated stub and its real
implementation share -- collapsed onto the same qualname key, and
_fact_aliases merged their alias/candidate data: a stub's x:
Fact[int] parameter leaked into a same-named real implementation's own,
unrelated x local, flagging a valid x == other. Fixed by folding each
def's own line number into its qualname (f"{prefix}{child.name}#
{child.lineno}") in both helpers identically -- safe here in a way it
would not be in fact_field_readers.py's identical-looking helper: this
module's qualname is purely internal (fact_equality_misuse_sites returns
only (lineno, col_offset), never a qualname-derived key), so there is no
external format to keep stable. Verified empirically: still zero existing
hits in the real repository, and a positive control confirms two
independent same-named functions are still each checked for real misuse
on their own (neither silently swallows the other's genuine violation).
Six new tests: tuple-unpacking, for, with, and except-as shadowing,
plus a two-independent-same-name-functions-both-flagged positive control
and the @overload-shaped negative control itself.
A further Codex review round found two more real false positives, both
fixed -- neither a lambda nor a comprehension, nor a class body,
introduced a scope of its own before this fix, so each leaked whatever
alias happened to be in scope around it straight through. (1) A
class-body-level assignment (fact = rec.bases_fact written directly in
a class body, not inside a method) was attributed to whatever function
enclosed the class, since only a nested FunctionDef contributed a scope
range of its own -- a ClassDef contributed none. A class body is
actually its own namespace (an ordinary name assigned there is a class
attribute, visible only as self.x/Class.x, never as a bare name to
the enclosing function), so this let a class-body assignment masquerade
as a real local of the enclosing function, and an unrelated, later
fact == 1 elsewhere in that function (a genuine module-global fact)
was flagged as misuse. Fixed by giving each ClassDef its own scope
range in _enclosing_qualnames (f"{prefix}{child.name}#{child.lineno}
<class-body>"), the identical mechanism a FunctionDef already gets --
a nested method's own, narrower range still overrides it for the
method's own lines. Nothing ever looks this qualname up as a lexical
parent (_lexical_function_parents only ever produces function-def-
derived keys), so it correctly has no effect on anything outside the
class body -- while a genuine misuse within the class body itself is
still caught, since the class body is now a real scope of its own rather
than a black hole. (2) A lambda parameter and a comprehension's own
for target were likewise never recorded as local bindings, since
neither introduced a scope of its own either -- fact = rec.bases_fact
outer, then (lambda fact: fact == other)(1) or [fact == other for
fact in values], each shadow the outer alias with an unrelated local
exactly the way a nested def's parameter or a for-loop's own target
already does, but the shadow went unrecognized. Fixed the identical way
a def already is: both ast.Lambda and each comprehension kind
(ListComp/SetComp/DictComp/GeneratorExp) now get their own scope
range in _enclosing_qualnames (disambiguated by line and column,
since several could share one line) and their own entry in
_lexical_function_parents -- a lambda/comprehension is a real Python
closure over its enclosing scope, exactly like a nested def, so it
becomes the new nearest_func for anything nested inside it too, not
just a leaf scope. _fact_aliases's own binding-collection walk now
records a lambda's parameters (reusing the identical FunctionDef/
AsyncFunctionDef branch, since ast.Lambda.args shares the same
ast.arguments shape -- a lambda parameter can never carry an
annotation, so the Fact-typed-annotation check on it is always a
harmless no-op) and a comprehension's own for target(s) (via the
already-existing _bound_names() recursive-unpacking helper). Verified
empirically: still zero existing hits in the real repository, and three
positive controls confirm a genuine misuse inside a lambda, inside a
comprehension, and directly inside a class body are each still caught.
Six new tests: the class-body leak and its positive-control counterpart,
the lambda shadow and its positive control, the comprehension shadow and
its positive control.
A further Codex review round found three more real gaps -- one a
structural correctness bug in the lambda/comprehension fix itself (a
false negative, not the false positives every round so far had been),
the other two small, bounded extensions -- all fixed. (1) Resolving a
scope by line alone, not by exact position, could hide a genuine
misuse, not merely flag a spurious one. fact = rec.bases_fact outer,
then (lambda fact: fact == other)(1); return fact == other on one
line: the SECOND fact == other (the real outer alias, textually
sharing the lambda's line but not part of it at all) was silently
misattributed to the lambda's own shadowing scope, since the previous
fix's scope map was still keyed dict[int, str] -- one qualname per
physical line -- so whichever scope happened to be inserted last for
that line (the lambda) won for the entire line, including code that
was never part of the lambda's body. This is more serious than the
false positives every prior round in this section fixed: those rejected
valid code (a review cost); this one let a real Fact[T] misuse pass
silently, the exact failure this whole check exists to prevent. Fixed by
switching _enclosing_qualnames from a line-keyed dict[int, str] to a
list of exact ((start_line, start_col), (end_line, end_col), qualname)
spans, resolved by a new _qualname_at() -- the smallest span whose
(start, end) lexicographically brackets a query (lineno, col_offset)
position, not merely the smallest span sharing its line (correct because
every span this module produces nests strictly inside its lexical
parent's own span, a laminar family, so comparing by (end_line -
start_line, end_col - start_col) -- line span first, column span only
as a same-line tiebreaker -- always picks the correctly-nested one).
Every one of the eight call sites that previously did qualnames.get(node.
lineno, "<module>") now does _qualname_at((node.lineno, node.
col_offset), qualnames). Verified against the review's own exact repro:
the shadowed fact == other inside the lambda is correctly NOT flagged,
while the real outer fact == other on the same line now IS. (2) first
= second = rec.bases_fact -- a chained assignment gives every plain-name
target the identical RHS value, unlike a tuple-unpacking target, but the
candidate collector was restricted to exactly one target. Fixed by
looping over every target and registering a candidate for each
plain-Name one, all sharing the same RHS -- a tuple/list target among
the same chain still contributes nothing here (only locally_bound, via
the pre-existing _bound_names() path), since it still has no single
value of its own. (3) (fact := rec.bases_fact) == other -- an inline
assignment expression (ast.NamedExpr), invisible to _is_fact_typed_
expr (which never unwrapped it) and to the alias tracking (which only
ever walked plain assignment statements, never an expression embedded
inside a larger one). Fixed with two changes: _is_fact_typed_expr now
unwraps a NamedExpr to its own .value, exactly as Fact-typed as its
RHS; and _fact_aliases's binding-collection walk registers the
walrus's target as an ordinary alias candidate too, so a later reuse of
the bound name (if (fact := rec.bases_fact) is not None: return fact ==
other) is also caught, not just an inline use at the assignment
expression's own site. Deliberately not chasing PEP 572's own
scope-hopping rule for a walrus used inside a comprehension (which
binds into the comprehension's enclosing scope, not its own) -- an
accepted, narrow residual, since qualnames has no notion of that rule
and the walrus alias is registered under whatever scope its own position
resolves to. Verified empirically: still zero existing hits in the real
repository. Four new tests: the exact same-line lambda/outer-alias repro
(with an explicit column check pinning which of the two identical-text
fact == other occurrences was reported), a chained assignment, an
inline walrus comparison, and a walrus alias reused on a later line.
A further Codex review round found two more real gaps, both in the
walrus/parameter machinery this section had just built -- both fixed.
(1) The walrus fix's own "deliberately not chasing PEP 572's
scope-hopping rule" residual was, on inspection, backwards -- a real
false positive, not merely an accepted narrow gap. The previous round
exempted every walrus target from locally_bound on the theory that
its scope-hopping rule always applies; that rule is real, but only fires
when the walrus sits directly inside a comprehension -- everywhere
else, a walrus binds to its immediately enclosing scope exactly like an
ordinary assignment. Unconditionally exempting it meant an entirely
ordinary, comprehension-free rebinding (fact = rec.bases_fact outer,
then def inner(other): (fact := 1); return fact == other) failed to
shadow the outer alias, flagging valid code. Fixed by hopping only the
genuine comprehension case out to its real PEP 572 binding scope (the
nearest enclosing non-comprehension scope, walking lexical_parents --
moved earlier in the function so this same walk can use it -- through
every nested comprehension layer, not just the innermost one) and
treating every other case as an ordinary local binding, added to
locally_bound the same as any other assignment target. (2) A
parameter's own default value can be Fact-typed, and nothing examined
it. fact = rec.bases_fact; def inner(fact=fact): return fact ==
other -- calling inner() with no override genuinely runs the
comparison against the outer Fact value (a default is evaluated once, in
the enclosing scope, at def/lambda time), but the parameter was
already unconditionally excluded from the inherited alias set regardless
of what its own default was. Fixed by pairing args.defaults/
kw_defaults with their parameters (positional defaults right-align
against posonlyargs + args; kw_defaults pairs positionally with
kwonlyargs, None for one with no default) and resolving each pending
default in a dedicated pass after the fixed point has already
stabilized every scope's own alias set -- a default expression must be
checked against its enclosing scope's final alias set, not the
function's own (which excludes this exact name by construction), so this
can't be folded into the same single fixed-point pass every other alias
source already uses. Verified empirically: still zero existing hits in
the real repository, and positive/negative controls confirm a walrus
alias used without shadowing, a walrus hopping out of a comprehension,
and an ordinary (non-Fact) parameter default all still behave correctly.
Five new tests: the walrus-shadow false positive and its walrus-hops-out
positive control, a default referencing an outer alias, a directly
Fact-typed default, and an ordinary-default negative control.
A further Codex review round found two more real gaps -- one a false
positive in the same shadowing family as the class-body/lambda/
comprehension fixes above, the other a false negative in the same
"attributed to the wrong scope" family as the default-value fix just
above -- both fixed. (1) Structural pattern matching's own capture
forms were never recorded as local bindings. case fact: (a bare
ast.MatchAs capture), case [*fact]: (ast.MatchStar), and case
{**fact}: (ast.MatchMapping's own rest) are all real local bindings,
exactly like a for loop target or an except ... as name: handler, but
nothing collected any of them -- a nested function's own case fact:
failed to shadow an outer fact = rec.bases_fact alias, flagging a valid
fact == other comparison against the captured (arbitrary) matched
value. Fixed with a new _match_pattern_names(), recursively collecting
every bound name from a pattern (also descending into MatchSequence/
MatchMapping/MatchClass/MatchOr, since a capture can nest inside
any of them), wired into a new ast.Match branch in the binding-
collection walk -- match/case introduces no scope of its own in
Python, so every case's captures are attributed to the match
statement's own position. (2) A walrus inside a parameter's own
default expression was attributed to the wrong scope. def inner(x=
(fact := rec.bases_fact)): -- Python evaluates a default expression in
the enclosing scope (the same rule the previous round's default-value
fix already relies on), but the generic, position-based NamedExpr
handling resolves a query position to the smallest span containing it
-- and a default expression is textually part of the def/lambda's own
span, so the walrus was misattributed to inner's own body scope, not
the scope its value is actually usable in. A later, genuinely outer
fact == other therefore saw no alias for fact at all. Fixed by
explicitly walking each default expression (args.defaults/
kw_defaults) inside the FunctionDef/AsyncFunctionDef/Lambda
branch that already handles defaults, registering any embedded
NamedExpr directly in the enclosing scope (lexical_parents[
qualname]) as an ordinary local binding -- and recording each one's
id() in a new default_walrus_ids set so the generic, position-based
NamedExpr branch skips it rather than also (mis)processing it a second
time. Verified empirically: still zero existing hits in the real
repository, and positive controls confirm a genuine misuse reached
through a match capture (no shadowing) is still caught. Five new tests:
MatchAs/MatchStar/MatchMapping shadowing, a positive control for a
real match-capture misuse, and the default-expression-walrus repro.
A further Codex review round found three more real gaps, all in this
same shadowing/scope-attribution family, all fixed. (1) nonlocal/
global were treated identically to an ordinary local rebinding for
shadowing purposes -- wrong, since neither introduces a new local at
all. def outer(rec): fact = rec.bases_fact; def inner(): nonlocal
fact; hit = fact == other; fact = 1 -- the fact = 1 reassignment inside
inner correctly makes fact a target the existing shadowing machinery
records in locally_bound[inner], but nonlocal fact means this is the
same variable as the outer alias, not a fresh local one -- so the read
on the line before the reassignment should still see the outer alias and
be flagged, and it wasn't. The identical gap applies to global. Fixed
with a new ast.Global/ast.Nonlocal branch recording each declared
name against its own qualname, followed by a post-walk pass subtracting
every declared name from that same qualname's locally_bound set --
after every ordinary binding source has already been collected, so a
genuinely different local reassignment in the same function (not
declared nonlocal/global) is unaffected. (2) A parameter default's own
alias never propagated to a function nested inside the one declaring the
default. def inner(rec, fact=rec.bases_fact): def nested(): return
fact == other -- the previous round's default-value fix resolves
pending_defaults in one pass after the depth-ordered inheritance/
candidate fixed point has already run to completion, so nested's own
inheritance from inner was computed before inner's own default-derived
alias for fact existed at all, and nested never saw it. Fixed by
wrapping both the depth-ordered inheritance/candidate pass and the
pending-defaults resolution together in one outer fixed-point loop (while
outer_changed: ...), so a second iteration through inheritance sees
inner's now-resolved fact and propagates it to nested exactly the
way an ordinary parent alias already would -- guaranteed to terminate,
since every alias set only ever grows over a finite universe of
(qualname, name) pairs. (3) A class body's own top-level code did not
inherit from its lexically enclosing scope at all. def outer(rec):
fact = rec.bases_fact; class C: result = fact == other -- a class body
is ordinary code that inherits from its enclosing scope like any other
(it's only invisible to a method defined inside it, since Python's LEGB
rule skips the class layer specifically for a nested function's own
closure), but _lexical_function_parents() had no entry for a class-body
qualname at all, so it inherited nothing and the read was missed. Fixed
by adding a class_qualname -> nearest_func entry to the same parents
dict the class-body walk already builds, without disturbing the existing,
correct method-skips-its-class-body behavior (a method gets its own,
independent qualname entry via the unchanged FunctionDef branch).
Verified empirically: still zero existing hits in the real repository,
and mypy/ruff both stayed clean (one incidental fix along the way --
the new pending-defaults loop variable had to be renamed off target,
since that name was already inferred as ast.expr from an unrelated
for target in node.targets: loop earlier in the same function, and
Python's lack of block scoping meant reusing it produced a real mypy type
error, not just a style nit). Nine new tests: the nonlocal/global
exemption and its negative control (an undeclared local reassignment
must still shadow), the nested-function default-propagation repro, the
class-body-inherits repro and its negative control (a class body with no
enclosing Fact alias), and a test pinning that a method still does not
see its own class body's local reassignment of the same name -- it sees
straight through to the enclosing function's real alias instead.
A further Codex review round found three more real gaps in this same
family, all fixed. (1) A class body's own LOAD_NAME lookup is
statement-order-aware, unlike every other scope's LOAD_FAST, and the
previous round's shadowing fix didn't account for it. fact = rec.
bases_fact outer, then class C: hit = fact == other; fact = 1 --
Python resolves fact dynamically at each class-body statement against
whatever the class namespace holds so far, not statically pre-determined
by "is this name assigned anywhere in this class body" the way a
function's LOAD_FAST is -- so hit's read, occurring before the
later fact = 1 reassignment, genuinely sees the outer alias in real
Python, but the existing shadowing subtraction (whole-scope, position-
blind, correct for every other scope) excluded it entirely, missing a
real misuse. This module has no statement-order-aware lookup of its own
(building one would mean turning _fact_aliases's per-scope answer into
a per-position one, a materially larger change), so the fix takes the
same over-approximating-is-safe direction the module already argues for
the opposite case (nonlocal/global): a class body's own local rebinding
now never shadows what it inherits at all, gated on the <class-body>
qualname suffix _enclosing_qualnames already produces. The accepted
cost is symmetric to the finding this closes: a class body that
reassigns fact to something ordinary before using it now reads as a
false positive -- the same "false positive is far cheaper than the false
negative it closes" trade-off this module states throughout, just landing
on the opposite scope this time. (2) A nested def/class statement's
own name was never recorded as a local binding in the scope that
contains it. def fact(): ... then fact == other in the same
containing scope -- Python's LOAD_FAST/LOAD_NAME resolves fact to
the function object just defined, an ordinary local, but the existing
branch only ever recorded the nested scope's own parameters, never the
definition's name in its container -- so an outer Fact alias of the
identical bare name was never shadowed, flagging valid code. Fixing this
correctly needed a new, dedicated helper rather than reusing the
position-based _qualname_at lookup every other binding site already
uses: a scope-introducing node's own span always starts at its own
(lineno, col_offset), so querying that exact position resolves to the
node's own smallest containing span, not its parent's -- the wrong
answer for "what scope contains this definition." _def_containing_
qualnames() is a small, standalone recursive walk (mirroring _lexical_
function_parents's own shape, deliberately not skipping a class
layer the way that function does, since a def/class statement's binding
target is a different question from the closure-scope chain), keyed by
position rather than qualname string, consulted only by this one new
FunctionDef/AsyncFunctionDef/ClassDef branch. (3) global fact
was still routed through ordinary lexical_parents inheritance, which is
wrong for global specifically even though it's exactly right for
nonlocal. nonlocal fact genuinely means "the nearest enclosing
function's own fact" -- precisely what lexical_parents[qualname]
already gives every other inherited name, so the previous round's fix was
correct for that half. global fact means "module-scope fact, full
stop" -- it must bypass every intervening function layer's own
inheritance entirely, even one with its own unrelated, non-Fact-typed
local of the identical bare name. Routing it through ordinary inheritance
was wrong in both directions at once: an intervening function's own
unrelated local could shadow (hide) a genuinely Fact-typed module-level
name, and, symmetrically, an intervening function's own genuinely
Fact-typed local of the same bare name could be wrongly attributed to an
unrelated module-level one. Fixed by tracking global-declared names in
a separate global_declared dict (alongside the existing, still-shared
nonlocal_or_global shadowing-exemption set) and adding a dedicated step
to the outer fixed-point loop: a global-declared name is excluded from
the ordinary parent-inheritance step entirely and instead resolved
directly against aliases["<module>"]. Verified empirically: still zero
existing hits in the real repository, and mypy/ruff both stayed clean.
Seven new tests: the class-body read-before-reassignment repro, the
nested-def and nested-class shadowing repros and their shared negative
control (a real misuse still caught with no nested definition present),
and the global-through-an-intervening-local repro with its reversed-
values negative control.
A further Codex review round found two more real gaps, both fixed.
(1) A comprehension walrus was never actually marked as a local binding
in the scope PEP 572 hops it out to. The scope-hop fix (several rounds
above) computes binding_qualname -- the walrus's real PEP 572 target
scope, hopping out of every enclosing comprehension layer -- correctly,
but the locally_bound mark itself was gated on binding_qualname ==
walrus_qualname, i.e. only when NO hop occurred. That's backwards
relative to that same fix's own stated intent ("every other case" gets
the identical locally_bound treatment): [(fact := x) for x in values]
sitting directly inside inner, with an outer fact = rec.bases_fact
alias, hops fact's real binding out to inner itself under PEP 572 --
a genuine local of inner -- but with the mark skipped whenever a hop
happens, inner's own inheritance step never learned this, and a later,
real fact == other in inner (reading the comprehension's own ordinary
int, not the outer Fact alias) was wrongly flagged. Fixed by marking
locally_bound[binding_qualname] unconditionally, regardless of whether
a hop occurred -- binding_qualname is already the correct target scope
either way, so the conditional never needed to exist. (2) A global/
nonlocal-declared name's own write side was still attached to the
declaring function's own qualname, even though the read side (two rounds
above) was already correctly routed. def seed(rec): global fact; fact
= rec.bases_fact genuinely writes module-scope fact -- but the
candidate this assignment produces was recorded under seed's own
qualname regardless, so a sibling function reading the identical
module-level fact through ordinary inheritance (not its own global
declaration) never saw it as Fact-typed at all, missing a real misuse.
The identical gap applies to nonlocal. Fixed with a new
_declared_target_scope() helper and a dedicated post-walk pass
(alongside the existing shadowing-subtraction pass, for the identical
reason: a global/nonlocal statement can appear anywhere in its
function's body, even after the assignment it governs, so this can't be
decided candidate-by-candidate during the single forward walk) that
redirects every candidate and every direct alias recorded under a
declaring qualname to the scope the assignment actually writes to --
<module> for global, the nearest enclosing function for nonlocal.
Verified empirically: still zero existing hits in the real repository,
and mypy/ruff both stayed clean (the same "variable name already
inferred as a different type elsewhere in this function" mypy trap this
file's history has hit twice before required renaming two local variables
away from target/declared). Six new tests: the comprehension-walrus
shadowing repro and its negative control (a real misuse still caught with
no comprehension walrus present), the global-write-visible-to-a-sibling
repro and its negative control (an ordinary, undeclared local assignment
must not leak into the module's own alias set), and the identical
write-side routing test for nonlocal.
A further Codex review round found two more real gaps, both fixed.
(1) The write-side routing fix's own candidate-move approach broke on a
same-writer-scope RHS indirection. global fact; local = rec.bases_fact;
fact = local -- fact's own candidate value is the bare ast.Name
local, which is only ever a real local of the writer's own scope
(seed), never of <module>. The previous round's fix moved the raw
(name, value) candidate tuple straight into the target scope's own
candidate list, where the inner fixed point then checked local against
the target's own alias set -- local was never going to appear there,
silently breaking the exact RHS indirection this whole write-side fix
exists to close. Fixed by no longer moving candidates at all: a declared
assignment's candidate stays in its writer's own scope, resolving
normally there (so local resolves against seed's own alias set, same
as any other same-function alias). A new step, run each outer-fixed-point
iteration right after a scope's own inner candidate resolution, then
checks whether a global/nonlocal-declared name has become confirmed
Fact-typed within its writer's own scope this iteration and, only then,
propagates it into the scope the assignment actually writes to -- so
fact (confirmed via local, within seed) propagates to <module>
the same outer iteration, with no separate resolution context to drift
out of sync with the writer's own. (2) A comparison inside a
parameter's own default or annotation resolved against the function's
own body scope, not the enclosing scope it actually evaluates in. fact
= rec.bases_fact; def inner(fact=(fact == other)): ... -- Python
evaluates a default at def-time in the enclosing scope (the same
binding-timing rule the earlier default-embedded-walrus fix already
relies on), but fact_equality_misuse_sites()'s own site-to-qualname
resolution is purely position-based, and the comparison's own position is
textually inside inner's span -- so it was checked against inner's
own alias set, where inner's own parameter fact has already removed
the inherited alias (the shadowing fix, several rounds above), missing a
comparison that at the point it actually runs still reads the outer
alias. Fixed with a new _default_and_annotation_scope_overrides(),
walking every parameter default and annotation expression's own subtree
and mapping each descendant node's id() to the function's lexical
parent -- consulted by fact_equality_misuse_sites()'s own ast.Compare
handling before falling back to the ordinary position-based lookup.
Verified empirically: still zero existing hits in the real repository,
and mypy/ruff both stayed clean. Four new tests: the same-scope-RHS-
indirection repro for both global and nonlocal, the default-
comparison repro, and a negative control (a default comparison against
an unrelated, non-Fact value must not be flagged).
A further Codex review round found two more real gaps, both fixed --
plus a file-size split this round's own new tests triggered. (1) The
nonlocal write-side routing fix always targeted the immediate lexical
parent, but nonlocal can skip multiple enclosing functions. Python
resolves nonlocal fact to the nearest enclosing function scope that
actually binds the name itself, not merely the one right above -- outer
binds fact, middle (nested in outer) never touches it at all,
setter (nested in middle) does nonlocal fact; fact = rec.bases_fact
-- this genuinely writes outer's fact, skipping middle entirely,
but the previous, immediate-parent-only routing published the write to
middle instead, where nothing reads it (a sibling of middle,
reader, never saw it). Fixed by walking outward from the immediate
parent, at each step checking whether that ancestor's own locally_
bound set actually contains the declared name -- locally_bound is
already exactly the right test, since it already excludes a name an
ancestor only holds via its own nonlocal/global declaration (the
shadowing-exemption subtraction from several rounds above), so the walk
naturally continues past an ancestor whose own binding is itself
borrowed from further out, the identical case Python's own resolution
skips. (2) An import statement was never recorded as a local
binding at all. import json as fact or from pkg import item as fact
inside a nested function shadows an inherited Fact alias exactly like any
other assignment form, but the binding collector had no ast.Import/
ast.ImportFrom branch, so a nested function's own import-bound fact
never shadowed an outer alias. Fixed with a new branch mirroring every
other binding-form branch's own shape, splitting an unaliased dotted
import (import a.b.c) on its first . to match Python's own
import-binding rule (only the top-level package name binds in the
importing scope without an explicit as). Verified empirically: still
zero existing hits in the real repository, and mypy/ruff both stayed
clean. Four new tests: the multi-level nonlocal-skip repro, both import
forms' shadowing repros, and a negative control (a real misuse still
caught with no import shadowing present).
The four new tests above pushed tests/test_fact_detector_misuse.py
past the architecture gate's 1200-line test-file cap (1224 lines) --
split the same way test_mutation_run_scoping.py already did for an
identical reason (see tests/CLAUDE.md's own note on that split). The
whole TestFactEqualityMisuseSites class had grown into two genuinely
distinct halves: the core misuse-detection contract (direct attribute
reads, getattr/constructor-call recognition, alias chains, annotated
assignments) and, by far the larger half after fourteen-plus rounds of
scope-attribution findings, every shadowing/scope-resolution test
(parameter/local/comprehension/lambda/walrus/match-capture shadowing,
closure inheritance through nested functions and class bodies,
nonlocal/global read- and write-side routing, parameter-default/
annotation scope resolution, import-based shadowing). Split the second
half out into a new sibling file, tests/test_fact_detector_misuse_
scoping.py (TestFactEqualityMisuseSitesScoping), leaving the original
file at 273 lines and the new one at 997 -- both comfortably under the
cap, with zero test behavior change (all 82 tests, the same set as
before the split, still pass).
A further Codex review round found four more real gaps, all fixed --
one in annotation recognition, three in scope resolution. (1)
_is_fact_typed_annotation only ever recognized a bare Fact or a
subscript whose own value was Fact -- an Optional[Fact[T]]/
Union[Fact[T], None]/PEP 604 Fact[T] | None wrapper hid the misuse
underneath it entirely. def f(value: Fact[int] | None, other): return
value == other produced no finding at all. Fixed by making the function
genuinely recursive over the three wrapper shapes: an ast.BinOp with
ast.BitOr unwraps both operands (X | None), an Optional[...]
subscript unwraps its slice, and a Union[...] subscript unwraps every
element of its slice tuple (or the bare slice, for a single-argument
Union[X]) -- each via the same fact_names-keyed leaf check a bare
Fact already used, so Fact[int] nested arbitrarily deep inside any
combination of these still resolves. A new _annotation_head_name()
helper recognizes the wrapper's own name whether spelled bare
(Optional[...]) or module-qualified (typing.Optional[...]). (2)
A default expression containing its own nested lambda/comprehension
scope had every one of its descendants force-attributed to the enclosing
scope, including nodes that actually evaluate inside that nested
scope. With an outer fact = rec.bases_fact, def f(cb=lambda fact:
fact == other): ... was rejected even though the lambda's own parameter
fact genuinely shadows the outer alias for the comparison inside the
lambda's own body -- the blanket ast.walk(subtree) walked straight
through the lambda's own boundary, overriding the inner Compare node's
scope to the outer function's, discarding the shadowing the lambda's
own already-correct position-based resolution would have given it. Fixed
with a new _iter_default_subtree() walker: it still yields every
descendant (so the override map is unaffected for the common,
no-nested-scope case), but stops expanding past any node that itself
introduces a scope (FunctionDef/AsyncFunctionDef/Lambda/any
comprehension) -- that boundary node is still yielded (harmless, since
only an ast.Compare id is ever looked up), just never descended into,
leaving its own body to ordinary position-based resolution, which already
has a real span and qualname for it. (3) node.returns -- a return
annotation -- was omitted from the scope-override traversal entirely,
even though Python evaluates it in the defining (enclosing) scope exactly
like a parameter default or a parameter annotation: fact = rec.
bases_fact; def inner(fact) -> (fact == other): ... was missed, since
inner's own parameter fact shadows the outer alias for the function
body, but the return annotation evaluates before that parameter even
exists. Fixed by appending node.returns (when present -- ast.Lambda
has no such attribute) to the same subtrees list defaults/parameter
annotations already populate. (4) A method's own parameter default is
evaluated while its containing class body executes -- ordinary
LOAD_NAME class-body code, not a closure lookup -- but the
pending_defaults resolution pass looked up the default's evaluation
scope via lexical_parents, which intentionally skips the class layer
for the different question of a method body's own free-variable
lookup. class C: fact = rec.bases_fact; def m(self, value=fact):
return value == other produced no finding, since value was never
marked as a Fact alias -- lexical_parents.get("C.m", ...) skips straight
past C's own class-body scope to whatever encloses the class, which
carries no fact alias in this shape at all. Fixed by resolving the
default's own evaluation scope through _def_containing_qualnames
instead (already computed once per call inside _fact_aliases, unlike
_default_and_annotation_scope_overrides's own separate copy, since
_fact_aliases needed it first for the walrus-target-hop case several
rounds above) -- the syntactic containing scope, class layers included --
while the method body's own fact parameter (with no default at all)
still correctly shadows in its body, verified by a dedicated negative
control, since _fact_aliases's shadowing-subtraction mechanism the
class-skipping lexical_parents chain still governs is untouched by this
fix. pending_defaults grew a fourth tuple element (the def's own
(lineno, col_offset), needed since _def_containing_qualnames is
position-keyed, not qualname-keyed) to carry this through; _default_and_
annotation_scope_overrides itself was also switched from
_lexical_function_parents to _def_containing_qualnames for the
identical reason -- a comparison found directly inside a method's own
default (not just an aliased parameter later read in the body) needs the
same containing-class-scope answer. Verified empirically: still zero
existing hits in the real repository, and mypy/ruff both stayed
clean. Nine new tests: three annotation-wrapper positive cases plus a
negative control (an ordinary int | None parameter), the nested-lambda-
default exclusion plus its no-lambda negative control, the return-
annotation case, and the class-body-default case plus its
method-parameter-still-shadows negative control.
A further Codex review round found the identical nested-scope-boundary
gap in a sibling collector, one function over. _default_and_
annotation_scope_overrides()'s own Compare-attribution walk had just
been fixed (previous round) to stop at a nested lambda/comprehension
boundary via _iter_default_subtree(), but the walrus-inside-a-default
collector -- a separate loop, a few lines above in _fact_aliases(),
that publishes a default-embedded walrus target as an alias of the
enclosing scope -- still used a plain, unrestricted ast.walk(default_
expr). fact = 1; def configure(cb=lambda: (fact := rec.bases_fact)):
... -- the lambda is only created at def-time in the enclosing scope;
the walrus inside its body binds fact in the lambda's own scope only
when the lambda is later called, never the enclosing one -- but the
unrestricted walk crossed that boundary anyway, wrongly publishing the
lambda-local walrus target as a real alias of the enclosing (here,
module) scope, so a later, genuinely unrelated fact == other read past
that point was rejected. Fixed by switching this collector to the same
_iter_default_subtree() walker the sibling fix already introduced --
one shared boundary-aware primitive for both consumers, rather than a
second copy of the same fix. Verified empirically: still zero existing
hits in the real repository, and mypy/ruff both stayed clean. Two new
tests: the nested-lambda-walrus exclusion and a negative control (a
walrus directly inside a default, no nested scope in the way, must still
be published to the enclosing scope exactly as before).
A further Codex review round found three more real gaps in the same
containing-scope machinery, all fixed. (1) _def_containing_
qualnames() recorded a containing scope for a def/class statement
but never for a Lambda. A lambda has no name to bind (unlike a
def/class statement, its introduction is an expression, not a
statement) -- but its own default values still evaluate at
lambda-creation time in whatever scope directly contains it, the
identical rule a method's own default already relies on this function
for. fact = rec.bases_fact; cb = lambda x=fact: x == other inside a
function had no entry to resolve against, so both the pending-default
alias resolution and the comparison-scope override silently fell back to
"<module>" regardless of the lambda's real containing scope. Fixed by
recording containing[lambda.lineno, lambda.col_offset] = scope_qualname
in the Lambda branch, mirroring the FunctionDef/ClassDef branches
exactly. (2) A nested class's own base/keyword expressions were
attributed to the inner class's own body scope instead of the scope
that actually contains the class statement. class Outer: fact =
rec.bases_fact; class Inner(make_base(fact == other)): ... -- a base
class or metaclass keyword executes while the class Inner statement
itself runs, in Outer's own scope, never inside Inner's own (not yet
even created) body -- but _enclosing_qualnames assigns the entire
ClassDef span, bases and keywords included, to the inner class-body
scope (correct for the body's own statements, wrong for the header that
precedes them), and _default_and_annotation_scope_overrides() had no
ClassDef branch at all to correct it. Fixed by extending that function
with a ClassDef branch: walk node.bases/(kw.value for kw in
node.keywords) through the same _iter_default_subtree() boundary-aware
walker the FunctionDef/Lambda branch already uses, overriding each to
the ClassDef's own containing scope (from def_containing) rather than
its class-body scope. (3) The walrus-inside-a-default collector's own
enclosing computation still used lexical_parents, not
def_containing, even after the sibling pending_defaults resolution
and _default_and_annotation_scope_overrides() were both already fixed
to use the latter (two rounds above). A method's own default-embedded
walrus is evaluated while its containing class body executes -- ordinary
class-body code, not a closure lookup -- but lexical_parents
intentionally skips that class layer for the different question of a
method body's own free-variable lookup: fact = 1; class C: def
f(self, x=(fact := rec.bases_fact)): ... wrongly bound the walrus target
to the module's fact (skipping straight past C's own class-body
scope), so a later, genuinely unrelated module-level fact == other
read past that point was rejected, and the actual class-body-scoped
fact the walrus meant to bind was invisible to a use elsewhere in that
same class body. Fixed by switching this one remaining call site to
def_containing.get((node.lineno, node.col_offset), "<module>"),
matching its two siblings. Verified empirically: still zero existing
hits in the real repository, and mypy/ruff both stayed clean. Five
new tests: the lambda-default-inherits-containing-scope case, the
nested-class-header case, the method-default-walrus-binds-in-class-
namespace case and its positive control (the same walrus target visible
elsewhere in that class body).
A further Codex review round found a structural bug in _lexical_
function_parents() itself, distinct from every containing-scope fix
above. All of those rounds fixed which primitive (lexical_parents
vs. def_containing) a caller consults; this one is a bug in lexical_
parents's own construction. _lexical_function_parents()'s recursive
descent switched to a def/lambda's new qualname before visiting
any of its own children -- including its default values, parameter
annotations, return annotation, and decorators, all of which evaluate at
def/lambda-creation time, in whatever scope was active before that
new qualname takes over (the identical binding-timing rule _fact_
aliases()'s/_default_and_annotation_scope_overrides()'s own default/
annotation handling already relies on). fact = rec.bases_fact in f,
then def g(fact, cb=[fact == other for _ in xs]): ... -- the
comprehension executes while g is being defined (in f's own scope,
before g's own parameter fact even exists to shadow anything) and
genuinely closes over f's alias, but was wrongly parented under g
itself, where g's own same-named parameter incorrectly shadowed it.
Fixed by splitting the single recursive visit() into two cooperating
functions: dispatch() does the actual scope-introducing-node matching
(the original visit()'s if isinstance(...) chain, unchanged in
substance), while a new, thinner visit() just applies dispatch() to
every direct child of a node. This split matters because a def's own
default/annotation/decorator expressions now need to be re-dispatched
(via a new def_time_subtrees() helper, mirroring _default_and_
annotation_scope_overrides()'s own subtrees construction almost
exactly) with the old nearest_func, while only the real function body
executes with the new one -- and a subtree re-dispatch needs the full
dispatch() match logic (in case the subtree is itself a bare Lambda/
comprehension), not a plain visit() that would skip straight into its
children. Generalized to Lambda's own defaults too, not just def
(the identical Codex-reported repro reproduces for g = lambda fact,
cb=[fact == other for _ in xs]: cb), and to decorators (a decorator's
own arguments evaluate at def-time in the enclosing scope by the
identical rule, even though the reported finding didn't name that
shape) -- matching this repo's own "fix the cause, not the instance"
convention rather than patching only the literal reported repro.
A real regression was caught and fixed while implementing this, not by
a separate review round: the first version of the split reused visit()
(not dispatch()) for a function's own body statements and a lambda's
own body expression, silently breaking the plain-closure case with no
default at all. visit(stmt, qualname + ".", qualname) for each body
statement treats stmt as the node whose children get dispatched, not
as a dispatch candidate itself -- so a nested def/class/Lambda/
comprehension directly in the body (the ordinary, common shape every
closure-inheritance test in this whole file already exercises) was never
matched by dispatch() at all, and its own parents[] entry was never
recorded. Caught immediately by re-running the existing test suite before
adding new tests for this fix (test_a_plain_closure_with_no_default_is_
still_caught pins this exact regression), not by a fresh review round --
worth recording as the same "verify against the existing suite before
trusting a refactor, not just the new repro it was written for" discipline
this plan's own earlier rounds have needed more than once. Fixed by
calling dispatch(), not visit(), for both the FunctionDef/
AsyncFunctionDef body-statement loop and the Lambda body expression.
A second, smaller version of the same class of bug (kw_defaults
carrying a None element -- an ordinary "this keyword-only parameter has
no default" marker, not a bug -- reaching dispatch() unfiltered and
crashing on ast.iter_child_nodes(None)) was caught the same way, before
it ever reached CI, and fixed by filtering None out of def_time_
subtrees()'s own kw_defaults half, matching _default_and_annotation_
scope_overrides()'s own pre-existing if subtree is None: continue
guard for the identical list.
Verified empirically: still zero existing hits in the real repository,
and mypy/ruff both stayed clean. Six new tests, in a new sibling file
(tests/test_fact_detector_misuse_def_time_scope.py -- test_fact_
detector_misuse_scoping.py was already at the architecture gate's
1200-line cap, the same reason that file was split out of test_fact_
detector_misuse.py in the first place): the reported comprehension-
default-on-a-def case, the identical case on a lambda, a negative
control (a comprehension shadow genuinely inside the body, not a
default, must still be caught by the parameter), the plain-closure
regression guard, a keyword-only-default-after-a-no-default-keyword-arg
case (pinning the kw_defaults-filtering fix), and the decorator
generalization.
A further Codex review round found _def_containing_qualnames() had
the identical unconditional-visit(child, qualname + ".", qualname) bug
_lexical_function_parents() above was just fixed for, in its own
separate walk over the same tree. _def_containing_qualnames() answers
a related but distinct question (which scope a def/class statement
directly, syntactically binds into, and -- since a later slice -- which
scope a Lambda is directly, syntactically created in), used by the
pending-default alias resolution and the comparison-scope override
elsewhere in this file, but it had its own, separate recursive walk that
never received the def_time_subtrees()/dispatch()/visit() split
above. fact = rec.bases_fact in f, then def g(fact, cb=lambda x=
fact: x == other): ... -- the inner lambda's own x=fact default
evaluates while g is being defined, in f's own scope, before g's own
parameter fact even exists -- but the unconditional recursion recorded
the lambda's containing scope as g instead of f, so g's own
same-named parameter fact incorrectly appeared to shadow the lambda's
x=fact default and the misuse went undetected. Fixed with the identical
split: a second def_time_subtrees() (duplicated rather than shared --
this module deliberately keeps each scoping helper local to the function
that uses it, matching this file's own existing convention of two
independent, structurally identical copies rather than a premature shared
abstraction) plus a dispatch()/visit() split for _def_containing_
qualnames(), re-visiting a def-time subtree under the old
scope_qualname while only the real body executes under the new one.
Verified empirically: still zero existing hits in the real repository,
and mypy/ruff both stayed clean. Two new tests, appended to the
existing tests/test_fact_detector_misuse_def_time_scope.py (room
remained under the architecture gate's 1200-line cap): the reported
nested-lambda-default-inside-another-def's-default case, and a negative
control confirming a lambda genuinely created inside g's own body (not
one of g's own def-time subtrees) is still correctly parented under
g, so g's own parameter still shadows the outer alias there.
Two further Codex review rounds found two more real gaps in the same area, both real Python scoping exceptions this scoping machinery had never modeled, both fixed.
(1) A comprehension's own outermost generator's iterable evaluates in
the enclosing scope, not the comprehension's own. This is a genuine
CPython semantic: a comprehension compiles to an implicit generator
function, and only the first for's iterable is evaluated before
that function is even called (passed to it as an argument) -- the
element expression, every if filter, and every non-first for's
iterable all run inside the comprehension's own new scope, after its
own target(s) have already bound. fact = rec.bases_fact; [x for fact in
(fact == other,) for x in fact] -- the comparison sits inside the first
generator's own iterable, which runs before fact (the comprehension's
own target) exists to shadow anything, so it still reads the outer
alias. All three scoping functions (_enclosing_qualnames,
_lexical_function_parents, _def_containing_qualnames) previously
attributed the whole comprehension -- outermost iterable included -- to
its own new scope uniformly, missing this. Notably, _enclosing_
qualnames's own docstring had already predicted and explicitly accepted
this exact gap as "vanishingly rare, and not the shape any review round
has found" -- a review round then found it.
Fixed with the identical technique in all three functions, each adapted
to its own return shape: _enclosing_qualnames (which threads a
qualname string through its walk, not just a prefix, so it can name
what "the enclosing scope" actually is) registers a narrower,
independent span for the outermost iterable's own source range, tagged
with the incoming qualname rather than the comprehension's own new
one -- _qualname_at's existing smallest-span-first resolution then
picks this override for any position inside that one iterable, while the
comprehension's own broader span still covers everything else. The
whole-node walk still revisits the same iterable afterward (a real,
accepted small waste, not a bug -- since it's the identical span, the
strict < comparison _qualname_at already uses never lets a
same-size, later-registered entry displace the correct, earlier one).
_lexical_function_parents/_def_containing_qualnames (which don't
build spans at all, only qualname-keyed maps) instead re-dispatch the
outermost iterable directly under the old nearest_func/
scope_qualname, mirroring exactly how each already re-dispatches a
def/lambda's own def-time subtrees under the old scope. Verified
empirically: still zero existing hits in the real repository, and
mypy/ruff both stayed clean. New tests, in tests/test_fact_detector_
misuse_def_time_scope.py: the reported case, a negative control
confirming only the first generator is special (a genuine shadow in a
second generator's own iterable is still correctly excluded), and a
case confirming a real closure (not just a bare comparison) found in
the outermost iterable resolves correctly too.
(2) A class's own base classes, keyword arguments, and decorators all
evaluate while the class statement itself executes, in whatever scope
directly contains it -- never inside the new class's own body -- and
_def_containing_qualnames's ClassDef branch dispatched all of them
(bases, keywords, decorators, and body) uniformly under the new
class-body qualname. An earlier round had already fixed the direct-
read case for _default_and_annotation_scope_overrides's own separate
override map (class Inner(make_base(fact == other)): ...), but that fix
only widens which qualname a bare read resolves to when the read sits
directly in the header text -- it says nothing about a nested closure
(a def/lambda/comprehension/another class) found there, whose own
containing-scope entry is a completely different lookup, owned by
_def_containing_qualnames itself. class Inner((lambda x=fact:
make_base(x == other))()): ... -- the lambda's own default x=fact
evaluates in Outer's namespace (wherever the class Inner(...):
statement itself lives), not Inner's, but the unconditional dispatch
recorded the lambda's containing scope as Inner<class-body> instead,
so x was never recognized as the outer alias. Fixed by splitting the
ClassDef branch's dispatch the same way the FunctionDef/Lambda
branches already are: every base expression and keyword value (plus,
generalizing to a shape the reported repro didn't name but the identical
timing rule covers, every decorator) is re-dispatched under the
incoming scope_qualname, while only the class's own body statements
dispatch under the new class_qualname. This closes the gap for both a
direct read and a nested closure automatically, since a nested def/
lambda dispatched under scope_qualname now correctly records its own
containing[] entry against that same outer scope -- no separate change
to _default_and_annotation_scope_overrides was needed, since that
function already reads def_containing (this fix's own output) for every
def/lambda's own position, whether inside a class header or not.
Verified empirically: still zero existing hits, mypy/ruff both stayed
clean. New tests: the reported base-expression case, a metaclass-keyword
case (generalizing beyond the one reported shape), and a negative control
confirming a comparison genuinely inside the class body is still
correctly attributed there -- shadowed via a nested method's own
parameter (a mechanism already known to work) rather than a class-body
reassignment, since class-body-level reassignment shadowing is a
separate, pre-existing, unrelated gap this module doesn't track at all
(confirmed unaffected by this fix, before and after, by direct
comparison against the pre-fix code).
A further Codex review round found a real regression in the comprehension fix above, plus one more real gap, both fixed.
(1) The comprehension fix's own "harmless, same span" reasoning was
correct for _enclosing_qualnames, but wrong for its two siblings. The
first version of that fix dispatched the outermost iterable once, under
the old scope, then still finished with a blanket visit(child, ...)
over the whole comprehension -- reasoned as harmless, since the blanket
walk reaches the identical iterable a second time with the identical
span, and _qualname_at's strict size < best_size tie-break never lets
a same-size later entry replace an earlier one. That reasoning holds for
_enclosing_qualnames's own spans list, but _lexical_function_
parents/_def_containing_qualnames don't build a spans list at all --
they write straight into a plain dict (parents[qualname] = ...,
containing[pos] = ...), where a second write to the same key
unconditionally overwrites the first, no tie-break involved. [x for
fact in (lambda y=fact: (y == other,))() for x in fact] -- the lambda's
own default y=fact is correctly dispatched once, under f's own scope,
by the fix's first half; the blanket re-walk then reaches the same
lambda a second time and overwrites its correct containing[]/parents[]
entry with the comprehension's own (wrong) scope, silently missing the
misuse. Fixed uniformly across all three functions by never re-walking
the outermost iterable at all: the comprehension's own body is now walked
explicitly, field by field (elt/key+value for a DictComp, each
generator's target/ifs, and every non-first generator's own iter),
instead of through a blanket walk that would reach the already-handled
iterable again. _enclosing_qualnames itself needed a genuine dispatch/
visit split for this (it never had one before, unlike its two siblings
-- see the def-time-subtree fix earlier in this section), since dispatching
a specific field like elt as its own candidate (it might itself be a bare
Lambda, as in [lambda: x for x in y]) needs a function that matches a
given node, not visit's existing "match every child of a container"
shape. New tests: the reported lambda-in-outermost-iterable case, and a
DictComp-specific case confirming its key/value split is covered by
the same explicit-field dispatch.
(2) _fact_aliases()'s candidate collection deliberately excluded every
tuple/list-unpacking assignment target, reasoned as "has no single value
to attribute" -- true for the general case (a, b = pair, one opaque
value with no per-element sub-expression), but not for old_fact, new_fact
= old.bases_fact, new.bases_fact, where the RHS is itself a literal
tuple display of the identical length: each target element genuinely does
have its own value, the same way a chained assignment's every target
already does. Fixed with _paired_unpacking_candidates(), a recursive
helper matching a Tuple/List target against a structurally identical
Tuple/List value element-by-element (nesting through a further tuple
target the same way _bound_names() already does), returning no
candidates at all -- rather than a partial, best-effort pairing -- the
moment either side isn't a literal display, the lengths disagree, or
either side contains a Starred element (which captures an
arbitrary-length slice with no single corresponding RHS sub-expression to
pair it against). New tests: paired tuple unpacking, paired list
unpacking, nested paired unpacking, and two negative controls (the
pre-existing opaque-single-value case, still correctly excluded, and a
starred-target case confirming no candidate is derived for it without
spuriously flagging an unrelated comparison in the same function).
Both fixes' new tests went into tests/test_fact_detector_misuse_def_
time_scope.py (room remained under the architecture gate's 1200-line
cap). Verified empirically: still zero existing hits in the real
repository, and mypy/ruff both stayed clean.
A further Codex review round found two more real gaps, both direct siblings of already-fixed shapes above, both fixed.
(1) A walrus inside a class base or metaclass keyword expression had no
binding-side counterpart to the read-side fix already covering that same
header text. _default_and_annotation_scope_overrides()'s ClassDef
branch already routes a bare ==/!= comparison found directly in a
base/keyword expression to the scope containing the class statement --
but _fact_aliases()'s own walrus-binding handling (the branch that lets
(fact := rec.bases_fact) be usable later, not just recognized
inline) only ever special-cased a FunctionDef/AsyncFunctionDef/
Lambda's own default/annotation subtrees, with no equivalent branch for
ast.ClassDef at all. class C(make_base(fact := rec.vtable_fact)): ...
therefore bound fact in the not-yet-created class-body scope (the
generic, position-based NamedExpr branch's default attribution), not
the scope actually executing the class header -- silently losing an
alias a later, genuinely outer fact == other needs. Fixed with the
identical mechanism the FunctionDef/Lambda branch already uses,
mirrored into a new ast.ClassDef branch: walk every base and keyword
value through _iter_default_subtree(), register any walrus found there
against def_containing's own entry for the class statement, and mark
it via default_walrus_ids so the generic branch doesn't also
(mis)process it. One subtlety worth recording: ast.ClassDef already
matches an earlier, unrelated elif branch in the same walk (the
def/class own-name binding fix from an earlier round) -- placing the new
branch there would have made it unreachable dead code, since Python's
elif chain stops at the first match per node per loop iteration. The
new branch instead joins the second, independent if/elif chain the
FunctionDef/Lambda default-handling branch already starts (Python
allows a fresh if after an elif chain ends; the two chains run
sequentially, not exclusively, for a single node), confirmed correct by
tracing a ClassDef node's own execution path through both chains
directly rather than assuming placement.
(2) A single for/comprehension loop target bound, one iteration at a
time, to every element of a literal Tuple/List display of
Fact-typed values had no propagation at all -- the loop-target
counterpart of _paired_unpacking_candidates()'s already-fixed
assignment-side handling. for fact in (old.bases_fact, new.bases_
fact): return fact == other only ever recorded fact as an ordinary
local binding (shadowing an outer alias correctly), never as a
candidate in its own right, so the misuse this loop actually performs
went undetected. Unlike the paired-unpacking case -- where distinct
targets each pair with a distinct RHS element -- a loop target reuses
the same name across every iteration, so the alias only holds if
every element is definitively Fact-typed, not merely one of them
(for x in (rec.bases_fact, some_other_call()): must stay unflagged,
since x is only sometimes a Fact). Fixed by checking, for both ast.
For/ast.AsyncFor and a comprehension's own each-generator loop, that
the target is a bare Name, the iterable is a literal Tuple/List
display, and every element satisfies _is_fact_typed_expr() before
registering one representative element as a candidate (the fixed-point
resolver only needs one Fact-typed candidate value per name to mark it
known). Verified against the exact reported repro plus its comprehension
form ([fact == other for fact in (old.bases_fact, new.bases_fact)]),
a negative control with a mixed Fact/non-Fact tuple, a negative control
with a bare (non-literal) iterable, and a negative control confirming
the alias stays scoped to the function that actually established it
(an unrelated sibling function's own same-named parameter is unaffected).
Both fixes verified empirically: still zero existing hits in the real
repository, and mypy/ruff both stayed clean. New tests in tests/
test_fact_detector_misuse_def_time_scope.py (room remained under the
architecture gate's 1200-line cap): TestClassHeaderWalrusContainingScope
and TestForLoopLiteralCollectionAliases.
A further Codex review round found one more real gap in the loop-target
fix above. _is_fact_typed_expr() deliberately never resolves a bare
ast.Name -- answering "is this name a Fact" needs the whole-tree alias
fixed point, which doesn't exist yet during the single collection pass
that builds candidates -- so the "every element satisfies
_is_fact_typed_expr()" gate from the previous fix silently rejected an
otherwise-ordinary alias refactor: old_fact = old.bases_fact outer,
then for fact in (old_fact,): fact == other, is a real misuse (fact
genuinely holds old_fact's value, which is genuinely Fact-typed), but
old_fact is a bare name, not a structural attribute/constructor
expression, so it never satisfied the gate at collection time.
The loop-target case needed something the existing candidates
mechanism can't express: candidates is disjunctive (a name becomes
known the moment ANY ONE of its recorded values resolves -- the right
model for "this name was assigned this value, or this other value, at
different points"), but a loop target bound to every element of one
tuple needs every element to resolve before the target itself does --
a conjunctive requirement. Fixed with a new, parallel structure,
tuple_loop_candidates: dict[str, list[tuple[str, list[ast.expr]]]],
populated by the same collection-time gate as before except an element
is now accepted either because it's structurally Fact-typed (unchanged)
or because it's simply a bare ast.Name (deferred, not yet confirmed) --
an element that is neither still disqualifies the whole loop outright,
preserving the original conservative guarantee. Consulted inside the
same while changed: per-qualname fixed point candidates itself
already participates in: at each pass, a tuple_loop_candidates entry's
target becomes known once every one of its elements is either
structurally Fact-typed or a Name already present in known -- the
identical isinstance(value, ast.Name) and value.id in known resolution
candidates' own loop already performs for a single value, just
conjoined across a whole tuple instead of checked for one. Since known
persists across outer fixed-point passes and only ever grows, this
naturally converges the same way every other alias resolution in this
module does.
Verified against the exact reported repro, its comprehension form, a
mixed alias-and-direct-read tuple (both element kinds must each resolve
independently), a negative control where the bare-name element never
resolves to anything (must stay unflagged, confirming the deferred
element is only eligible, not automatically accepted), and a negative
control mixing a resolved alias with an unrelated non-Fact call
(confirming the conjunctive requirement holds regardless of which
specific element fails it) -- plus the full existing suite, confirming
the original tuple-of-direct-reads shape from the previous round is
unaffected. mypy/ruff both stayed clean, still zero existing hits in
the real repository. New tests:
TestForLoopLiteralCollectionResolvesThroughExistingAliases in tests/
test_fact_detector_misuse_def_time_scope.py.
A CodeRabbit review round found one more real gap: _fact_aliases()'s
walrus-collection loop only ever walked node.args.defaults/
kw_defaults, never a parameter's own annotation or the -> return
annotation. def inner(x: (fact := rec.bases_fact)): ... -- absent
from __future__ import annotations, Python evaluates a parameter
annotation at the identical def-time, in the identical containing scope,
a default value does -- but this loop's own construction (mirroring only
half of _default_and_annotation_scope_overrides()'s already-wider
subtrees list, this module's read-side sibling function) never
included either, so a walrus there fell through to the generic,
position-based NamedExpr branch and was misattributed to the
function's own body scope, the identical failure mode every earlier
round in this section already fixed for defaults specifically. Fixed by
widening the loop's own subtree collection to mirror _default_and_
annotation_scope_overrides()'s exactly (every parameter's own
annotation, plus returns when present) -- unconditionally, regardless
of whether the module actually has from __future__ import annotations
in effect, matching that sibling function's own already-established,
deliberately conservative choice: a walrus that in fact never executes
under postponed evaluation registering a spurious alias is a false
positive, the safe direction this module accepts throughout, rather than
adding a second, narrower postponed-annotation-detection rule that would
only complicate this loop without closing a real gap in the safe
direction. Verified against the reported parameter-annotation repro, its
return-annotation sibling, and a negative control confirming the
existing nested-scope-boundary rule (_iter_default_subtree(), already
relied on for defaults) applies identically to a lambda nested inside an
annotation. Full existing suite unaffected. mypy/ruff both stayed
clean, still zero existing hits in the real repository. New tests:
TestWalrusInAnnotationsContainingScope in tests/
test_fact_detector_misuse_def_time_scope.py.
A further Codex review round found two more real gaps in
_is_fact_typed_expr()'s and the loop-target machinery's own coverage,
both fixed.
(1) _is_fact_typed_expr() never unwrapped a conditional expression.
(old.bases_fact if condition else new.bases_fact) == other -- and the
equivalent fact = old.bases_fact if condition else new.bases_fact;
fact == other -- both genuinely produce a Fact-typed result regardless
of which branch actually runs, but ast.IfExp had no branch in this
function at all, unlike the already-handled ast.NamedExpr unwrap.
Fixed with a new ast.IfExp branch requiring both node.body and
node.orelse to independently resolve as Fact-typed -- deliberately
narrower than NamedExpr's unconditional unwrap, since an IfExp
genuinely produces one of two different values depending on
condition, so only the case where both are guaranteed Fact-typed
regardless of outcome is a real, unconditional Fact-typed result;
old.bases_fact if condition else some_other_call() must stay
unflagged, the identical every-branch-must-agree principle the
loop-target literal-collection fix earlier in this section already
applies to a tuple's own elements. Verified against both the inline and
assigned-then-compared repros, plus a negative control with one
non-Fact branch.
(2) The loop-target literal-collection fix only ever handled a bare
ast.Name target -- a tuple-unpacking target was silently excluded
entirely. for fact, tag in ((old.bases_fact, "old"), (new.bases_fact,
"new")): fact == other -- each target position genuinely has its own
per-iteration value, but neither the single-target elif branch nor
anything else in this loop matched a Tuple/List target at all. Fixed
with a new elif branch reusing _paired_unpacking_candidates() --
the identical elementwise pairing ast.Assign's own unpacking handling
already relies on -- once per iteration element (each iteration element
is exactly the "one assignment's worth" of value that function already
knows how to pair against the loop's own, unchanging target shape). Any
single iteration element that isn't itself a literal display of matching
length (or that trips the starred-element exclusion
_paired_unpacking_candidates() already applies) disqualifies the
whole loop via an all_iterations_paired flag, rather than silently
pairing only some iterations -- the identical "no candidates at all over
a partial pairing" principle that function's own docstring already
states, extended across iterations instead of within one. Once every
iteration pairs successfully, each target name's own per-iteration
values are collected and registered together, subject to the identical
every-element-Fact-typed-or-deferred-name conjunctive requirement the
simple-target case already applies via tuple_loop_candidates. Verified
against the reported repro, a negative control confirming the sibling
unpacked position (tag, never Fact-typed) stays unflagged even though
fact in the same loop is, a negative control where one iteration
element fails to pair (disqualifying the whole loop, not just that
iteration), and a negative control for a starred unpacking target.
Both fixes verified empirically: still zero existing hits in the real
repository, mypy/ruff both stayed clean. New tests:
TestConditionalExpressionsRecognizedWhenBothBranchesAreFactTyped and
TestForLoopUnpackingTargetsResolveElementwise in tests/
test_fact_detector_misuse_def_time_scope.py.
Split into a sibling module once these fixes pushed the file past the
AI-readiness file-size gate's own 2000-line hard cap. Every review
round in this section added real, dense docstring explaining why --
the file-size check does not distinguish code from documentation, and
the growth was entirely legitimate (each finding needed its reasoning
recorded, per this file's own "known gaps over risky reactive patches"
and "fix the cause, not the instance" conventions), so the fix is a
mechanical extraction, not a diet. The eight lexical-scope-resolution
building blocks every alias-resolution function in this module builds on
(_enclosing_qualnames/_qualname_at/_QualnameSpans,
_lexical_function_parents, _def_containing_qualnames,
_bound_names, _paired_unpacking_candidates, _match_pattern_names)
moved, unchanged, into a new sibling leaf module,
scripts/fact_detector_misuse_scope.py, imported by
fact_detector_misuse.py via a sys.path guard mirroring
check_ai_readiness.py's own identical one -- needed because a bare,
non-dotted from fact_detector_misuse_scope import ... only resolves
when scripts/ itself is on sys.path, which is guaranteed when this
module is run directly (Python adds its own directory automatically) or
when check_ai_readiness.py was imported first in the same process (it
already inserts its own directory before importing fact_detector_
misuse the identical bare way) -- but not guaranteed for a test file
that imports scripts.fact_detector_misuse on its own, as two of this
module's own test files already do. Verified by running each of those
two test files in isolation (not just as part of the full suite), which
is exactly the scenario a missing guard would silently pass in a
full-suite run and fail only in isolation. fact_detector_misuse.py
dropped from 2058 to 1369 lines; the new module is 739. No behavior
change -- every moved function is bit-for-bit identical to its original,
confirmed by the full existing test suite passing unchanged (142 tests
across all three test files) and a fresh mypy/ruff pass on both
files. Registered in scripts/CLAUDE.md's inventory table (the
script-inventory AI-readiness check's own requirement).
A further Codex review round found four more real gaps, all fixed.
(1) _is_fact_typed_expr()'s own IfExp branch can't resolve a
bare-Name alias, since it has no access to a scope's known set.
old_fact = old.bases_fact; new_fact = new.bases_fact; fact = old_fact if
cond else new_fact; fact == other is a real misuse -- both branches are
aliases the surrounding fixed point already confirmed Fact-typed -- but
the structural, scope-independent _is_fact_typed_expr() deliberately
never resolves a bare name at all. Fixed with a new
_candidate_resolves_to_fact(value, fact_names, known): the
fixed-point-aware sibling every ordinary candidates entry already got
via its own inline isinstance(value, ast.Name) and value.id in known
check, generalized to recurse through an IfExp's own two branches too
(each independently required to resolve -- structurally, as an
already-known alias, or itself another nested IfExp -- before the
conditional as a whole is trusted). Both the ordinary candidates loop
and tuple_loop_candidates' own per-element check now go through this
one function instead of duplicating the same inline check twice.
(2) A comprehension's tuple-loop-target candidate always registered
every element against the comprehension's own scope, even the first
generator's own iterable, which actually evaluates in the parent
scope. fact = rec.bases_fact; [fact == other for fact in (fact,)] --
the tuple element fact names the outer alias, but the comprehension's
own target is also fact, shadowing it in the comprehension's own
scope; checking the element there checks the shadowed name against
itself and never resolves. This needed a genuine data-model change, not
just a smarter check: tuple_loop_candidates now pairs each element with
its own resolution qualname (list[tuple[ast.expr, str]], not a flat
list[ast.expr]) -- the resolved target name still becomes known in the
comprehension's own scope (where the actual read happens), but each
element is checked against aliases.get(elt_qualname, set()), which for
the first generator is the position _qualname_at() resolves via the
narrower override span _enclosing_qualnames() already registers for
that exact iterable, tagged with the comprehension's own incoming
(parent) qualname. A plain for loop and every generator after the
first pair every element with their own (unchanged) qualname, so this is
a strict widening, not a behavior change for the already-fixed cases.
(3) The comprehension's own tuple-loop-target branch only ever matched
a bare ast.Name generator target -- the comprehension counterpart of
finding (2) two Codex rounds ago, applied to a plain for loop, was
never extended to a comprehension's own generator. [fact == other for
fact, tag in ((old.bases_fact, "old"),)] was invisible. Fixed with the
identical _paired_unpacking_candidates()-per-iteration-element
machinery the for/AsyncFor branch already uses, applied per
generator -- paired with each element's own resolution qualname from
finding (2), and registered under the comprehension's own scope the
same way finding (2)'s fix already does.
(4) A whole-subject match capture was recorded only as an ordinary
local binding, never as an alias of the match subject. match
rec.bases_fact: case fact: return fact == other -- case fact: (a bare
capture) and case SomeClass() as fact: (an as-pattern) both bind the
entire subject unconditionally whenever that case matches, making the
captured name a real alias of node.subject, not merely an arbitrary
shadow the way a nested capture inside a larger structural pattern
(case [x, y as fact]:, capturing only a sub-part) correctly still is.
Both shapes are exactly case.pattern itself being an ast.MatchAs with
a real name -- Python's own grammar for a top-level capture/as-
pattern. Fixed by registering (case.pattern.name, node.subject) as an
ordinary candidate whenever isinstance(case.pattern, ast.MatchAs) and
case.pattern.name is not None, reusing the existing candidates
fixed point rather than adding a new mechanism.
Verified against each finding's own reported repro plus negative
controls (an unresolved conditional branch; an unrelated first-iterable
name; the sibling unpacked position staying unflagged; a nested
structural capture; a non-Fact match subject) and the full existing
suite, confirming every previously-fixed shape in this section is
unaffected. mypy caught two real variable-redefinition/type-narrowing
issues in the same pass (a same-named local reused with a different type
across the for-loop and comprehension branches) -- fixed by renaming,
not by suppressing. ruff/mypy both stayed clean, still zero existing
hits in the real repository, fact_detector_misuse.py at 1523 lines
(well under the 2000-line hard cap). New tests:
TestConditionalExpressionResolvesThroughAliasBranches,
TestComprehensionFirstIterableResolvesAgainstTheParentScope,
TestComprehensionUnpackingTargetsResolveElementwise, and
TestWholeSubjectMatchCapturesAreAliases in tests/
test_fact_detector_misuse_def_time_scope.py.
A fifth finding, from the next review round: a direct conditional
comparison operand -- never assigned to an intermediate variable at all --
still bypassed alias resolution, even after finding (1) above fixed the
assignment-candidate half of the identical IfExp shape.
old_fact = rec.bases_fact; new_fact = rec.bases_fact; (old_fact if cond
else new_fact) == other returned no misuse site. The two IfExp checks
already in the module are not the same check: _is_fact_typed_expr()'s own
IfExp branch (used by _fact_aliases()'s fixed point when registering a
candidate) requires both branches to be structurally Fact-typed on their
own -- it never resolves an alias Name in either branch, since alias
resolution is the fixed point's job, not this structural predicate's.
fact_equality_misuse_sites()'s own terminal is_fact_typed() closure --
the one that actually decides whether a raw ==/!= operand counts,
applied after the whole fixed point has already converged -- had no IfExp
branch of its own at all: it fell through to isinstance(node, ast.Name),
which a conditional expression never satisfies, so a direct conditional
operand could not be recognized regardless of what either branch resolved
to. Fixed by giving is_fact_typed() its own recursive IfExp branch,
mirroring _candidate_resolves_to_fact()'s reasoning but adapted to this
call site's available state: it checks each branch via is_fact_typed()
itself (so a nested conditional operand resolves too) against the
comparison's own qualname, reading the final, already-converged
aliases mapping rather than an in-progress fixed-point known set, since
by the time this terminal check runs _fact_aliases() has already finished.
Kept the same AND semantics every other IfExp handling in this module
already uses (_is_fact_typed_expr, _candidate_resolves_to_fact): both
branches must resolve, since a single-branch match cannot be told apart
from an ordinary conditional expression that merely happens to read one
Fact attribute among other unrelated locals -- a real, if narrow, accepted
false-negative direction, not new to this fix.
Verified against the reported repro, a nested-conditional variant, and two
negative controls (neither branch resolves; only one branch resolves) via
direct python3 -c reproduction before writing tests, plus the full
existing suite confirming every previously-fixed shape stays unaffected.
ruff/mypy both clean, still zero existing hits in the real repository,
fact_detector_misuse.py at 1529 lines (well under the 2000-line hard cap).
New tests:
TestDirectConditionalOperandsResolveThroughAliasBranches in tests/
test_fact_detector_misuse_def_time_scope.py.
A CodeRabbit review round re-raised the postponed-annotations question
this module's own walrus-in-annotation collection loop already reasoned
through and deliberately declined to special-case -- investigated afresh,
the standing decision stands, not re-implemented. The finding: _fact_
aliases() registers a walrus target found inside a parameter's own
annotation or the -> return annotation as bound at def-time
unconditionally, even when the module carries from __future__ import
annotations (PEP 563), under which an annotation expression is never
evaluated at all -- it is stored as an unparsed string, so the walrus
inside it never actually executes and the name it would have bound is
never really available. This is not a new observation: the exact same
walrus-collection loop's own inline comment already states this trade-off
explicitly ("absent from __future__ import annotations ... unconditionally
walking here too matches that established, deliberately conservative
choice rather than adding a second, narrower rule: a walrus that in fact
never executes under postponed evaluation registering a spurious alias is
a false positive, the safe direction this whole module already accepts
throughout"). Re-verified the reasoning still holds rather than accepting
it on faith: (1) this repository's own AGENTS.md convention mandates
from __future__ import annotations in every scanned production file
(abicheck/**/*.py), so a correct implementation would need to gate on a
per-module fact (does the scanned file itself carry the future import)
threaded specifically into the annotation-embedded half of the walrus
walk -- default-expression walruses are unaffected by PEP 563 and must
keep binding eagerly regardless, so the two subtrees this loop currently
treats identically (walrus_subtrees = [*node.args.defaults, *node.args.
kw_defaults], with annotations appended after) would need to split back
apart, undoing the very unification that closed the earlier annotation-
vs-default gap this class of finding exists to prevent reopening; (2) the
scenario itself -- a walrus operator inside a type annotation, assigning a
Fact[...]-typed value as a side effect of annotating a parameter -- is
adversarial, not a pattern any real detector code in this repository
would plausibly write, unlike every other alias shape this module has
special-cased so far (each traced to an ordinary, unremarkable refactor a
real contributor could genuinely perform); (3) the failure direction is
already the accepted one throughout this module -- an over-approximated
alias only ever risks flagging a comparison that, if the annotation truly
never executes, would itself raise NameError before reaching the
comparison, so the practical cost of not fixing it is a spurious ERROR on
code that cannot run as written, not a missed real misuse. Given the
fix's real complexity (a second, narrower rule threaded through one
specific subtree of an already-hardened, many-times-reviewed collection
loop) against a scenario with no plausible real-world occurrence, this is
documented as a known, deliberate, already-reasoned-through accepted
false-positive direction rather than implemented -- consistent with this
module's own established "the safe direction this whole module already
accepts throughout" principle, not a gap distinct from what the code
already states. No code change; replied to the review thread pointing at
the existing docstring's own reasoning.
Two more findings from the same review round, both real, both bounded
extensions of the same recursive-resolver mechanism the direct-
conditional-operand fix just established. (1) _is_fact_typed_expr()'s
own NamedExpr branch unwraps a walrus to its .value and checks it
structurally, but it deliberately never resolves a bare Name there (the
same limit every structural check in this module accepts -- alias
resolution needs scope, which a single node can't supply). Neither of the
two places that can supply scope had a matching NamedExpr branch of
their own: _candidate_resolves_to_fact() (the fixed-point-aware
resolver a candidate's own value is checked against) and is_fact_typed()
(fact_equality_misuse_sites()'s own terminal comparison-operand
predicate). old_fact = rec.bases_fact; (copy := old_fact) == other (a
direct comparison operand) and old_fact = rec.bases_fact; fact = (copy
:= old_fact); fact == other (assigned through an intermediate name
first) were both missed -- _is_fact_typed_expr correctly declined to
resolve the wrapped bare old_fact, and neither caller had anywhere else
to turn. Fixed by giving both functions the identical recursive
NamedExpr branch already established for IfExp one round earlier:
_candidate_resolves_to_fact(value.value, fact_names, known) and
is_fact_typed(node.value, qualname) respectively -- each simply
recurses into the walrus's own value through the same resolver, so a
nested walrus ((a := (b := old_fact)) == other) resolves too, and a
walrus wrapping a genuinely non-Fact name stays correctly unflagged. (2)
The parameter-default resolution loop's own inline check --
_is_fact_typed_expr(default, fact_names) or (isinstance(default,
ast.Name) and default.id in aliases.get(parent, ())) -- was a narrower,
duplicated re-implementation of exactly what _candidate_resolves_to_fact()
already generalizes: it had no IfExp (or now NamedExpr) branch of its
own, so a default composing two already-known aliases through a
conditional expression (old = rec.bases_fact; new = rec.vtable_fact; def
inner(value=old if cond else new): return value == other) was rejected
outright, even though both branches were already confirmed Fact-typed
aliases in the parent scope. Fixed by replacing the inline check with a
direct call to _candidate_resolves_to_fact(default, fact_names,
aliases.get(parent, set())) -- the identical resolver every other
candidate site already uses, so this loop can no longer independently
drift from what the rest of the module considers Fact-typed, and it
inherits the NamedExpr fix above for free.
Verified against all three reported repros (direct NamedExpr operand,
assigned-through NamedExpr alias, composed IfExp default), a nested-
NamedExpr variant, and two negative controls (a NamedExpr wrapping a
non-Fact name; a composed default with only one Fact-typed branch,
pinning the same AND semantics established for the direct-conditional-
operand fix) via direct python3 -c reproduction before writing tests.
ruff/mypy both stayed clean, still zero existing hits in the real
repository, fact_detector_misuse.py at 1533 lines (well under the
2000-line hard cap). New tests:
TestNamedExpressionsResolveThroughAliasBranches in tests/
test_fact_detector_misuse_def_time_scope.py.
Two more findings from the next review round, plus a test-file split
the second push over the 1200-line cap. (1) _is_fact_typed_annotation()'s
generic fallback branch (return _is_fact_typed_annotation(annotation.value,
fact_names)) is correct for a plain Fact[int] subscript -- annotation.
value is the Fact head name itself -- but wrong for Annotated[Fact[int],
metadata] (PEP 593): there, annotation.value is Annotated, not Fact,
and the real type lives in the subscript's own slice (its first element;
everything after is arbitrary metadata, never itself a type to check).
def f(value: Annotated[Fact[int], "meta"], other): return value == other
was invisible. Fixed with a dedicated head == "Annotated" branch,
mirroring the existing Optional/Union branches' own tuple-vs-bare-slice
handling: recurses into the slice's first element only (elts[0] when the
slice is a Tuple, else the bare slice), leaving every later metadata
element untouched. Composes correctly with the existing Optional
handling (Annotated[Optional[Fact[int]], "meta"]), since each branch
only ever recurses through the same function. (2) The ast.Match branch's
existing whole-subject-capture recognition (case fact:/case SomeClass()
as fact:) had no counterpart for an OR pattern where every alternative
independently captures the whole subject under the same name -- case
(list() as fact) | (tuple() as fact): fact == other was invisible, since
case.pattern there is ast.MatchOr, never itself an ast.MatchAs.
Python requires every alternative of an OR pattern to bind the identical
set of names (a SyntaxError otherwise), but not the identical binding
shape -- case C(x=fact) | (D() as fact): legally binds fact in both
branches, but only the second branch binds it to the whole subject, so
consistency of the bound name alone is not sufficient evidence. Fixed
by requiring every single alternative to itself be exactly ast.MatchAs
with the identical .name before treating it as an alias of node.
subject -- verified this correctly rejects the mixed-shape case above
while still accepting a bare-capture-mixed-with-an-as-pattern OR
(case fact | (tuple() as fact):) and a three-way OR, since each
alternative there is independently a whole-subject MatchAs.
Verified against both reported repros, a multi-metadata-item variant and
an Annotated+Optional composition for finding (1), a three-way OR and
a bare-capture-mixed-with-MatchAs OR for finding (2), and negative
controls for each (a non-Fact Annotated type; a non-Fact match subject;
the mixed-binding-shape OR pattern) via direct AST reproduction before
writing tests. Adding these tests pushed tests/test_fact_detector_misuse_
def_time_scope.py to 1198 of its own 1200-line test-file cap -- two lines
of headroom, effectively none -- so its tail (the four most recent
alias-resolution-edge-case test classes: direct conditional operands,
NamedExpr, Annotated, MatchOr) was split into a new sibling file,
tests/test_fact_detector_misuse_alias_edge_cases.py, mirroring how that
file was itself split out of test_fact_detector_misuse_scoping.py --
mechanical extraction, every class moved unchanged, verified both in
combination and in isolation. ruff/mypy both stayed clean, still zero
existing hits in the real repository, fact_detector_misuse.py at 1572
lines (well under the 2000-line hard cap), the def-time-scope test file
back down to 940 lines, the new edge-case test file at 299 lines. New
tests: TestAnnotatedWrapperUnwrapsToItsFirstSliceElement and
TestMatchOrPropagatesWholeSubjectCaptures, both now in tests/
test_fact_detector_misuse_alias_edge_cases.py.
Two more findings from the next review round, both bounded extensions
of already-established mechanisms. (1) None of _is_fact_typed_expr(),
_candidate_resolves_to_fact(), or fact_equality_misuse_sites()'s own
terminal is_fact_typed() predicate had a BoolOp branch, so (old.
bases_fact or new.bases_fact) == other was missed entirely, as was the
assignment-through-alias form. Python's and/or always return one of
their own operands verbatim -- never a synthesized True/False, only
ever short-circuiting to whichever operand its own truthiness picks -- so
if every operand of a BoolOp is guaranteed Fact-typed, the result is
too, regardless of which one runtime truthiness actually selects: the
identical every-operand-must-agree principle the IfExp branches already
established, generalized from two branches to BoolOp.values's arbitrary
operand count (an a or b or c chain is one BoolOp node with three
values, not two nested ones). Fixed by giving all three functions the
identical recursive BoolOp branch, mirroring each one's own existing
IfExp branch exactly. (2) The comprehension-scope-hopping walrus
collection branch (an earlier round's own fix) registers a hopped
target's candidate under binding_qualname (the enclosing scope PEP 572
hops it out to), but paired it with node.value unconditionally checked
against that same scope's converged aliases at fixed-point time -- even
though the walrus's RHS is still textually written, and can only ever
resolve, in the comprehension's own scope. [(captured := fact) for
fact in (rec.bases_fact,)]; captured == other was missed: fact (the
comprehension's own for-bound target) is never a known alias of
binding_qualname (the enclosing function), only of the comprehension's
own scope. This is the identical "one shared qualname per entry can't
express both a binding scope and a resolution scope" problem tuple_loop_
candidates's own per-element (elt, elt_qualname) pairing already
solves for a tuple's elements -- applied here to a single scalar walrus
value instead. Fixed with a new cross_scope_candidates dict, keyed by
the binding qualname exactly like candidates, but each entry additionally
carrying the value's own resolution qualname (walrus_qualname, only
populated when a real hop occurred -- the no-hop case still uses the
ordinary candidates dict unchanged); the fixed-point loop resolves each
entry's value against aliases.get(value_qualname, set()) rather than
the current pass's known set, mirroring tuple_loop_candidates's own
per-element resolution exactly. A walrus hopping out of nested
comprehensions (PEP 572 hops out of all of them, not just the innermost)
resolves correctly too, since walrus_qualname still names the
comprehension it was textually written in.
Verified against both reported repros, a three-way BoolOp chain, a
BoolOp-via-alias variant, alias-resolved BoolOp operands, a negative
control pinning the AND semantics (one non-Fact operand disqualifies the
whole expression) for finding (1); and a non-Fact-RHS negative control, a
double-hopped nested-comprehension variant, and a no-hop regression guard
for finding (2) -- all via direct AST reproduction before writing tests.
Zero existing hits, mypy/ruff both stayed clean,
fact_detector_misuse.py at 1642 lines (well under the 2000-line hard
cap). New tests: TestBoolOpResolvesWhenEveryOperandIsFactTyped and
TestComprehensionWalrusResolvesItsRhsInItsOwnScope, both in tests/
test_fact_detector_misuse_alias_edge_cases.py (410 lines, well under its
own 1200-line cap).
Two more findings from the next review round, both bounded extensions of
already-established mechanisms. (1) _iter_default_subtree() -- the
helper _default_and_annotation_scope_overrides() uses to attribute
everything inside a def/lambda's own default value or annotation to
its def-time (enclosing) scope -- stops descending the moment it reaches
any scope-introducing node, comprehensions included, since a comprehension
genuinely creates its own new scope for its elt/later generators. But a
comprehension's own outermost generator's iterable is the one
established exception to that rule (the identical carve-out
_enclosing_qualnames's and _lexical_function_parents's own
comprehension handling already give the same construct, documented in
their own docstrings): it evaluates in the scope enclosing the
comprehension, before the comprehension's own implicit function is even
called. _iter_default_subtree() never carried the same exception, so
fact = rec.bases_fact; def g(fact, cb=[x for x in (fact == other,)]):
... was silently missed -- the comparison sits inside exactly that
exempt iterable, but the walk stopped at the ListComp boundary before
ever reaching it. Fixed by giving _iter_default_subtree() the identical
one-generator carve-out: on reaching a comprehension, its outermost
iterable is pushed back onto the walk (still under the def-time scope this
whole function exists to attribute), and everything else in the
comprehension is left correctly opaque. The exception recurses through the
walk's own stack the same way it already does in the two sibling
functions, so a doubly-nested comprehension's own outermost iterable
(itself nested two hops from the def) resolves correctly too, with no
extra logic needed. (2) _is_fact_typed_annotation() had no case for a
stringized (quoted) annotation -- def f(old_fact: "Fact[list[str]]",
other): return old_fact == other -- a real, common spelling (required
under from __future__ import annotations for anything evaluated lazily,
and used ad hoc even without it to break an import cycle or reference a
not-yet-defined name), invisible to every existing shape check since it
parses as a bare ast.Constant string rather than any of the Name/
Subscript/BinOp shapes the function already recognized. Fixed by
parsing the string as an expression (ast.parse(..., mode="eval")) and
recursing into the parsed body through the same function -- so the fix
composes for free with every wrapper the function already handles
(Optional[...], Union[...], the X | None PEP 604 spelling,
Annotated[...]), rather than needing its own copy of that logic. A
string that isn't valid Python at all (unrelated malformed input, not a
forward-reference annotation) degrades to False rather than propagating
a SyntaxError, matching every other best-effort parse in this module.
Verified against both reported repros -- a doubly-nested comprehension
variant and a lambda-default-body negative control (a genuine, ordinary
scope boundary, unlike a comprehension's outermost iterable) for finding
(1); a stringized annotation wrapped in Optional[...], one using the
PEP 604 X | None spelling, a malformed-string negative control, and an
unrelated non-Fact stringized-annotation negative control for finding
(2) -- all via direct AST reproduction before writing tests. Zero existing
hits, mypy/ruff both stayed clean, fact_detector_misuse.py at 1683
lines (well under the 2000-line hard cap). New tests:
TestDefaultComprehensionOutermostIterableStaysDefTimeScoped and
TestStringizedFactAnnotationsAreRecognized, both appended to tests/
test_fact_detector_misuse_def_time_scope.py (1062 lines, well under its
own 1200-line cap -- the natural home for both, since each is a def-time/
annotation-scope fix, not a plain alias-resolution one).
One more finding from the next review round, a bounded extension of an
already-established mechanism. The single-target for/comprehension
loop-binding branches (tuple_loop_candidates's own collection: "one loop
target bound, one iteration at a time, to every element of a statically
known display") recognized only ast.Tuple/ast.List, so for fact in
{old.bases_fact, new.bases_fact}: fact == other (a set display) and for
fact in {old.bases_fact: 1, new.bases_fact: 2}: fact == other (a dict
display, iterated as its keys) were both invisible, as was the identical
gap in the duplicated comprehension-generator branch -- an ordinary choice
of container literal, not a different question from the Tuple/List
case already handled. Fixed with one shared _static_display_elements()
helper (Tuple/List/Set → .elts, Dict → .keys, None for
anything else or for a dict containing a **expansion -- a None entry
in ast.Dict.keys, whose own value could be anything, so the whole
display can't be treated as statically enumerable), used at both the
for-loop and comprehension single-target sites instead of duplicating
the three-way shape check at each.
Verified against both reported repros (set-display for, dict-keys
for), the identical pair for the comprehension form, a mixed-element
set negative control (only some elements Fact-typed -- must stay
unflagged, mirroring the existing tuple negative control), a **expansion
dict negative control, and an empty-display negative control ({},
Python's only empty-display literal -- there is no bare empty-set syntax,
and set() is a call, correctly not recognized as a display at all), all
via direct AST reproduction before writing tests. Still zero existing
hits, mypy/ruff both stayed clean, fact_detector_misuse.py at 1714
lines (well under the 2000-line hard cap). New tests:
TestStaticDisplayLoopTargetsRecognizeSetAndDictKeys in tests/
test_fact_detector_misuse_alias_edge_cases.py (493 lines, well under its
own 1200-line cap).
Three more findings from the next review round, all bounded extensions
of already-established mechanisms, plus one declined as a re-raise of an
already-decided, documented tradeoff. (1) The tuple-unpacking loop/
comprehension branches (the destructured-target sibling of the set/dict-
display fix above) still gated on a hand-rolled isinstance(node.iter,
(ast.Tuple, ast.List))/isinstance(generator.iter, (ast.Tuple, ast.List))
of their own rather than reusing _static_display_elements() -- the
identical drift risk the simple-target case was already fixed for one
round earlier. for fact, tag in {(rec.bases_fact, "old")}: fact == other
(a set of tuples) was invisible, as was the identical dict-keys and
comprehension form. Fixed by reusing the already-computed display_elts/
gen_display_elts local (computed once per branch, shared by both the
simple-target and tuple-unpacking cases) instead of a second, independent
shape check. (2) None of the four loop/comprehension collection branches'
own element-admission gate (_is_fact_typed_expr(elt, fact_names) or
isinstance(elt, ast.Name)) recognized a composed expression
(NamedExpr/IfExp/BoolOp) as a candidate worth deferring to
fixed-point time, even though _candidate_resolves_to_fact() -- the
function that actually resolves a deferred candidate once known/
aliases are populated -- already has its own recursive branch for every
one of those three shapes. old = rec1.bases_fact; new = rec2.
vtable_fact; for fact in (old if cond else new,): fact == other was
rejected outright at collection time, since old if cond else new is
neither already Fact-typed (its own leaves aren't resolved as aliases
yet) nor a bare name. Fixed with one shared _admissible_loop_element()
gate (_is_fact_typed_expr(...) or one of _DEFERRED_CANDIDATE_NODE_
TYPES -- Name/NamedExpr/IfExp/BoolOp), replacing all four
independent copies of the old two-clause check at once, so a future
composed shape added to _candidate_resolves_to_fact() only needs
updating in this one admission set to reach every collection site. (3)
Fact[int](...)/Fact[int].present(...) -- a generic specialization of
Fact, which subscripting produces as a _GenericAlias whose own
__call__/attribute access delegates straight through to the real class
-- is still exactly Fact at runtime, but the callable is an
ast.Subscript (or an ast.Attribute whose .value is one), invisible
to the constructor-call recognition in _is_fact_typed_expr(), which
only ever unwrapped a bare ast.Name. Fixed with a single unwrap of an
ast.Subscript callee (and, separately, of an ast.Attribute's own
.value) before the existing Name-in-fact_names checks, composing
for free with both existing shapes rather than needing a duplicate check.
(4) Declined, citing an already-documented rationale rather than
re-implementing. A module-qualified Fact annotation (value: model.
Fact[int]/value: fact_module.Fact) is the identical "no type
inference, match by import spelling" residual _imported_fact_aliases()'s
own docstring already documents and accepts for the module-qualified
constructor-call form (fact_model.Fact.present(...)) -- both trace to
the same root cause (_is_fact_typed_annotation's/_is_fact_typed_expr's
constructor-call branch each assuming a bare ast.Name rather than an
arbitrary ast.Attribute chain) and the same accepted-gap paragraph in
that docstring ("This module's own module docstring already accepts as
inherent to a pure-AST heuristic, not a specific miss worth chasing
indefinitely"). Documented as covering the annotation spelling too rather
than re-litigated as a fresh finding.
Verified against all three reported repros for findings (1)-(3) (the
destructured set/dict-keys forms and their comprehension duplicates; the
composed-IfExp loop element and its comprehension duplicate; both
generic-specialized constructor spellings), plus negative controls for
each (a destructured set with one non-Fact element; an IfExp with one
non-Fact branch) and a regression guard confirming the unspecialized
Fact(...) form still works, all via direct AST reproduction before
writing tests. Still zero existing hits, mypy/ruff both stayed clean,
fact_detector_misuse.py at 1780 lines (well under the 2000-line hard
cap). New tests: TestDestructuredLoopsRecognizeSetAndDictKeyDisplays,
TestComposedLoopElementsDeferToTheAliasFixedPoint, and
TestGenericSpecializedFactConstructorsAreRecognized, all appended to
tests/test_fact_detector_misuse_alias_edge_cases.py (608 lines, well
under its own 1200-line cap).
One more finding from the next review round, a bounded extension of an
established mechanism. The match statement's whole-subject-capture
handling (a bare case fact:/case SomeClass() as fact:, or a
whole-subject MatchOr) only ever recognized case.pattern itself being
a top-level MatchAs -- a structural sequence pattern capturing a
sub-part of the subject, not the whole thing, was invisible: match
(rec.bases_fact, tag): case (fact, _): return fact == other -- fact is
definitively the subject tuple's first, Fact-typed element, but neither
existing branch applies (case.pattern is an ast.MatchSequence, not a
bare MatchAs/MatchOr-of-MatchAs). Fixed with a new
_paired_match_sequence_candidates() -- the match/case sibling of
_paired_unpacking_candidates() this module already uses for ordinary
tuple-unpacking assignment -- pairing a structural sequence pattern's own
captures against a statically-known Tuple/List subject's elements,
positionally, recursing through a nested sequence pattern matched against
a nested Tuple/List subject element the identical way its assignment-
unpacking sibling already nests. Deliberately not all-or-nothing the
way _paired_unpacking_candidates() is, though: a non-capturing
sub-pattern at one position (a wildcard _, a literal, a class pattern)
does not disqualify a real capture found at another position, since a
structural pattern routinely mixes captures with non-capturing
sub-patterns as completely ordinary code -- only a genuine shape mismatch
(a pattern/subject length disagreement, more than one MatchStar) makes
the whole pairing untrustworthy, since then no position can be
confidently attributed to the right subject element at all. A
MatchStar's own captured name (case (fact, *rest):) binds a runtime
list, not a single value, and is deliberately not treated as a Fact-typed
candidate here (_match_pattern_names() still records it as an ordinary
local bound name, a real shadow, just not an alias source).
Fixing this correctly required updating one existing test, not just
adding new ones. test_ignores_a_nested_capture_inside_a_structural_
pattern (a pre-existing negative control, added for the whole-subject-
capture fix earlier in this module's history) used match [rec.bases_fact,
1]: case [x, y]: return x == other as its repro -- which its own
docstring correctly names as "must not be treated as an alias of the
[whole] subject," but its assertion (== [], no finding at all) also
happened to encode the narrower, not-yet-fixed gap this same round
closes: once elementwise pairing exists, x genuinely is rec.
bases_fact (the subject's first element), so x == other now correctly
IS a real misuse, and asserting == [] there would pin the bug rather
than guard the fix. Reproducing the pre-fix suite confirmed this test
failed immediately, for exactly that reason, once the new branch landed.
Fixed by changing the test's own subject to a dynamic (non-statically-
known) one, pair rather than a literal display, which the elementwise
pairing correctly still returns no candidates for (subject isn't a
recognized static display) -- preserving the test's original, narrower
purpose (guard against the whole-subject-alias mechanism over-eagerly
matching a partial capture) without it silently drifting into pinning a
now-fixed gap.
Verified against the reported repro, a nested-sequence-pattern variant, a
leading-star and a trailing-star variant (confirming a captured position
binds to the correct, Python-semantics-accurate subject element on either
side of a *rest), a wildcard-position positive control (a non-capturing
sub-pattern elsewhere must not block a real capture), and negative
controls (a captured but genuinely non-Fact element; a dynamic subject; a
pattern/subject length mismatch), all via direct AST reproduction before
writing tests. Still zero existing hits, mypy/ruff both stayed clean,
fact_detector_misuse.py at 1797 lines and fact_detector_misuse_scope.py
at 803 lines (both well under the 2000-line hard cap). New tests:
TestStructuralSequencePatternCapturesPairWithTheSubject, appended to
tests/test_fact_detector_misuse_alias_edge_cases.py (710 lines, well
under its own 1200-line cap).
Two more findings on the same pairing machinery (7th review round),
both real, both bounded extensions of the mechanism already built for
the sequence-pattern fix above. (1) _paired_unpacking_candidates()'s
own blanket "any Starred element anywhere disqualifies the whole
pairing" rule was one rule too many: a starred target element (fact,
*rest = old.bases_fact, new.bases_fact, extra) captures a runtime-length
slice with no single corresponding value, but the fixed-position
elements before and after it still line up unambiguously against the
value display's own (starless, so fixed-length) elements — the identical
before/after-the-star split _paired_match_sequence_candidates() already
uses for a MatchStar. A starred value element (x = (*a, b)) stays a
genuine disqualifier throughout, since a dynamic expansion's own length
isn't statically known. Fixed by splitting the target's own Starred
position (at most one, matching Python's own grammar) into before/after
slices and pairing each against the value's own front/back elements by
position, mirroring the sequence-pattern helper's shape exactly; the star
element itself never produces a candidate, matching that helper's
identical treatment of MatchStar's own captured name. This reaches both
a plain assignment and a for loop's own unpacking target, since the
latter already reuses this same helper per iteration element. (2) The
sequence-pattern fix's own docstring explicitly scoped itself to
MatchSequence and left every MatchMapping pattern to the
whole-subject-capture handling, which never applies to a structural,
sub-part-capturing pattern — so case {"fact": fact}: against a literal
{"fact": rec.bases_fact} subject stayed an unrecognized capture. Fixed
with a new _paired_match_mapping_candidates(), the MatchMapping
sibling of _paired_match_sequence_candidates(): pairs a pattern's own
literal keys against a literal Dict subject's own literal keys
independently per key (unlike sequence pairing's positional,
whole-pairing-disqualifying shape, one key missing or unresolvable in the
subject contributes no candidate for that key without disqualifying
the others), requires every subject key to be a literal ast.Constant
with no **expansion entry, and recurses into a further sequence or
mapping sub-pattern the identical way the sequence helper already
recurses into a further sequence. **rest is deliberately not treated
as a candidate here either, mirroring MatchStar's identical treatment.
Two pre-existing tests pinned the exact gap fix (1) closes, and were
corrected rather than left pinning a now-fixed bug (the same discipline
the sequence-pattern round's own test_ignores_a_nested_capture_inside_a_
structural_pattern correction already established): both
TestElementwiseTupleUnpackingAliases::test_ignores_a_starred_target_
pairing and TestForLoopUnpackingTargetsResolveElementwise::test_ignores_
a_starred_unpacking_target asserted == [] for a starred-target
unpacking whose fixed position is now correctly Fact-typed — reproducing
the pre-fix suite confirmed both failed immediately once the new
before/after-the-star pairing landed, for exactly that reason. Each was
rewritten to isolate its own original, narrower purpose (confirming the
starred capture itself is never treated as a Fact alias — now checked via
a comparison involving the starred name specifically) from the new,
correctly-detected fixed-position finding, rather than asserting == []
against a case that is now a genuine, non-spurious hit.
Verified against the reported starred-target repro (leading and trailing
star, plain assignment and for-loop unpacking), a starred-value
regression control (still correctly excluded), and the mapping-pattern
repro plus its own negative controls (**rest not blocking a sibling
capture, a key absent from the subject, a dynamic non-Dict subject, a
non-literal subject key, a nested sequence pattern inside a mapping
pattern), all via direct AST reproduction before writing tests. Still
zero existing hits, mypy/ruff both stayed clean,
fact_detector_misuse.py at 1815 lines and fact_detector_misuse_scope.py
at 917 lines (both well under the 2000-line hard cap). New tests:
TestStarredUnpackingTargetsStillPairFixedPositions and
TestStructuralMappingPatternCapturesPairWithTheSubject, appended to
tests/test_fact_detector_misuse_alias_edge_cases.py (851 lines, well
under its own 1200-line cap).
One more finding on the def-time scope-resolution machinery (8th review
round): decorator expressions were never given the def-time treatment
their siblings already have. _default_and_annotation_scope_overrides()
already resolves a parameter default/annotation and a ClassDef's own
base/keyword expression against the scope that directly, syntactically
contains the def/class statement, since Python evaluates each of
those before the function/class exists — but a decorator
(@deco(fact == other)) is evaluated the identical way, at the identical
time, and this function's own subtree collection had no decorator_list
entry for either shape at all. fact = rec.bases_fact; @deco([x for x in
(fact == other,)]) def f(fact): ... was silently missed:
_enclosing_qualnames assigns the whole FunctionDef's decorator-
adjacent lines to f's own body scope, where the parameter fact has
already shadowed the alias. Confirmed as a genuine asymmetry, not a fresh
question: _lexical_function_parents's and _def_containing_qualnames's
own def_time_subtrees()/dispatch() helpers already dispatch a
def's or class's decorator_list under the incoming (enclosing)
qualname — this function alone had never been extended to match. Fixed
by adding decorator_list to both the function/lambda branch's
subtrees (via getattr, since ast.Lambda carries none) and the
ClassDef branch's own base/keyword loop — the identical fix, applied to
the one remaining site that needed it.
Verified against the reported repro, its class-decorator sibling, a
regression control confirming the fix doesn't leak the decorator's own
resolution into the function's real body scope (a genuine parameter
still correctly shadows there), a non-Fact decorator negative control,
and both a nested-function and a method decorator (resolving against
their own real enclosing function/class-body scope respectively,
mirroring the identical class-body-vs-function distinction the existing
method-default handling already draws), all via direct AST reproduction
before writing tests. Still zero existing hits, mypy/ruff both stayed
clean, fact_detector_misuse.py at 1843 lines (well under the 2000-line
hard cap). New tests:
TestDecoratorExpressionsResolveAgainstTheContainingScope, appended to
tests/test_fact_detector_misuse_alias_edge_cases.py (946 lines, well
under its own 1200-line cap) rather than the more thematically obvious
test_fact_detector_misuse_def_time_scope.py, which had only ~105 lines
of headroom left under its own cap.
One more finding on the whole-subject match-capture machinery (9th
review round): a MatchAs node can itself nest a further MatchAs,
not just a structural sub-pattern. case fact as alias: parses as
MatchAs(pattern=MatchAs(name="fact"), name="alias") — a genuinely
nested MatchAs, distinct from case SomeClass() as fact:'s own
wrapped MatchClass (a real structural sub-pattern, correctly left
alone) — but the existing top-level-capture branch only ever registered
case.pattern.name (the outer alias), leaving the inner fact to
_match_pattern_names()'s ordinary local-shadow treatment even though
it is equally a real alias of the whole subject. Fixed with a new
_matchas_chain_names(), walking the chain of nested MatchAs nodes and
collecting every name along it — for the ordinary, non-chained case it
still returns exactly the single outer name, so this is a strict
generalization, not a behavior change for either existing shape. The
sibling MatchOr branch (added in an earlier round for case (C() as
fact) | (D() as fact):) had the identical gap one level up: its own
per-alternative check only compared the outer name across
alternatives, not the full nested chain, so a mismatched inner name
(case (fact as alias) | (other_fact as alias):, where only the
outer name alias is actually guaranteed identical by Python's own
same-name-set-across-alternatives grammar rule) could have been
trusted as safe when it isn't. Fixed by requiring every alternative's
full _matchas_chain_names() result to match the first alternative's,
term for term — the identical "only trust when every alternative is
structured identically" principle that branch's own docstring already
states, now applied at the right granularity.
Verified against the reported repro, the inner-name-alone and outer-
name-alone cases independently, both together in one comparison, the
existing single-level as-pattern regression control, an OR pattern
with identical nested chains (still trusted) and one with mismatched
inner names (correctly not trusted), and a negative control confirming
a nested capture inside a genuine structural sub-pattern (case [x, y as
fact]:) stays a sub-part capture, not a whole-subject one, via direct
AST reproduction before writing tests. Still zero existing hits,
mypy/ruff both stayed clean, fact_detector_misuse.py at 1862 lines
and fact_detector_misuse_scope.py at 945 lines (both well under the
2000-line hard cap). New tests:
TestNestedMatchAsChainsPropagateWholeSubjectCaptures (7 tests),
appended to tests/test_fact_detector_misuse_alias_edge_cases.py (1035
lines, still under its own 1200-line cap).
Two more findings on the same commit (10th review round), both in the
structural-pairing per-position handling
(_paired_match_sequence_candidates()/_paired_match_mapping_
candidates()), not the whole-subject MatchAs/MatchOr branch the
previous round fixed. (1) The per-position handling still only ever
extracted sub_pattern.name from a bare MatchAs, so a chained
MatchAs at a structural position (case (fact as alias,): return fact
== other) recorded only the outer alias, missing the inner fact --
the identical nested-MatchAs shape the previous round's fix already
closed at the whole-subject level, just unreached at the per-position
level. A structural sub-pattern wrapped by MatchAs at a position
(case ((fact, _) as alias,):) fell through every branch untouched
too, since the position's own top-level node is MatchAs, not
MatchSequence/MatchMapping directly. (2) The sequence-pairing
function accepted a starred subject (match (*extras, rec.
bases_fact): case (_, fact):) with no guard at all -- a dynamic
expansion of unknown length means no pattern position can be
confidently attributed to a known subject element, the identical rule
_paired_unpacking_candidates() already applies to a starred value
display, just missing on this structural-match sibling.
Fixed both per-position gaps with one new shared helper, _paired_sub_
pattern_candidates(), which both _paired_match_sequence_candidates()
and _paired_match_mapping_candidates() now delegate their per-position
step to instead of their own ad hoc MatchAs/MatchSequence/
MatchMapping branching: it unwinds a chained MatchAs via the
existing _matchas_chain_names() (recording every name along the chain
against the identical sub-subject), then finds the chain's innermost
non-MatchAs wrapped pattern -- skipping past every already-recorded
MatchAs layer so a name is never registered twice -- and recurses into
it structurally when it is itself a MatchSequence/MatchMapping.
Fixed the starred-subject gap with the same any(isinstance(elt,
ast.Starred) ...) guard _paired_unpacking_candidates()'s own value
display already uses, applied to _paired_match_sequence_candidates()'s
subject elements before any positional pairing is attempted (mapping
pairing is unaffected -- it pairs by literal key, not position, so a
starred sequence element has no analogue there).
Verified against both reported repros (chained MatchAs at a sequence
position, starred subject), the chained-MatchAs shape reproduced at a
mapping position too, a structural sub-pattern wrapped by MatchAs
(both the inner structural capture and the outer wrapping alias,
independently), and every existing sibling case re-verified unaffected
(a plain single-level case fact:, a sequence wildcard position, a
MatchStar's own captured name staying an ordinary shadow rather than
an alias, a non-tuple/list subject, an unstarred subject, and a
length-mismatch subject), all via direct AST reproduction before
writing tests. Still zero existing hits, mypy/ruff both stayed
clean, fact_detector_misuse_scope.py at 996 lines (well under the
2000-line hard cap). New tests:
TestNestedMatchAsChainsInsideStructuralPositions (3 tests),
TestStructuralSubPatternWrappedByMatchAsAtAPosition (2 tests), and
TestStarredSubjectDisqualifiesStructuralSequencePairing (2 tests),
appended to tests/test_fact_detector_misuse_alias_edge_cases.py (1140
lines, still under its own 1200-line cap).
Separately, this same round brought PR #929's branch up to date with
main, closing a CI architecture step failure (abicheck/contract_
evidence.py:86: model -> policy is forbidden, a compare -> model ->
policy -> compare dependency cycle) that this PR had already documented
as a pre-existing, unrelated base-branch regression (PR comments
5453660592/5453762855). main had since merged a fix for the identical
cycle (PR #931, fix/contract-evidence-model-policy-cycle); the branch
was 48 commits behind, so merging main in (a clean merge, no conflicts)
was sufficient -- no independent architecture fix was needed on this
branch. Verified with python scripts/check_architecture.py reporting
0 error(s) on the merge commit.
A further Codex review round found _paired_sub_pattern_candidates()
had no MatchOr branch, the identical OR-pattern shape the whole-case.
pattern level already recognizes, just unreached at the per-position
level. case ((C() as fact) | (D() as fact),): deterministically
binds fact to the sequence position's own Fact-valued element
regardless of which alternative matched -- every alternative is a
top-level MatchAs capturing that one position's whole value under the
identical name -- but _paired_sub_pattern_candidates()'s MatchAs/
MatchSequence/MatchMapping branches don't match a MatchOr node at
all, so unmigrated_fact_reader_sites()'s match_detector_misuse
sibling (fact_equality_misuse_sites()) returned no site. Fixed by
extracting the existing whole-case.pattern OR-pattern trust rule
("every alternative is exactly MatchAs with the identical full nested
chain of names, via _matchas_chain_names()") into a new shared
_trusted_matchor_chain_names(), used by both the pre-existing
whole-subject branch (refactored to call it, not reimplement it, so the
already-correct case couldn't silently diverge from the newly-added
one) and a new MatchOr branch in _paired_sub_pattern_candidates()
itself -- deliberately not recursed into an alternative's own wrapped
structural sub-pattern, matching the whole-subject branch's identical
restriction, rather than widening the trust rule at the same time as
relocating it.
Verified against the reported repro, the identical shape reproduced at a
mapping position, three negative controls (mismatched alternative names,
one alternative binding a field rather than the whole value, a chained
MatchAs with mismatched inner names), and the whole-subject MatchOr
branch re-verified unaffected by the refactor (both its positive and
negative case), plus every other _paired_sub_pattern_candidates()
sibling form (plain MatchAs at a position, a chained MatchAs, a
structural sub-pattern wrapped by MatchAs, the starred-subject guard),
all via direct AST reproduction before writing tests. Still zero
existing hits, mypy/ruff both stayed clean, fact_detector_misuse.py
at 1842 lines and fact_detector_misuse_scope.py at 1057 lines (both
well under the 2000-line hard cap). New tests split into a dedicated
sibling file, tests/test_fact_detector_misuse_matchor_structural_
positions.py (7 tests, four classes:
TestMatchOrAtAStructuralSequencePosition/
TestMatchOrAtAStructuralMappingPosition/
TestWholeSubjectMatchOrStillWorksAfterTheSharedHelperRefactor), rather
than appended to test_fact_detector_misuse_alias_edge_cases.py, which
had only ~60 lines of headroom left under its own 1200-line cap.
Three more findings on the same commit (a further Codex review round), all bounded extensions -- one self-inflicted regression caught and fixed before landing.
(1) case (fact,) as whole: -- a structural pattern wrapped by a
top-level as-pattern -- recorded only whole, never recursing into
its own wrapped MatchSequence. The whole-case.pattern dispatch in
fact_equality_misuse_sites() treated MatchAs exclusively as a
whole-subject capture (an inline reimplementation of a subset of the
rules _paired_sub_pattern_candidates()'s own per-position handling
already states in full, including this exact wrapped-structural-pattern
case). Fixed by deleting the whole-case.pattern dispatch's own five
if/elif branches entirely and delegating the whole thing to
_paired_sub_pattern_candidates(case.pattern, node.subject) -- the same
shared primitive, reused whole rather than kept as two independently-
maintained subsets of the same rules that could (and, per this finding,
already had) silently diverge again. Verified every existing whole-
subject shape unaffected (bare capture, SomeClass() as fact, a chained
MatchAs, the whole-subject MatchOr, a structural sequence/mapping
sub-part, a bare wildcard contributing no candidate) plus the new
wrapped-structural shape at both a sequence and a mapping position, all
via direct AST reproduction before writing tests.
(2) A literal display indexed at a statically known position/key
((rec.bases_fact,)[0], {"x": rec.bases_fact}["x"]) fell through
_is_fact_typed_expr() entirely -- ast.Subscript had no branch at
all. Fixed with a new Subscript branch: a literal ast.Tuple/
ast.List indexed by a literal integer resolves to that element (a
Starred element anywhere disqualifies the whole display, the identical
rule _paired_unpacking_candidates() already applies to a starred value
display); a literal ast.Dict indexed by a literal key resolves to the
matching value (a **expansion entry disqualifies the whole display,
the identical rule). A negative literal index ([-1]) needed its own
unwrap: Python parses it as ast.UnaryOp(op=ast.USub(), operand=
ast.Constant(...)), not a bare ast.Constant -- caught by this fix's
own first round of empirical verification (the negative-index repro
failed against the initial implementation, before any test was written
for it) rather than shipping the gap silently. Composes for free with
every other recursive shape (IfExp/BoolOp/NamedExpr/a nested
resolved element), verified unaffected. One residual, documented in
the function's own docstring rather than chased further: this does
not recurse into the display itself when it is a further, resolvable
Subscript (((rec.bases_fact,), 1)[0][0], two levels of indexing
before ever reaching a literal display) -- doubly-indirect subscript
chaining over a literal display has no real precedent in this codebase,
unlike the single-level form the reported finding actually named.
(3) def f(Fact, other): return Fact(1) == other -- an ordinary
parameter reusing the constructor's own name -- was still treated as the
real constructor. _is_fact_typed_expr()'s constructor-call
recognition is a pure, scope-blind lookup against a single whole-tree
fact_names set, with no shadow-awareness at all, unlike the shadowing
this module already applies to alias names via _fact_aliases().
The first fix attempt was wrong, caught before landing by direct
reproduction against a genuine import (this file's own "verify every
sibling form" discipline catching a self-inflicted bug, not just the
reported one): reusing _fact_aliases()'s own broader internal
locally_bound set (returned as a new second value from that function)
seemed like the natural, already-computed answer -- but that set also
records every ast.ImportFrom binding, and a genuine from abicheck.
model.fact import Fact (the correct, ordinary way to bring the real
constructor into scope at all) is itself exactly such a binding.
Subtracting that broader set silently treated the legitimate import that
establishes Fact as though it shadowed Fact, disabling constructor
recognition module-wide the moment any file imported it normally --
confirmed by direct reproduction (from abicheck.model.fact import
Fact; def f(other): return Fact(1) == other stopped being detected at
all) before a single test was written, and reverted in full. Fixed
instead with a new, deliberately narrower _locally_bound_parameter_
names(), collecting only function/lambda parameter names (never
imports or assignment targets) per function's own qualname -- a
parameter is never a legitimate way to bind the real Fact class,
unlike an import, so this collector carries no equivalent risk. Also
needed closure-scope inheritance the parameter-only collector doesn't
give for free: def outer(Fact): def inner(other): return Fact(1) ==
other -- inner's own parameter set is empty, since Fact is
outer's parameter, but inner still genuinely closes over it -- so a
new _shadowed_constructor_names() walks the real lexical-function-
parent chain (_lexical_function_parents(), the identical closure-scope
chain _fact_aliases()'s own alias inheritance already walks), unioning
every ancestor's own bound-parameter set. Verified against the reported
repro, a genuine unshadowed import (both bare Fact and an aliased
Fact as F, both classmethod and plain-constructor spellings -- the
exact regression class the reverted first attempt introduced), a
sibling function with no shadowing parameter still detecting the real
constructor, single/double/class-nested closure inheritance of the
shadow, an unrelated attribute-field-access recognition path staying
unaffected by the shadow, and a shadowed imported-alias name too, all
via direct AST reproduction before writing tests.
Still zero existing hits across all three, mypy/ruff both stayed
clean, fact_detector_misuse.py at 1896 lines and fact_detector_
misuse_scope.py at 1109 lines (both well under the 2000-line hard cap).
New tests split into three dedicated sibling files (none of the
existing files had enough headroom left under their own 1200-line caps
to safely absorb this many new cases): tests/test_fact_detector_
misuse_as_pattern_wraps_structural.py (10 tests),
tests/test_fact_detector_misuse_static_subscript.py (20 tests), and
tests/test_fact_detector_misuse_constructor_shadow.py (11 tests, the
TestGenuineImportStillRecognizedAfterTheFix class specifically pinning
the self-inflicted regression the first attempt introduced, not just
the originally reported finding).
A further Codex review round found two more real gaps in the same two fixes, both closed -- and this round's own instruction was explicit about not repeating the previous round's self-inflicted mistake: verify against a realistic sibling case, not just a synthetic one, before calling either fix done.
(1) _locally_bound_parameter_names() covered only parameters, so
every other local-binding form still shadowed nothing: Fact =
lambda x: x, for Fact in factories:, and a comprehension target named
Fact were all still read as the real constructor and called, even
though Python resolves each to the local value, not model.fact.Fact.
Renamed and widened to _locally_bound_constructor_shadow_names(),
covering parameters, plain/annotated-assignment/walrus/for-loop/
comprehension targets (via _bound_names(), so nested tuple-unpacking
targets are covered too), and imports -- with the identical carve-out
the reverted first attempt's mistake already established, now applied
correctly instead of blanket-excluded: an ast.ImportFrom only counts
as a shadow when its own alias.name is not literally "Fact" -- the
exact structural test _imported_fact_aliases() itself uses to decide a
name belongs in fact_names, so the two functions cannot disagree.
from x import Fact/from x import Fact as F (any source module --
this module's own established "match by name alone" stance) are
correctly not shadows regardless of x; from x import SomethingElse
as Fact (renaming an unrelated import to the exact spelling Fact) is
one, since _imported_fact_aliases() never adds a name to fact_names
for that shape (it requires the original imported name to be "Fact",
not the local alias) -- without the exclusion, a bare ast.Import is
never exempt at all, since _imported_fact_aliases() only ever
recognizes ImportFrom. Deliberately still narrower than _fact_
aliases()'s own general-binding walk in one respect, documented as an
accepted residual rather than reused outright: match/case pattern
captures and with/except ... as targets aren't collected, since each
would need the identical multi-round hardening _fact_aliases()'s own
history already applied to its general collection before it could be
trusted for this independent purpose too -- reusing that machinery
directly is exactly the coupling the first, reverted attempt already
showed is dangerous.
Verified against all three reported shapes, five more sibling forms
(annotated assignment, walrus, a renaming import, a bare import ...
as Fact, a nested tuple-unpacking target), and -- this time -- the
exact realistic sibling case the previous round's instruction named
explicitly: a genuine, unrenamed from abicheck.model.fact import Fact
(and its aliased as F form) confirmed still detected, a classmethod
constructor confirmed still detected, a sibling function with no local
shadow confirmed unaffected, and a shadow correctly not leaking past
its own function into an unrelated one -- all via direct AST
reproduction before writing a single test, closing the exact gap the
first attempt's own skipped step left open. No self-inflicted regression
this round.
(2) The static-subscript fix's own selected element was still passed
through the purely structural _is_fact_typed_expr() alone: fact =
rec.bases_fact; (fact,)[0] == other -- a bare alias inside an otherwise
statically resolvable display -- went unrecognized, since _is_fact_
typed_expr() deliberately never resolves a bare ast.Name (that needs
known/aliases, which a structural predicate alone doesn't have).
Fixed by extracting the resolution step itself into a new
_static_subscript_element() (mirroring _static_display_elements()'s
own "one extraction, several alias-aware/structural callers" shape,
carrying forward the identical negative-index/Starred/**expansion
rules the original fix already stated), then routing it through both
of this module's alias-aware resolvers -- a new Subscript branch in
_candidate_resolves_to_fact() (the fixed-point resolver, recursing
through itself with known in scope) and a new Subscript branch in
fact_equality_misuse_sites()'s own is_fact_typed() (the top-level
comparison-operand resolver, recursing through itself with qualname in
scope) -- instead of only ever landing back in the purely structural
_is_fact_typed_expr(), whose own Subscript branch was simplified to
call the shared extraction helper too.
Verified against both reported repros (tuple and dict displays), the
identical shape with the selected element further assigned to a new
local before comparison (exercising the fixed-point resolver
specifically, not just the top-level one), a chained alias inside a
subscript (fact2 = fact, composing the new Subscript recursion with
the fixed point's own existing alias-chain resolution), a negative-index
alias, a wrong-position alias staying correctly unrecognized, and every
pre-existing literal-display/non-literal-index/starred-display/non-alias
regression re-verified unaffected, all via direct AST reproduction
before writing tests.
Still zero existing hits across both, mypy/ruff both stayed clean,
fact_detector_misuse.py at 1959 lines (only ~41 lines of headroom left
under the 2000-line hard cap -- the very next finding in this area will
likely need a split into a sibling module) and fact_detector_misuse_
scope.py at 1170 lines (well under its own cap). New tests appended to
the two dedicated sibling files these two mechanisms already own (both
had ample headroom, unlike the script file itself):
TestNonParameterShadowsSuppressRecognition (9 tests) in
tests/test_fact_detector_misuse_constructor_shadow.py, and
TestAliasResolutionInsideAStaticSubscript (8 tests) in
tests/test_fact_detector_misuse_static_subscript.py.
A further round tracked assignment aliases of the Fact constructor
itself (Codex review, fresh evidence): F = Fact or make_fact =
Fact.present, followed by F(1) == other/make_fact(1) == other,
went unrecognized -- F/make_fact is a genuine, if local, name for
the identical constructor, not merely a shadow of it, but neither
_is_fact_typed_expr()'s scope-blind lookup nor the shadow-tracking
fix above had any notion of a local rebinding extending the
constructor set. Reproduced both sub-cases directly (0 hits, expected
1 each) before designing a fix.
Given fact_detector_misuse.py's tight headroom (1959/2000 lines),
both new collectors -- _constructor_alias_names() (F = Fact: a
bound name behaves exactly like Fact itself, in both call and
further-attribute-access position, so it's added into the effective
fact_names set the same way a shadow is subtracted) and
_constructor_method_alias_names() (make_fact = Fact.present: an
unbound classmethod reference, tracked as its own, separate set,
recognized only against a direct ast.Call whose func is a bare
ast.Name, never composed into the general substitution set -- a
further attribute access off it, make_fact.foo, would be nonsensical)
-- went into fact_detector_misuse_scope.py, which had ample room. The
existing hand-rolled closure-scope-chain walk in fact_detector_misuse.
py (_shadowed_constructor_names()) was also generalized into a new
shared _scope_chain_union() helper (in the scope module) and reused
for both the shadow set and the two new alias sets, netting a small
reduction in the primary file's own line count that offset most of the
new wiring code -- both fixed-cap files stayed within budget without a
module split this round. Both F = Fact and make_fact = Fact.present
are recognized only via a plain single-target ast.Assign (no type
inference, "one hop only" -- a further transitive rename G = F is a
documented, accepted residual, the identical limit already accepted
elsewhere in this module's alias tracking).
Proactive sibling verification for this round found a real false
positive of its own, before it shipped, not a reported finding:
def f(Fact, other): F = Fact; return F(1) == other has a parameter
named Fact shadowing the real constructor for the whole function, so
F = Fact binds F to that parameter's runtime value, not to the
real constructor -- registering F as a constructor alias
unconditionally would have fabricated a misuse site out of an unrelated
local rebinding, and the identical hazard applies to the classmethod-
alias form. Fixed by threading locally_bound_shadows/lexical_parents
into both new collectors and skipping registration whenever the RHS
name is itself shadowed anywhere in its own closure-scope chain (via
the same _scope_chain_union() walk), including through a nested
closure over the shadowing outer scope.
Verified via direct AST reproduction against 20 cases before writing
any test: both reported sub-cases, a classmethod alias of a different
method, aliasing through an already-import-aliased name (both bare and
classmethod forms), nested-closure inheritance of both alias kinds, a
sibling function unaffected, a bare alias composing with a further
attribute call, the transitive (two-hop) alias correctly not chased,
both alias kinds correctly not recognized via tuple-unpack (not a
single target), two negative controls (aliasing an unrelated name,
aliasing a non-Fact attribute access), both pre-existing regressions
(bare import, classmethod constructor) still recognized, the
shadow-guard false positive fixed in both forms, and the shadow-guard
holding through a nested closure. mypy/ruff both stayed clean;
fact_detector_misuse.py at 1966 lines and fact_detector_misuse_
scope.py at 1300 lines, both still under their respective caps. New
tests: TestConstructorAssignmentAliasesAreRecognized (13 tests) and
TestShadowedConstructorSuppressesItsOwnAlias (3 tests), both appended
to tests/test_fact_detector_misuse_constructor_shadow.py.
A further round found the enclosing-shadow walk itself was too blunt
(Codex review, fresh evidence): def outer(Fact): def inner(other):
global Fact; return Fact.present(1) == other -- inner's own global
Fact statement is Python's ordinary override of the closure rule for
that name: it routes every reference to Fact inside inner straight
to module scope, completely bypassing outer's own shadowing parameter.
_scope_chain_union()'s unconditional walk had no notion of global
statements at all and still unioned outer's shadow in, suppressing a
genuine misuse site of the real, unshadowed constructor. Reproduced
directly (0 hits, expected 1) before designing a fix.
Fixed with a new _global_declared_names() collector in
fact_detector_misuse_scope.py (maps each function's own qualname to
the names it declares via a direct global statement in its own body)
and an optional global_names parameter on _scope_chain_union()
itself: a name the starting qualname declares global is excluded
from every non-module ancestor's own contribution to the walk, while
still receiving whatever "<module>"'s own entry says, since the walk
reaches the module scope as its terminal node regardless. Threaded
through all three call sites that need it -- is_fact_typed()'s own
shadow-subtraction, and both _constructor_alias_names()'s and
_constructor_method_alias_names()'s own internal shadow checks (the
identical hazard applies to an alias sourced from a global-declared
name: global Fact; F = Fact must still register F as a real alias).
The alias-addition half of is_fact_typed()'s own union (constructor
aliases bound in an ancestor scope, reached via ordinary closure) is
correctly left unaffected -- global changes what a name resolves to
within the declaring scope, not whether a different scope's own local
binding remains visible to its own nested closures the normal way.
Verified via direct AST reproduction against 10 cases before writing
tests: the reported case, the bare-call form, both new alias
collectors' own shadow checks correctly bypassed too, a doubly-nested
case where only the innermost scope declares global (bypassing two
levels of shadowing parameter at once), a regression guard confirming
the shadow still suppresses recognition with the global statement
removed, a global declaration with no shadowing parameter left
completely unaffected, sibling-function isolation, global of an
unrelated name correctly not bypassing a genuine Fact shadow (the
bypass is keyed by name, not merely "a global statement exists in this
scope"), and the subtle case where the module-level Fact itself is
shadowed by an ordinary module-level assignment (global only means
"look at module scope" -- it doesn't mean the module-level binding is
the real constructor, so recognition correctly stays suppressed there
too). mypy/ruff both stayed clean; fact_detector_misuse.py at 1974
lines (headroom now tight -- ~26 lines left under the 2000-line hard
cap; the next finding in this area will very likely need a further
split) and fact_detector_misuse_scope.py at 1356 lines (still ample).
New tests: TestGlobalDeclarationBypassesEnclosingShadow (10 tests),
appended to tests/test_fact_detector_misuse_constructor_shadow.py.
A further round found the shadow collector recorded only a def's
own parameters, never a nested def/class statement's own name
(Codex review, fresh evidence): def outer(other): def Fact(x): return
x; return Fact(1) == other was still read as the real constructor,
since nothing recorded the nested function's own name as a binding in
its containing scope -- the identical STORE_NAME/STORE_FAST rule an
ordinary assignment already gets. Reproduced directly for both the
def Fact and class Fact forms (0 hits each, expected 0 -- i.e. both
were false positives: the constructor was wrongly recognized) before
designing a fix.
Fixed entirely inside fact_detector_misuse_scope.py's existing
_locally_bound_constructor_shadow_names(): resolves each FunctionDef/
AsyncFunctionDef/ClassDef node's own containing qualname via the
already-existing _def_containing_qualnames() helper (already used
elsewhere in this module for the identical "a def/class statement's own
name binds into whatever namespace textually contains it" question --
distinct from the closure-scope chain _lexical_function_parents
answers, since a def/class statement's binding target is never about
free-variable lookup) and records the definition's own .name there.
No change to fact_detector_misuse.py itself was needed at all --
deliberate, given that file's own tight remaining headroom (1974/2000
lines) after the previous two rounds.
Verified via direct AST reproduction against 10 cases before writing
tests: both reported forms, each correctly scoped to its own containing
function only (a sibling function unaffected), a further-nested closure
correctly inheriting the def-shadowed name, a module-level def Fact
also shadowing, both pre-existing regressions (bare import, classmethod
constructor) still recognized, an unrelated nested def helper(...)
correctly not affecting recognition, and the trickiest composition --
a sibling nested def Fact in the same enclosing function that would
otherwise shadow it, with an inner global Fact still correctly
bypassing straight to module scope, confirming the two mechanisms
(this round's def/class shadow and the previous round's global bypass)
don't interfere with each other. mypy/ruff both stayed clean;
fact_detector_misuse.py unchanged at 1974 lines and
fact_detector_misuse_scope.py at 1369 lines (still ample headroom).
New tests: TestNestedDefinitionsShadowTheConstructorName (10 tests),
appended to tests/test_fact_detector_misuse_constructor_shadow.py.
A further round found _fact_aliases()'s own annotation recognition
had no shadow awareness at all (Codex review, fresh evidence): from
other_model import Value as Fact; def f(value: Fact, other): return
value == other -- the identical renaming import the constructor-call
path already recognizes as a shadow -- still marked value as
Fact-typed, since _is_fact_typed_annotation() (used for both a
parameter's own annotation and an AnnAssign's annotation) was called
with the raw, whole-tree fact_names set directly, with no per-scope
subtraction at all. Reproduced directly (1 hit, expected 0) before
designing a fix.
Fixed by reusing the exact same shadow machinery the constructor-call
path already built (_locally_bound_constructor_shadow_names(),
_global_declared_names(), _scope_chain_union()) rather than
building a second, parallel mechanism: computed once inside
_fact_aliases() itself (a small _effective_fact_names(qualname)
closure) and applied at both annotation call sites. Deliberately reused
rather than duplicated -- the two paths (constructor-call recognition
in fact_equality_misuse_sites(), and annotation recognition in
_fact_aliases()) answer the identical underlying question ("does the
bare identifier Fact at this scope refer to the real constructor, or
something else"), so a second independent implementation would risk the
two silently disagreeing the way the very first self-inflicted
regression in this saga already showed is dangerous.
Verified via direct AST reproduction against 11 cases before writing
tests: the reported parameter-annotation case, the equivalent
AnnAssign form, a parameter literally named Fact correctly
suppressing a sibling parameter's own annotation in the same function,
nested-closure inheritance of an annotation shadow, sibling-function
isolation (an unrelated function's own local shadow not leaking into a
different function's real annotation), the global bypass composing
correctly with annotation resolution too, four pre-existing regressions
(bare, subscripted, Optional-wrapped, and stringized annotations all
still recognized), and a negative control (an annotation naming an
unrelated type, unaffected either way). mypy/ruff both stayed
clean; fact_detector_misuse.py at 1993/2000 lines -- only ~7 lines
of headroom left, confirming the previous round's own prediction that
this file would need a further split very soon; fact_detector_misuse_
scope.py unchanged (this fix needed no new scope-module code, only
reuse of what already existed there). New tests:
TestAnnotationsResolveThroughTheSameShadowMachinery (11 tests),
appended to tests/test_fact_detector_misuse_constructor_shadow.py.
A further round found _static_subscript_element()'s own dict-key
resolution used a forward scan, returning the first matching key
rather than the last (Codex review, fresh evidence): a real Python
dict literal keeps the last value for a repeated (or merely
==-equal, e.g. 1/True) key -- ordinary dict-construction overwrite
semantics -- but {"x": Fact.present(1), "x": 0}["x"] was still
reported as a misuse site even though the value actually selected at
runtime is 0, not the Fact value. Reproduced directly (1 hit,
expected 0) before designing a fix. Fixed by scanning the full
zip(display.keys, display.values) sequence and keeping the last
match (via a match accumulator) rather than returning on the first
one -- the identical ==-based key comparison the loop already used is
what makes the 1/True collision resolve correctly too, with no
separate logic needed for it. Verified against 8 cases via direct AST
reproduction before writing tests: the reported case, the reversed
case (Fact value as the last, correctly flagged), both directions of
the True/1 equality collision, a triple-duplicate key, and
regression guards (single non-duplicate key, wrong key, tuple/list
subscript resolution all unaffected). New tests:
TestLastMatchingKeyWins (6 tests), appended to
tests/test_fact_detector_misuse_static_subscript.py.
This fix pushed fact_detector_misuse.py to 1999/2000 lines --
genuinely out of headroom (the previous two rounds had already
flagged this as imminent). Rather than waiting for a third finding to
force an emergency split mid-fix, this round performed the split
proactively, per the standing instruction: fact_detector_misuse.py's
entire "does this expression/annotation/name resolve to a Fact[T]
value" machinery -- FACT_FIELD_NAMES, _imported_fact_aliases,
_is_fact_typed_expr, _static_display_elements,
_static_subscript_element, _admissible_loop_element,
_candidate_resolves_to_fact, _annotation_head_name, _is_fact_typed_
annotation, _fact_aliases -- moved to a new sibling module,
fact_detector_misuse_aliases.py, mirroring the exact precedent
fact_detector_misuse_scope.py's own split already established
(mechanical extraction, unchanged function bodies, not a redesign).
fact_detector_misuse.py itself now holds only the top-level scan entry
point (fact_equality_misuse_sites/check_fact_detector_misuse) and
the def-time default/annotation scope-override machinery
(_default_and_annotation_scope_overrides/_iter_default_subtree's own
caller). One further wrinkle the split surfaced: _iter_default_
subtree() (and the _SCOPE_INTRODUCING_NODE_TYPES constant it uses)
turned out to be needed by both remaining files -- the def-time
scope-override machinery that stayed in fact_detector_misuse.py, and
_fact_aliases()'s own pending-default resolution that moved to the
new module -- so it relocated a second level up, into
fact_detector_misuse_scope.py (the module both siblings already
import from), rather than either duplicating it or creating a new
import cycle. FACT_FIELD_NAMES/_imported_fact_aliases/_is_fact_
typed_expr/_static_subscript_element/_fact_aliases are re-exported
by fact_detector_misuse.py via the explicit X as X spelling
checker_policy.py already uses for ChangeKind, so every existing
from .fact_detector_misuse import FACT_FIELD_NAMES call site
(including this check's own test suite) is unaffected -- confirmed by
re-running the full test suite unchanged and reloading the module
directly. Resulting line counts: fact_detector_misuse.py 381 lines
(from 1999), fact_detector_misuse_aliases.py 1614 lines (new),
fact_detector_misuse_scope.py 1434 lines (up from 1369, absorbing
_iter_default_subtree) -- all three now with ample headroom under the
2000-line hard cap for the foreseeable next several rounds.
scripts/CLAUDE.md's inventory table updated with a new row for the
split-out module and revised text for the two existing rows it now
shares responsibilities with.
A further round raised two findings against the same commit; one was fixed, one was recorded as an accepted known gap rather than chased.
(1) Fixed: constructor-call/alias recognition treated any
Fact.<attr>(...) call as a constructor, regardless of which attribute
(Codex review, fresh evidence): Fact.value_or(fact, 0) == expected is
an ordinary, correct unwrap-then-compare -- value_or is an instance
method returning the bare T, never a Fact -- but was flagged as a
misuse, and _constructor_method_alias_names()'s own get =
Fact.value_or alias tracking repeated the identical mistake.
Reproduced directly (1 hit, expected 0) before designing a fix. Fixed
by adding _FACT_CONSTRUCTOR_METHOD_NAMES (fact_detector_misuse_
scope.py) -- the real Fact class's own six @classmethods that
literally return cls(...) (present, partial, not_collected,
unsupported, failed, not_applicable), explicitly excluding
value_or and is_present (a @property returning bool) -- and
checking the called/aliased attribute's own name against it at both
call sites (_is_fact_typed_expr()'s Attribute branch,
_constructor_method_alias_names()'s own check), the identical "no
type inference, match by name alone" stance FACT_FIELD_NAMES already
takes. Placed in the scope module (not the aliases module) because
_constructor_method_alias_names() -- which also needs it -- already
lives there and importing it from the aliases module would have
created a new import cycle (aliases already imports from scope, never
the reverse); _locally_bound_constructor_shadow_names()'s own existing
alias.name == "Fact" literal-string check already established that
this "generic lexical-scope module" isn't actually Fact-name-agnostic
at the code level, so this is consistent with that existing precedent,
not a new exception. Verified via direct AST reproduction against 10
cases before writing tests: the reported case, the aliased form, all
six real constructors still recognized (individually, by name, not just
as a count), a real constructor alias still recognized, is_present
(a non-call attribute access) correctly unaffected either way, an
unrecognized future method name correctly not a false positive
either, and both the constructor and non-constructor forms composing
correctly with the existing Fact[int] generic-specialization
resolution. New tests: TestOnlyRealConstructorMethodsAreRecognized (8
tests), appended to tests/test_fact_detector_misuse_constructor_
shadow.py.
(2) Recorded as a known gap, not fixed: a class body's own execution
order is not modeled. class C: hit = Fact.present(1) == other; Fact
= factory -- Python populates a class namespace statement by
statement, so hit's own Fact.present(1) genuinely resolves to the
real, imported constructor (the Fact = factory rebinding hasn't
executed yet at that point) -- but the whole-class-body shadow
subtraction this module already applies (matching ordinary function
scoping, where a name bound anywhere in the function is local to the
whole function regardless of line order) treats the entire class body
as shadowed the moment any rebinding appears in it anywhere,
including textually after a genuine early use. Reproduced directly
(confirmed: no hit, though a real misuse is present) before deciding
not to chase this. Deliberately left unfixed, for three compounding
reasons: (a) it requires genuine statement-order/execution-order
tracking scoped specifically to class bodies (which execute top-to-
bottom like a script, unlike a function body's real order-independent
static scoping) -- a new analytical capability this module has nowhere
else; (b) it directly contradicts the "no control-flow analysis" design
stance this module's own docstrings state repeatedly and explicitly
(e.g. _fact_aliases()'s own "this check has no control-flow analysis,
and a false positive here is far cheaper than the false negative it
prevents"); (c) the failure direction is a false negative only (a
missed detection, never a blocked legitimate PR) -- the strictly safer
of the two failure modes per this module's own stated philosophy, and
the described pattern itself (a class reassigning the literal name
Fact mid-body to an unrelated factory function) is sufficiently
exotic that the risk of a rushed, under-tested order-sensitive
implementation introducing a new false positive elsewhere outweighs
closing this one narrow miss. If a genuine instance of this pattern is
ever found in the real codebase (this scan still has zero baseline
hits), revisit then with a concrete case in hand rather than a
synthetic one.
A further round found two Codex findings against the same commit that
both traced back to one root cause: the constructor-alias-addition
step (F = Fact) was folded into the effective fact_names set via an
independent _scope_chain_union() walk from the shadow-subtraction
step, rather than one combined, nearest-scope-wins resolution.
(A) Annotation recognition never consulted constructor aliases at
all (Codex review, fresh evidence): F = Fact; def f(value: F,
other): return value == other -- value's annotation names F, a
genuine constructor alias, but _fact_aliases()'s own _effective_
fact_names() closure only ever subtracted shadows from the raw,
whole-tree fact_names set; it never added constructor aliases the
constructor-call path already folds in. Reproduced directly (0 hits,
expected 1) before designing a fix.
(B) An unconditional alias union re-added an alias a nearer scope
shadows (Codex review, fresh evidence): F = Fact; def f(F, other):
return F(1) == other -- f's own parameter F is an ordinary,
unrelated local reusing the outer alias's name, but the old
_scope_chain_union(qualname, constructor_aliases, lexical_parents)
walk unioned in every ancestor's own aliases unconditionally,
re-adding outer's F alias regardless of f's own nearer shadow --
a real false positive. Reproduced directly (1 hit, expected 0) before
designing a fix. Proactive sibling verification (per this repo's
established discipline) found an identical, unreported bug in
_constructor_method_alias_names(): make_fact = Fact.present; def
f(make_fact, other): return make_fact(1) == other had the exact same
symptom, since that check was also a separate, unconditional
_scope_chain_union() walk with no shadow awareness.
Both findings share the identical fix: a new shared primitive,
_resolve_effective_fact_names() (fact_detector_misuse_scope.py,
right after _scope_chain_union()), that folds shadow-subtraction and
alias-addition into ONE combined walk from a starting qualname up
through lexical_parents -- the first (nearest) scope that mentions
a given name, either as a shadow or as a constructor alias, decides
what that name means for every scope between it and the starting
qualname; a farther ancestor's mention of the same name is never
consulted (dict.setdefault-based, so the first write per name wins).
Within one scope, a constructor-alias mention is checked before a
shadow mention -- necessary because the shadow collector records every
assignment target unconditionally (including F in F = Fact itself),
so without this ordering the alias's own defining scope would
incorrectly read as "shadowed" rather than "aliased." The identical
global_names bypass _scope_chain_union() already established
composes unchanged: a globally-declared name still routes straight to
module scope, skipping every intervening scope's shadow and alias
mentions alike.
Wired into three call sites: is_fact_typed()'s own effective_fact_
names (replacing the two independent _scope_chain_union() calls),
the classmethod-alias direct-call check (previously its own separate
_scope_chain_union() membership test -- now _resolve_effective_fact_
names() called with an empty fact_names base, since that check needs
no base set, only "is this exact name a live classmethod alias here"),
and _fact_aliases()'s own _effective_fact_names() closure (which now
also computes _constructor_alias_names() internally, since annotation
resolution has its own independent qualnames/lexical_parents and
needed the same alias data the constructor-call path already had).
_scope_chain_union() itself is now unused in fact_detector_misuse.py
and fact_detector_misuse_aliases.py (both imports removed) but remains
in active use elsewhere in the scope module.
Verified via direct AST reproduction against 15 cases before writing
tests: both reported findings, the classmethod-alias sibling, both
alias forms still recognized without a shadow present, the alias still
recognized at its own defining scope (the same-scope tie-break),
an alias-annotation shadowed by a sibling nested parameter, a nested
closure inheriting an alias into its own annotation, the AnnAssign
annotation form, generic-specialized (F[int]) annotations, and five
pre-existing regressions (real Fact annotation, renaming-import
shadow in annotation, global bypass composing with annotations, bare
constructor, parameter-shadow-of-Fact-itself). mypy/ruff both
stayed clean; fact_detector_misuse.py at 401 lines,
fact_detector_misuse_aliases.py at 1645 lines,
fact_detector_misuse_scope.py at 1520 lines (newly past the 1500-line
soft limit -- a WARN, not an ERROR, with ample headroom left under
the 2000-line hard cap). New tests: a new dedicated file,
tests/test_fact_detector_misuse_nearest_scope_alias.py (15 tests,
TestAnnotationRecognitionConsultsConstructorAliases +
TestNearestScopeWinsForConstructorAliases) -- a new file rather than
appending to test_fact_detector_misuse_constructor_shadow.py, which
had grown to 1049/1200 lines and was getting tight.
A further round found the alias collectors missed two more binding shapes, both against the same commit.
(C) ast.AnnAssign bindings were invisible to both collectors
(Codex review, fresh evidence): make_fact: Callable[..., Fact[int]] =
Fact.present (or F: type[Fact[int]] = Fact) is an ast.AnnAssign,
but _constructor_alias_names()/_constructor_method_alias_names()
only ever matched ast.Assign. Reproduced directly (both bare and
classmethod forms, 0 hits each, expected 1) before designing a fix.
(D) A generic-specialized receiver was resolved by the direct-call
path but not the alias-collection path (Codex review, fresh
evidence): make_fact = Fact[int].present -- the direct call
Fact[int].present(...) was already recognized (_is_fact_typed_expr()
already unwraps a single Subscript receiver), but the alias
collector required a bare ast.Name receiver, so binding it to a name
first and calling through that name bypassed the gate. Reproduced
directly (both bare and classmethod forms, 0 hits each, expected 1)
before designing a fix.
Both findings share the identical fix: two small shared helpers in
fact_detector_misuse_scope.py, right before _constructor_alias_
names() -- _single_target_binding() (returns (target, value) for a
single-target ast.Assign or a valued ast.AnnAssign, None
otherwise) and _unwrap_generic_receiver() (unwraps a single
ast.Subscript receiver, the identical rule _is_fact_typed_expr()'s
own constructor-call recognition already applies). Both collectors' own
walks now check isinstance(node, (ast.Assign, ast.AnnAssign)) before
calling _single_target_binding() (needed for mypy's own type
narrowing on node.lineno/node.col_offset, since ast.walk() yields
the un-narrowed ast.AST base type) and unwrap the RHS receiver before
the existing fact_names membership check.
Verified via direct AST reproduction against 12 cases before writing
tests: both new findings in both bare and classmethod-alias forms, the
two combined (AnnAssign + Subscript together), a bare annotation
with no value correctly registering nothing, both new forms correctly
still suppressed by a parameter shadow, and three pre-existing
regressions (plain Assign bare and classmethod aliases, tuple-unpack
still not recognized). mypy/ruff both stayed clean;
fact_detector_misuse_scope.py at 1574 lines (still comfortably under
the 2000-line hard cap, though now further past the 1500-line soft
WARN threshold). New tests: TestAlternateBindingShapesAreRecognized
(10 tests), appended to tests/test_fact_detector_misuse_nearest_scope_
alias.py (still well under its own 1200-line cap).
A further round found a comparison embedded inside a deferred
annotation was still flagged as a real misuse site (Codex review,
fresh evidence): def f(x: Annotated[int, Fact.present(1) ==
sentinel]): ... under from __future__ import annotations (PEP 563)
-- the repository-mandated convention AGENTS.md requires throughout
abicheck/, this check's own real scan target -- stores every
annotation as source text, never evaluating it at runtime, so that
embedded comparison never actually executes. The unconditional
ast.Compare walk in fact_equality_misuse_sites() had no notion of
this and flagged that dead code as if it were live. Reproduced directly
(1 hit, expected 0) before designing a fix.
Fixed with two small, module-local helpers in fact_detector_misuse.py
itself (not the scope or aliases module -- this is neither a lexical-
scope-resolution concern nor a Fact-typedness-resolution concern, just
"which Compare nodes are dead code here"): _module_has_deferred_
annotations() (checks for a module-level from __future__ import
annotations) and _deferred_annotation_compare_ids() (collects the
id() of every ast.Compare found inside a parameter, return, or
variable annotation's own subtree, gated on the future import actually
being present -- empty otherwise, so a module without PEP 563
deferral keeps every embedded comparison as a genuine site, since it
really does execute at def-time there). Deliberately independent of
_default_and_annotation_scope_overrides()'s own similarly-shaped
subtree walk just above it in the file, even though both visit the
identical parameter-annotation/returns shape: that function's own
overrides dict conflates default-value, decorator, and class-base/
keyword subtrees together with annotation subtrees into one id set, and
only annotations are ever deferred by PEP 563 -- a default value,
decorator, or class base always evaluates eagerly regardless of the
future import, so reusing that dict's keys would have wrongly excluded
a genuine comparison inside one of those other subtrees too.
Verified via direct AST reproduction against 8 cases before writing
tests: the reported case, the equivalent return-annotation and
variable-annotation (AnnAssign) forms, a nested function's own
annotation, the negative control confirming the identical comparison
stays flagged without the future import, a real body-level comparison
still flagged, a default-value comparison (never deferred, regardless
of the future import) still flagged, ordinary Fact-typed annotation
type recognition (a separate mechanism entirely) left unaffected, and
an AnnAssign's own value (as opposed to its annotation) confirmed
never deferred either. mypy/ruff both stayed clean;
fact_detector_misuse.py at 469 lines (still ample headroom). New
tests: a new dedicated file, tests/test_fact_detector_misuse_
deferred_annotations.py (9 tests,
TestDeferredAnnotationsExcludeEmbeddedComparisons).
A further round found two more findings against the same commit: one a real bug, one a documentation-completeness gap.
(E) A literal None dictionary key was conflated with "the index
couldn't be resolved at all" (Codex review, fresh evidence):
_static_subscript_element()'s own slice-resolution used a single
index: object | None sentinel for both "resolved to the constant
value None" and "nothing resolved" -- so {None: Fact.present(1)}
[None], a perfectly ordinary None-keyed dict lookup that genuinely
selects the Fact value, silently declined to resolve at all,
bypassing the gate. Reproduced directly (0 hits, expected 1) before
designing a fix. Fixed by tracking whether the index was actually
resolved in a separate resolved: bool, rather than overloading
index is None for both meanings -- if not resolved: return None
replaces the old if index is None: return None. Verified via direct
AST reproduction against 10 cases before writing tests: the reported
case, the identical lookup via an intermediate alias, a None key
present but a different key selected (still correctly ignored), a
None key winning a duplicate-key tie as the last entry (composing
correctly with the earlier "last matching key wins" fix), and five
regressions -- **expansion still disqualifying the whole display even
with a None key present, a non-literal index still unresolvable, a
tuple/list indexed by None correctly staying unresolvable (a real
TypeError at runtime, unlike the dict case this fix addresses), and a
falsy-but-genuinely-resolved index (0) not mistaken for "unresolved"
either (the resolved flag's own regression guard). New tests:
TestNoneIsAValidDictionaryKey (7 tests), appended to
tests/test_fact_detector_misuse_static_subscript.py.
(F) The fact-detector-misuse gate was registered in scripts/
CLAUDE.md's inventory table but never added to the root AGENTS.md's
own canonical AI-readiness gate table (Codex review, fresh evidence)
-- the two tables serve different audiences (scripts/CLAUDE.md
documents what the script does and who imports it; AGENTS.md's
table is the canonical, single-page enumeration of every enforced
AI-readiness check), and this PR's original commit updated only the
former. Fixed by adding a fact-detector-misuse row to AGENTS.md's
table, alphabetically ordered immediately before the existing
fact-field-readers row (matching CHECKS dict's own ordering in
scripts/check_ai_readiness.py), summarizing the check's rule, its
three-file implementation, and its no-baseline/unconditional-error
posture. python scripts/check_docs_contract.py and python scripts/
check_ai_readiness.py both re-run clean at their existing baselines
(0 errors, 2/148 warnings respectively) after the addition.
Detector migration (fourth slice) — landed, closing this phase. Every
site the KNOWN_UNMIGRATED_READERS baseline recorded — 104 keys across 20
modules at the point this slice started — has migrated off the bare legacy
attribute onto its Fact[...] sibling; the baseline is now the literal
empty set (frozenset()), and the scan (unmigrated_fact_reader_sites/
check_fact_field_readers) stays exactly as live as before — a genuinely
new direct read anywhere under abicheck/ still fails the gate on sight,
with nothing left to exempt it. This is the exact gap the Status section of
docs/contribute/adr/063-one-semantic-pipeline.md previously recorded
("No detector has migrated to read the converted fields' Fact[...]
siblings yet") — closed, not deferred.
The migration is representation-only, per this phase's own Design
section and Acceptance criteria stated above — it does not change any
detector's emitted findings. The mechanism is a single, provable
invariant bridge_legacy_and_fact (this phase's own compatibility bridge,
landed in the first slice) already establishes for every constructed
RecordType/Param instance: the retained legacy field always equals
<field>_fact.value when <field>_fact.is_present, and the field's own
normal resting value ([] for bases/virtual_bases/vtable, None for
vptr_offset_bits, False for is_va_list) otherwise — by construction,
not by convention, since __post_init__ is the only place either
representation is ever written after construction. abicheck/model/fact.py
gained one new function, resolved_fact_value(fact: Fact[T] | None,
default: T) -> T, expressing exactly this: fact.value_or(default) when
fact is not None, else default (the Fact[T] | None union on the
parameter — not Fact[T] alone — matches the field's own declared type;
see below for why that union matters). Every migrated call site replaces
rec.bases with resolved_fact_value(rec.bases_fact, []) — a pure
re-spelling of the identical value the legacy field already held, verified
by the full test suite staying green (mypy 0 errors, ruff clean, the
complete non-integration/non-slow/non-golden suite unchanged) and by the
FP-rate/tier-accuracy gates staying at their existing baselines with no
new corpus case needed. This is deliberately not the same thing as
Fact.value_or(...)'s own documented "not detector-safe" warning
(Design section, above) being quietly ignored — that warning is about a
detector inventing a collapsed default that discards information the raw
legacy field never had ("old.vtable_fact.value_or([]) != new.vtable_fact.
value_or([]) reintroduces... the exact ambiguity"); resolved_fact_value
used at one of these 104 sites reproduces a value that already existed,
observable at the exact same fidelity, one hop away, on the still-live
.status of the very Fact[...] object each call reads from. A future
caller wanting genuine per-status handling still matches on .status
directly — this phase does not remove that capability, only stops forcing
every reader to go through the ambiguous legacy name to get an ordinary
value.
Why resolved_fact_value takes Fact[T] | None, not Fact[T] — a real
mypy gap the first attempt at this slice missed. RecordType.bases_fact
(and its four siblings) are declared Fact[list[str]] | None, not
Fact[list[str]] — required by this phase's own direct-construction
compatibility bridge (Design section: the field's real default is None,
distinct from Fact.not_collected(), so __post_init__ can tell "caller
passed nothing" apart from "caller explicitly asserted no evidence"). A
first pass at each migrated call site wrote the natural-looking
rec.bases_fact.value if rec.bases_fact.is_present else [] directly —
mypy correctly rejected essentially every one of them
(Item "None" of "Fact[list[str]] | None" has no attribute "is_present"):
the field is provably never None on a fully-constructed instance
(__post_init__ always resolves it), but mypy has no cross-instance view
of that invariant and must assume the declared type. resolved_fact_value
closes this once, centrally, rather than repeating an assert fact is not
None (or an equivalent ... or [] narrowing hack) at every one of the 104
sites — a real, if narrower, instance of this phase's own governing
principle (one concept, one representation) applied to the narrowing
step itself, not only to availability.
Two modules crossed the AI-readiness file-size hard cap purely from
this slice's added local-variable lines, and both were split the same way
this codebase already splits an oversized module — a fifth slice, not a
new design. diff_types.py (2018 lines) and dwarf_snapshot.py (2041
lines) each exceeded the unconditional 2000-line limit once every migrated
site's resolved_fact_value(...) assignment line was added; diff_types.py
already had three such sibling splits before this phase
(diff_types_abicc_parity.py/diff_types_field_facts.py/
diff_types_surface.py), so this is the same pattern, not a new one.
diff_types_vtable.py takes the self-contained TYPE_VTABLE_CHANGED
evidence-gating cluster (_vtable_transition_is_evidenced/
_vtable_transition_rests_on_unresolved_evidence/
_layout_evidence_is_unverifiable/_owned_virtual_signatures/
_owned_virtual_signatures_for_record/_diff_type_vtable) — every one of
these six functions was already only ever called from within this same
cluster or from the one external call diff_types.py makes into
_diff_type_vtable, so the split needed only one import back
(from .diff_types_vtable import _diff_type_vtable as _diff_type_vtable),
mirroring the as-aliased re-export convention the three pre-existing
siblings already use. dwarf_snapshot_datasources.py takes
show_data_sources and its three private formatting helpers
(_evidence_layer_line/_coverage_row_summary/_evidence_payload_summary)
— a human-readable L0-L5 diagnostic renderer with zero dependency on
_DwarfSnapshotBuilder or anything else in dwarf_snapshot.py — with
show_data_sources re-exported (as-aliased) from dwarf_snapshot.py so
cli_datasources.py/cli_dump_helpers.py/workflows/extraction.py and
their tests, all of which import it as from abicheck.dwarf_snapshot import
show_data_sources, are unaffected. Neither split changes any behavior,
only which module owns the code; both land at 1579 and 1915 lines
respectively, with headroom under the cap.
A second, narrower growth surface -- ADR-061's own architecture/debt.yaml
no-growth ledger, not the AI-readiness file-size cap above -- needed a
different fix, not a module split. Six already-no_growth-tracked files
(contract_evidence_collect.py/dumper_scoping.py/export_surface.py/
internal_leak.py/surface.py/type_reachability.py) each grew 2-9 lines
past their frozen adoption baseline purely from the two-line
bases = resolved_fact_value(...) / virtual_bases = resolved_fact_value
(...) local-variable pattern this migration otherwise uses everywhere --
architecture/debt.yaml's own README states the ledger "should only
shrink," so raising six baselines to absorb this migration's own overhead
would be exactly the kind of undocumented exception this ADR's Governing
Invariant section rejects. Splitting a module was not the fix here either
(none of the six is anywhere near the 2000-line hard cap the split above
addresses; architecture/debt.yaml's own per-file ceiling is a stricter,
independent budget the same file can be tracked under well before that).
The actual fix: RecordType gained two short, import-free convenience
methods, resolved_bases()/resolved_virtual_bases() (model/entities.py,
right after __post_init__, each a one-line call to the identical
resolved_fact_value primitive) -- a call site with an already-typed
RecordType in hand writes rec.resolved_bases() instead of
resolved_fact_value(rec.bases_fact, []), which needs no new from .model
import resolved_fact_value line at all (the six files above already import
RecordType for their own type annotations) and is short enough to often
collapse the two-line local-variable pattern back to the original
single-line call site the pre-migration code used. This is not a second
representation of the same concept -- both spellings resolve through the
one resolved_fact_value primitive in model/fact.py, and the method
exists purely to avoid an import cost at space-constrained call sites, the
same reasoning Fact.value_or()/bridge_legacy_and_fact() already coexist
under in that same module. The free-function form remains correct and used
elsewhere (diff_cpp_patterns.py's _is_empty_record(t: object) cannot
call a RecordType method at all, since t is deliberately typed object
there). All six files land exactly at their recorded adoption baseline
(1183/1289/1312/1651/1507/1504 lines respectively) with this migration's
reads included -- confirmed via python scripts/check_architecture.py
--base <this-branch's-actual-base-commit>, which is what isolates a PR's
own growth from unrelated already-on-main debt the same way CI's own
ARCHITECTURE_BASE does (a bare local run with no resolvable
origin/main tracking ref, as in a stale/shallow sandbox checkout, can
misattribute pre-existing debt to an unrelated PR -- passing --base
explicitly is the fix, not a ledger change).
With the empty baseline, the empty-baseline test
(tests/test_fact_field_readers.py::test_baseline_entries_are_real_sites)
is vacuously satisfied (there is nothing left to check is "real"), and
tests/test_fact_field_readers.py's own synthetic-violation tests continue
to pin the scan's real behavior against a throwaway abicheck/-shaped tree,
independent of the baseline's size. Phase 0 is therefore complete per its
own stated scope in this document's Goal/Design/Acceptance-criteria
sections above: Fact[T] exists, every one of the five converted fields'
producers construct it directly, every currently-known reader consults it
instead of the ambiguous legacy attribute, and both AI-readiness gates
(fact-detector-misuse/fact-field-readers) are live with nothing left in
either baseline for a known violation to hide behind. What remains
explicitly outside this phase's scope, unchanged: a detector branching on
FactStatus to actually change which findings it emits (Phase 5's job);
converting any field beyond the five this phase named (also Phase 5,
registry-driven); and removing the four retained legacy attributes
themselves, which Phase 10's own checklist already schedules for once the
widened reader check reports zero remaining readers outside the
compatibility bridge's own __post_init__ and serialization — true as of
this slice, but Phase 10 is where that removal actually happens, not here.
Phase 1 — finish the dump/scan typed-API convergence (closes AGENTS.md "PR C")¶
Goal. dump and compare's implicit-dump operand execute through the
same resolve_dump_request/execute_dump_request pair; scan's candidate
resolution executes through the shared workflows.artifact.execute.
_resolve_side_snapshot_impl primitive that pair itself calls internally
(already landed, per AGENTS.md's own record — scan has no DumpRequest-
shaped input for resolve_dump_request/execute_dump_request's own
signature to accept, so it converges one layer lower, not on that pair
verbatim; service_input_resolution is only a delegating facade
re-exporting this module's owner, per that facade's own docstring). No
entry point hand-rolls its own L2 seed, ADR-039 collector call, or AST
cache key.
Design. This is not new design — AGENTS.md's "PR C" note already names the two blockers precisely and one is closed:
- (Closed, carried over from main)
InputSpec.compile_db_filterexists and is threaded throughresolve_dump_request/resolve_compare_request. - (Open — the actual work of this phase) The default header backend
(castxml) must be available in CI/dev environments capable of running
this migration's parity tests; every measurement backing "PR C" so far
is clang-only. Either (a) obtain a working castxml build for the
parity-test lane (this plan's own investigation found conda-forge
0.7.0 segfaulting inside
clang::ParseASTin this environment — file that as its own upstream-castxml investigation, tracked separately, not blocking this phase's clang-only half), or (b) explicitly scope this phase's first landing to the clang backend and track the castxml parity gap as a named residual the same way AGENTS.md already does for every other castxml-unavailable finding in this file. If (b) is taken, this phase's own Acceptance criteria below must be scoped to match, not left stating unqualified convergence —--ast-frontenddefaults to castxml, so landing the typed execution path for clang alone while the default production backend stays on the untouched legacy path meansdump's actual default invocation has not converged, and claiming "every build shape in the parity corpus" passes would silently overstate that for any reader who doesn't separately check which backend each shape ran under. Taking (b) makes this phase's real deliverable "the clang backend converges, verified; the castxml backend is a tracked, named incomplete prerequisite, not a residual detail" — restated explicitly in the Acceptance criteria below, not left implicit in this Design section alone.
Once unblocked: route perform_elf_dump/handle_non_elf_dump through
execute_dump_request — the one remaining routing step (this phase's
worklist is smaller than the Goal above might suggest: scan_engine.
_build_new_snapshot needs no further work here, since its own routing
onto _resolve_side_snapshot_impl already landed per AGENTS.md's own
record). Fold the legacy -p/--compile-db auto-match into the L3→L2
fold as the sole source of compile-database-derived context when the
fold applies (already decided and landed per AGENTS.md's "legacy-match
overlap" entry) rather than re-deciding it here.
A third dump execution path exists, untouched by either of the two
above, and review caught this plan not naming it: the binary-less
dump --sources/--build-info branch (no SO_PATH), which calls
cli_buildsource.dump_source_only() — a pipeline that collects L3-L5
evidence into an otherwise-empty snapshot with no resolve_input call at
all, confirmed by reading the real code. execute_dump_request()
already refuses this shape explicitly (ValidationError when InputSpec.
path is None), with its own docstring stating exactly why this plan must
not paper over: InputSpec.path was deliberately widened to Path | None
so the shape is expressible as a typed request (letting --dry-run
resolve one through resolve_dump_request()), but executing it is "a
genuinely different pipeline... and routing it through here is its own
slice, not part of making the model able to say it" — the exact words the
function's own comment already uses. Landing this phase's two named
routings (perform_elf_dump/handle_non_elf_dump, scan_engine.
_build_new_snapshot) while leaving dump_source_only() as a third,
independent assembler would leave exactly the outcome this phase's own
goal forbids — more than one real dump pipeline after the phase ships,
just one fewer than before. Not migrated in this phase: routing
dump_source_only()'s L3-L5-only collection through execute_dump_
request() needs the executor to support a snapshot with no binary-derived
L0-L2 facts at all, which is a real, separate design question (what does
"the executor's post-processing hooks" — ADR-039's collector, the G31
header-graph attach — even mean for a snapshot with no ELF/PE/Mach-O side
to attach them to?), not a drive-by extension of the two routings this
phase already does. Tracked explicitly as this phase's own named residual,
the same way AGENTS.md tracks its other incomplete-migration findings,
rather than silently left for a future reader to discover was never
covered.
Files. abicheck/cli_dump_helpers.py (perform_elf_dump/
handle_non_elf_dump → call execute_dump_request instead of dumper.
dump() directly, keeping every existing post-processing hook —
ADR-039 collector, G31 header-graph attach, clang-layout-tool attach — as
hooks the executor calls, not logic removed); abicheck/
service_dump_pipeline.py (the executor gains the hook points);
abicheck/cli.py's dump_cmd (already builds a real DumpRequest per
AGENTS.md's record — this phase is where it starts being what actually
runs, not only what --dry-run renders).
Tests. tests/test_dump_cli_typed_api_parity.py's existing
_BUILD_SHAPES/xfail-gated known-divergent-shape mechanism becomes the
acceptance gate: every shape currently marked xfail for a named,
diagnosed divergence must flip to passing, with no new shape added to the
divergent list. A shape that cannot be closed this phase is demoted to a
tracked AGENTS.md "Known gaps" entry with the same rigor the existing ones
carry (a real repro, a named mechanism, not a guess).
Acceptance criteria. dump's CLI path and execute_dump_request
produce bit-for-bit identical snapshots (modulo timestamps/provenance) for
every build shape in the parity corpus under the clang backend, and for
the binary-having (SO_PATH) case only — this phase does not claim
convergence for --ast-frontend castxml, the
actual default, while option (b) above is in force, nor for the
binary-less dump --sources/--build-info shape, which still executes
through cli_buildsource.dump_source_only(), a third pipeline this phase
explicitly does not migrate (per the named residual above); a reader
checking this phase's own status must be able to see "clang + SO_PATH:
converged and verified; castxml: not yet verified; source-only: not
migrated at all" without inferring any of the three from the Design
section. cli_dump_helpers.
render_dump_dry_run() is deleted and --dry-run renders from the real
ResolvedDumpRequest for both backends (the dry-run path itself has no
castxml-specific execution to be blocked on). PR 3C (removing dump
--build-query/--build-compile-db, currently blocked on this per the
plan's own ordering rule) unblocks as a follow-on, not part of this phase,
and — per the same scoping — only once castxml parity closes too, not on
the strength of clang-only convergence alone, since --build-query/
--build-compile-db are reachable under either backend.
Landed (first slice), not the whole phase — read this before assuming
the Design section above is fully implemented. The environmental
precondition the Design section's option (a)/(b) split turned on no
longer holds the way it was written: castxml is genuinely available and
working in the environment this slice was implemented in (a
solver-resolved conda-forge install — clang 20.1.8 + libclang-cpp20.1 +
castxml 0.7.0, wrapped at /usr/local/bin/castxml to carry its
LD_LIBRARY_PATH — not the hand-assembled 0.7.0 build the Design
section's own investigation found segfaulting inside
clang::ParseAST; that specific segfault was an artifact of a bad manual
install, not of castxml 0.7.0 itself). test_dump_cli_typed_api_parity.py
-m integration is 16/16 green — but that file is itself clang-only, not
evidence of castxml coverage: every one of its subprocess invocations
hard-codes --ast-frontend clang and it is not parametrized by backend
(-k castxml against this file alone selects zero tests). What castxml's
newfound availability separately confirmed, run against the wider
integration suite instead (pytest tests/ -m "integration and not slow" -k
castxml), is that abicheck dump --ast-frontend castxml genuinely works
end to end in this environment today (38/38 green, only the two
pre-existing, unrelated xfails) — real, useful confirmation, but not the
same claim this section's acceptance criteria makes about the parity
corpus specifically. That field-level parity was, however, already closed
before this slice started (_CONTRACT_KNOWN_DIVERGENT_FIELDS/
_SCAN_KNOWN_DIVERGENT_SHAPES were both already empty — this slice found
no shape to flip from xfail to passing) — so what this slice actually
landed is narrower: only render_dump_dry_run()'s own migration named
above -- the function itself is not deleted (this section's Acceptance
criteria's "is deleted" language describes the whole phase's eventual
end state, not this slice), only its independent primitive-threading
implementation is. render_dump_dry_run now takes the real ResolvedDumpRequest
resolve_dump_request_for_cli already builds and reads
so_path/headers/sources/build_info/depth/collect_mode/
header_backend/dump_manifest off it, rather than being handed fifteen
independently-threaded primitives dump_cmd re-derived by hand — closing
this phase's own "Files" list item for that function. resolve_dump_request
itself never invokes castxml/clang (see its own docstring), so this
rendering path is backend-agnostic by construction, not merely by test
coverage; verified via the clang-scoped parity corpus above, the full fast
unit suite, and mypy/ruff clean.
The real routing step — perform_elf_dump/handle_non_elf_dump calling
execute_dump_request instead of dumper.dump() directly — was
investigated in this slice and NOT landed, for a reason that turned out to
be independent of castxml availability. Read docs/contribute/
known-gaps.md's "PR C" entry (its own newest addendum, appended by this
slice) for the full account. In short: two of the concerns the Design
section implicitly bundled under "post-processing passes driven by
CLI-only inputs" turned out, on a fresh read of the real code, not to be
blockers at all (the ADR-039 collector's attach_build_context_for_
parsed_headers already runs a second time inside _resolve_side_
snapshot_impl, and re-running it a third time from perform_elf_dump's
own existing hook is a safe, idempotent no-op, not a double-count; the
scope_header_dirs parameter perform_elf_dump's own dump() call
passes is provably redundant with what resolve_dump_request's own
public_header_dirs already carries). A third, genuinely structural
blocker was found instead: dump_cmd's legacy -p/--compile-db
auto-match (cli_helpers_compare._resolve_build_context_flags —
build_context_for_header/build_context_union_fallback, a completely
different code path from the P0.3 L3→L2 fold) runs strictly after
resolve_dump_request_for_cli already built the resolved object, and its
derived flags are the sole source of compile-database-derived context
for any header the L3→L2 fold itself does not independently match — a
still-live, still-documented fallback (dump --build-query/
--build-compile-db/-p/--compile-db are explicitly not yet removed).
resolve_dump_request/_resolve_side_snapshot_impl has no call to that
legacy matcher anywhere, so routing the primary parse through
execute_dump_request() as it stands today would silently drop that
fallback's flags for exactly the headers the fold doesn't match — a real
regression, not a refactor, for a project still relying on the legacy
match. Closing this needs the legacy match's computation moved earlier
(before resolve_dump_request_for_cli runs) and threaded into the
DumpRequest/CompileContext the resolved object carries, which is a
genuine, separate design question (which field absorbs a derived, not
user-typed, value, and how that squares with DumpRequest's documented
"records the run, not a second opinion about it" contract) — not a
same-session drive-by fix, and independent of which AST backend is used.
Not attempted here; recorded in known-gaps.md at the precision needed
for a future slice to start from the actual mechanism rather than
re-diagnosing it. This phase's Acceptance criteria section above is
therefore still not met for the routing half — only for the dry-run
half named in its own last two sentences, which was already correctly
scoped to "both backends" (dry-run resolution never invokes either
compiler, so nothing about it is backend-specific), unlike the routing
half's own now-corrected clang/castxml framing.
Update (2026-08-29): the design question named above is now answered and
landed, but the routing itself is still open. execute_dump_request
gained an additive legacy_compile_db_tokens: tuple[str, ...] = ()
parameter, threaded through _resolve_side_snapshot_impl into
_seeded_includes_and_compile_context — the same "optional pass-through
for dump's still-live CLI legacy flags" shape build_config/
build_query/build_compile_db already use on these same three
functions, rather than a new DumpRequest/CompileContext field. That
answers the design question the paragraph above left open: the legacy
match's derived value stays out of DumpRequest/CompileContext
entirely (preserving "records the run, not a second opinion about it"),
and rides as a separate, explicitly-legacy parameter instead — a caller
(today: nothing yet; dump_cmd itself still executes through
perform_elf_dump, not execute_dump_request) that already computed the
legacy match's flags can now thread them through and have them reach the
real parse, with the P0.3 fold's own result still winning outright
whenever it applies (verified by a dedicated precedence test — see
known-gaps.md's dated addendum to the "PR C" entry for the full
mechanism and the real-g++-build regression test,
tests/test_legacy_compile_db_typed_threading.py). perform_elf_dump
still does not call execute_dump_request() — that remains exactly the
routing restructuring described above (its own try/except/
ResolvedArtifactPlan cleanup handling needs to delegate rather than
call dumper.dump() directly, keeping the second try block's hooks
applied to the returned DumpResult), left as its own slice for the same
reason stated there: this exact code area's review history (18+ numbered
findings on the adjacent fold alone) argues for landing one
independently-verifiable piece at a time rather than combining the
threading and the routing in one change. This phase's Acceptance criteria
section is therefore still not met for the routing half.
Update (2026-08-29, same day): the routing was attempted as its own slice
and NOT landed — and the "purely the control-flow restructuring itself, not
a new correctness question" framing above is retracted as wrong. A
dedicated session read both execution paths and the whole typed pipeline end
to end and built the parameter-by-parameter parity map, then found two
structural blockers neither this section nor the known-gaps entry had
anticipated. Both are recorded in full, with the exact mechanisms and the
list of previously-suspected blockers that were ruled out with evidence, in
docs/contribute/known-gaps.md's "ADR-063 Phase 1" entry (its "Fourth
correction (2026-08-29)" addendum). In short:
- A (ELF only) — the L2 seed's inferred-build-dir cleanup has two
mutually exclusive correct drain points.
perform_elf_dumpmust drain after its header-graph and clang-layout second passes (they re-parse headers under the seeded dirs);_resolve_side_snapshot_implmust drain beforeembed_side_build_source(whose own inferred query otherwise self-contends on the sameflockfor up to 600s). They do not conflict today only becauseperform_elf_dumpruns no embed inside its plan; routing makes the conflict live. This is exactlyDumpResult's own documented "Lifetime caveat", and its fix is the pair-aware/lifetime redesign PR 3A already scopes as separate work. - B (both paths) —
execute_dump_requestembeds L3-L5, applies dependency scoping and enforces the depth floor inside resolution, while thedumpCLI does all three at write time incli_buildsource._write_snapshot_output, after provenance stamping. Routing reorders all three, which changes what the post-processing hooks see, enforces the depth floor before the embed that actually fills L3-L5 for adump, and swaps a Click error for aValidationError.
handle_non_elf_dump is free of A (no second pass; it already drains where
the shared primitive does) but is blocked by B alone, so converting the
smaller PE/Mach-O path first does not isolate a safely-landable slice
either. This phase's Acceptance criteria section remains unmet for the
routing half; closing it now has two named prerequisites (the seed-cleanup
ownership redesign, and a decision on where dump's embed/enforce/scope
stanza belongs) rather than being a mechanical restructuring.
Update (landed, ELF): dump's real ELF run now routes through
execute_dump_request. Both blockers A and B above were closed —
frontends/cli/commands/dump.py's ELF branch now builds a second,
execution-scoped ResolvedDumpRequest (re-pointed at the normalized
so_path, requested_depth nulled so the shared pipeline's own depth gate
never fires ahead of _write_snapshot_output's existing one) and calls
frontends.cli.dump_execute.execute_dump_cli_run, which runs
service_dump_pipeline.execute_dump_request — perform_elf_dump is
retired for this call site (kept defined for its own direct unit tests).
Landed as commit 0b69fc3 ("refactor(cli): migrate dump's real ELF run
onto execute_dump_request (PR C)") plus three Codex-review follow-up
commits (collect-mode/folded-compiler forwarding, keeping the new module
out of the CLI-registration import cycle, and treating an explicit
dump --config/--build-query as trusted operator input). See
docs/contribute/known-gaps.md's "PR C" entry for the exact mechanism
each blocker's fix uses.
Update (2026-09-01, landed): the PE/Mach-O half is closed the identical
way, completing this phase's routing half for both binary formats. The
same structural finding that blocked a from-scratch PE/Mach-O migration
above no longer applies once the ELF slice had already resolved blockers A
and B for the shared pipeline itself — execute_dump_request/
_resolve_side_snapshot_impl were already format-generic
(fmt-parameterized: is_elf=True if fmt == "elf" else None,
pdb_path=side.pdb unconditional, attach_build_context_for_parsed_headers/
embed_side_build_source both format-unconditional), so no second design
investigation was needed — only the caller was still routing PE/Mach-O
around it. frontends/cli/commands/dump.py's PE/Mach-O branch now builds
the identical execution-scoped ResolvedDumpRequest and calls the same
execute_dump_cli_run; handle_non_elf_dump is retired for this call
site (kept defined, unchanged, for its own direct unit tests — the same
discipline perform_elf_dump already follows). Both were deleted
outright on 2026-09-05 by Track 1 of
docs/contribute/plans/duplication-and-convergence-assessment.md, together
with cli_dump_non_elf.py and cli_dump_protocols.py, once "kept alive by
its own unit tests" was all either had left; see that plan's Phase 6 item 1
for what the retirement surfaced. The shared real-run tail
(execute, stamp provenance, write the snapshot) both formats now need was
factored into a new frontends/cli/dump_execute.
execute_and_write_dump_cli_run so commands/dump.py — already at the
architecture gate's 800-line production cap — did not grow net lines when
the PE/Mach-O branch stopped being a single delegating call to
handle_non_elf_dump. Verified via the existing mock-based CLI/unit test
suite (four CLI-dispatch tests that previously monkeypatched
handle_non_elf_dump were rewritten to patch
abicheck.service_dump_native._dump_pe/_dump_macho instead, the same
depth below the format dispatch the pre-existing ELF-side precedent,
test_compile_context_parity.py::test_dump_reads_compile_block_from_config,
already patches abicheck.dumper.dump at) — not a real PE/Mach-O
toolchain end-to-end run: no PE/Mach-O toolchain was available in this
environment to do a byte-for-bit parity check the way the ELF migration's
own tests/test_dump_cli_typed_api_parity.py corpus does for ELF. This
phase's Acceptance criteria section's routing half is therefore now met
for both binary formats that execute through this pipeline at all. The
one remaining, permanent exception is unchanged from this phase's own
Design section: cli_buildsource.dump_source_only() (the binary-less
dump --sources/--build-info path) is explicitly out of scope for this
phase and was not touched — it has no execute_dump_request() call to
route through at all (ResolvedDumpRequest's own docstring: a
binary-less request has no InputSpec.path for execute_dump_request to
resolve), and remains its own separate pipeline.
Phase 2 — EntityId/ScopePath as the one identity primitive¶
Goal. Every place that currently computes identity from a string
(dict key, name, qualified_name, a synthetic ctor/dtor key) instead
computes it through one shared EntityId resolver — "computed once"
here means one algorithm, called the same way everywhere, not a value
cached anywhere on the model; see the carrier note below for why, and for
where genuine per-snapshot caching actually lands.
Design. abicheck/model/identity.py: ScopePath (an immutable tuple
of typed segments — Namespace(name), Record(name, access),
InlineNamespace(name, version_tag), Anonymous(kind, ordinal),
LocalToFunction(owner)) names only the containing scope, never the
leaf declaration itself.
Each segment type states which of its own fields are identity and which
are payload — a bare @dataclass(frozen=True) would make every field
identity by default, which is wrong for at least one of the five.
Record(name, access)'s access (public/protected/private) is carried
on the segment because a nested record's access is a real fact a
consumer may want, but it is not part of where the nesting scope is —
two snapshots of the same class with a member's access level changed
still name the identical containing scope, and EntityId is what diff
matching keys on (Phase 2's own stated purpose). Making access part of
Record's equality/hash would turn an access-level change into a
spurious identity mismatch — the matcher would see "removed, then added"
at a different EntityId instead of "this declaration changed," for a
property this plan does not intend identity to track. Record therefore
defines __eq__/__hash__ over name alone (access stays a plain,
non-identity field, the dataclass equivalent of field(compare=False)).
Anonymous(kind, ordinal)/LocalToFunction(owner) are the opposite case:
both fields are identity, deliberately, since nothing else disambiguates
two sibling anonymous structs or two same-named locals in one function —
an ordinal/owner that is dropped from identity would silently
re-introduce exactly the sibling-collision class this phase's own
(ScopePath, kind, leaf_name, extra) correction exists to close, one
level down. ordinal is a deterministic per-parent sequence
number assigned at parse time (the same position-in-the-scope-stack
counter entry.scope's widening already has to track to build
Anonymous segments at all, not a second counter invented for this), not
a DWARF offset or other environment-sensitive value that could differ
between an otherwise-identical old/new pair — deterministic within one
parse, which is what makes it a legitimate disambiguator for two
anonymous siblings that coexist in the same snapshot.
This is not the same claim as "stable across revisions," and a first
draft of this phase's wording did not distinguish the two — review
correctly caught that an ordinal is a within-parse index, not an
across-snapshot identity. Inserting a new anonymous sibling before
existing ones (an ordinary, unremarkable source edit — a new anonymous
union added earlier in a header than existing ones) shifts every later
sibling's ordinal, which changes their ScopePath and therefore their
whole EntityId even though nothing about those later siblings' own
declarations changed — the old/new matcher would read every one of them
as "removed, then re-added at a new identity," exactly the false-positive
shape this plan's own diff-matching discipline exists to eliminate, not
introduce at a lower level. No stable discriminator for this case is
adopted here, and none is asserted as if one were — the two candidates
this codebase already has experience with are each independently
documented, in AGENTS.md's own "Known gaps," as unreliable for this exact
purpose: a source-location anchor (file:line:col, the same shape
AGENTS.md's "lambda-closure churn" entry already names as "per-translation-
unit and compiler-ordering dependent... a rebuilt consumer can fail to
resolve the symbol," reproduced there as a real false-positive source,
not merely a theoretical risk) and a structural/content fingerprint of the
anonymous scope's own members (circular here specifically — those
members' own identity is what ScopePath is being built to resolve, so
fingerprinting them to identify their parent scope has nothing yet to
fingerprint). Reconciliation semantics that treat an ordinal shift as a
rename rather than a removal-and-addition (matching on the shifted
sibling set's relative order, or deferring to a different signal when an
insertion is detected) are a real, separate design this plan does not
attempt to pick under continued review pressure for the third time in
this same section. Until designed, this is an accepted, documented
limitation of Anonymous identity specifically — the same "attempted
twice, reverted twice... accept the... limitation" discipline this
codebase's own AGENTS.md already establishes for comparably-shaped
identity problems (anonymous-type-marker collisions, the ctor/dtor
lambda-closure entries) — not a silent gap this plan is claiming away. Namespace(name)/
InlineNamespace(name, version_tag) are identity on every field
unconditionally — a namespace has no non-identity payload to exclude, and
an inline namespace's version_tag is exactly the dimension ADR-025's own
versioned-inline-namespace-alias handling already keys matching on, so
excluding it here would silently re-widen the v1/v2-shaped collision
that machinery exists to avoid. EntityId therefore always carries the leaf
declaration's own name as an explicit component, for every kind — not
only for functions. A first draft of this phase defined EntityId as
just (ScopePath, kind), which collides any two sibling declarations of
the same kind in the same scope (ns::A and ns::B, two enums, two
variables, two typedefs — the function-overload collision this phase
already fixed once is one instance of this same shape, not a
function-specific special case, and fixing it only for functions left
every other kind exposed). The corrected shape is EntityId = (ScopePath,
kind, leaf_name, extra), where leaf_name is the declaration's own
(unqualified) name for every kind, and extra is kind-specific and empty
for most kinds — () for a record/enum/typedef/constant, the mangled-name
discriminator described below for a variable specifically, and the
callable-signature discriminator described further below for a function
specifically (the one case a bare name is still insufficient, since two
overloads share both scope and name). OccurrenceId (an EntityId plus a
disambiguator for the already-documented "two declarations, one identity"
case ADR-062 Phase 0 already solves at the storage layer — reused here,
not reinvented). Generalizes ADR-046/048's source-graph identity (already
real, USR-based) by making EntityId the single identity both the
flat snapshot and the source graph reference, rather than two graphs with
their own identity schemes that happen to usually agree.
**A variable's EntityId carries its own mangled spelling in extra —
a bare (ScopePath, "variable", leaf_name, ()) is not enough either, for
the identical reason AGENTS.md already states for the pre-existing
matcher this identity replaces: two exported variables sharing the same
scope and leaf name but differing mangled names (e.g. two distinct,
non-overloadable template-instantiation statics, or a declaration-vs-
definition spelling mismatch the mangler doesn't collapse) are two
different exports, not one — "variables enable no alias tier at all ...
a display-name join would hide a real removal" (AGENTS.md's own
finding_identity.py/SymbolIdentityIndex entry). A first draft of this
phase gave every non-function kind the identical empty extra = (),
which collapses exactly that pair into one EntityId and would make
Phase 2's diff_symbols.py migration pair the wrong two variables (or
miss a real removal) wherever it currently relies on
SymbolIdentityIndex's mangled-name-only matching. Fixed the same way
the function case is: EntityId's variable variant is (ScopePath,
"variable", leaf_name, extra=mangled_name) when a mangled name exists
(the common case for any variable with external linkage), falling back to
extra=() only for the genuinely mangling-free case (no linker symbol at
all — e.g. a variable known only from a header declaration with no
corresponding binary evidence), mirroring the function fallback's own
scope for exactly the same reason rather than inventing a second rule.
A function's EntityId carries a callable-signature discriminator —
ScopePath plus a bare name is not enough. f(int) and f(double) share
the same ScopePath and the same function kind discriminator; without a
third component, EntityId collapses two genuinely distinct overloads into
one id, and since this phase directs diff matching and every other
semantic consumer to key on EntityId rather than re-deriving their own
fallback, OccurrenceId's per-record disambiguator (built for the
unrelated "same identity, duplicate declaration" case) does not repair
this — it is not a per-overload discriminator. EntityId does not invent
a new scheme for this: it carries the existing tiered resolution
finding_identity.resolve_function_identity/SymbolIdentityIndex and
ADR-048's normalized identity already establish — mangled name first when
one exists (the common case, already globally unique per overload), and
only for the genuinely mangling-free case (a non-extern "C" function on
a DWARF-only snapshot) the same normalized-signature fallback tuple that
code already computes. That fallback tuple includes the callable's own
qualified name, not only its parameter types and CV-qualifiers — a first
draft of this phase omitted the name, which would have collapsed two
genuinely distinct functions with the same scope and the same parameter
types (ns::f(int) and ns::g(int)) into one EntityId, exactly the
collision class this phase exists to close rather than introduce. The
real primitive, finding_identity.normalized_signature(qualified_name,
kind, param_types), already puts qualified_name first in its tuple for
precisely this reason ("two identically-declared overloads never
collide" — but that guarantee only holds because the qualified name is
in the tuple); this phase's fallback keeps that shape, it does not
narrow it. EntityId's function variant is therefore (ScopePath,
"function", leaf_name, extra=mangled_name | (param_types,
cv_qualifiers)) — leaf_name per the general shape above, extra
carrying exactly the signature discriminator a function additionally
needs — not a bare (ScopePath, "function") and not a signature tuple
with the name left out — generalizing the existing tiered primitive into
the one identity every consumer reads, rather than proposing a simpler one
that regresses what the codebase already gets
right.
There is no new carrier field on RecordType/Function/any other model
dataclass in this phase, and consumers do not read a stored EntityId off
a declaration — both would be the wrong fix, and a first draft of this
phase left the question open rather than answering it, which review
correctly read as "nowhere for the promised identity to live." The
resolver function (model.identity.entity_id_for_record(rec) and its
siblings for enum/typedef/function/variable/constant) derives ScopePath
from structural scope data the parsers already track internally during
the AST walk — not, as an earlier draft of this note claimed, from
RecordType.qualified_name/name alone. That claim does not survive
checking the real parser code: entry.scope (dumper_clang.py's/
dumper_castxml.py's own internal scope-tracking list, built up while
walking the AST, collapsed into qualified_name via "::".join([*entry.
scope, name]) at the point a declaration is finalized) is a plain
list[str] of bare names, with no per-segment kind tag at all — it cannot
distinguish a record nested in a record from the same names nested in a
namespace, or an inline-namespace segment from an ordinary one, because
that distinction was never captured in the first place, not merely
discarded during string-joining. A resolver operating on qualified_name
alone is working from a representation that is structurally insufficient
for ScopePath, not one the resolver fails to parse correctly — no
amount of cleverness in model/identity.py recovers information the
parser itself never recorded. The real fix reaches one layer further
back than the resolver: entry.scope itself is widened, in both
dumper_clang.py and dumper_castxml.py, from list[str] to a list of
typed segment records — each push onto the scope stack (entering a
namespace, a record, an inline namespace, an anonymous scope, a
function-local scope during AST traversal) already has, at that exact
point, the one piece of information qualified_name alone throws away:
which AST node kind it is actually processing (a clang NamespaceDecl
vs. CXXRecordDecl vs. an inline-namespace-tagged NamespaceDecl; a
castxml <Namespace> vs. <Struct>/<Class> XML element), plus
whatever kind-specific data that node already carries (a record's access
specifier, an inline namespace's version tag). Recording that tag
when the scope is entered, rather than trying to reconstruct it later
from the flattened string, is what makes ScopePath constructible at
all — model.identity.entity_id_for_record(rec) and its siblings take
this typed scope list (not qualified_name) as their real input, with
qualified_name/name kept exactly as they are today for every
consumer that still wants the flat display spelling.
This still leaves one real question this phase cannot paper over with a
third redesign: _find_opaque_types/type_reachability.py's other
consumers run after parsing, against an already-built AbiSnapshot —
they have no access to the parser's local typed-scope list by the time
they run, only to RecordType's own fields. "Call the resolver
on demand" only works where the typed scope data the resolver needs is
still in scope, which is true during parsing and false for every
post-parse consumer this phase's own acceptance criteria require
migrating (diff_filtering.py's ambiguity-tracking helpers, explicitly
named for deletion below). Two earlier framings of this section each
answered a different half of the real question and missed the other:
round 15's "no new carrier field, call the resolver on demand" is correct
for where the computation happens (parse time, not a cached field) but
silently assumed every consumer could reach that computation, which this
round's finding shows is false for any consumer running after parsing.
Resolving it for real needs one of two shapes, and this plan does not
pick one under continued review pressure a third time: (a) EntityId
actually is computed once, at parse time, and carried forward on the
model objects after all — which means Phase 2 does introduce a field
(RecordType.entity_id/equivalent per kind), contradicting this
section's earlier "no carrier" framing, with its own schema bump and
round-trip test; or (b) every post-parse consumer this phase lists for
migration is deferred to land with Phase 6 instead of before it, since
Phase 6's raw-fact capture is the one place in this plan's own sequencing
that already has the typed scope data (SemanticIR's CanonicalEntity
is built from it directly), making Phase 2 define the types and the
algorithm while Phase 6 is where real declarations actually get resolved
identities. This is named explicitly as Phase 2's own open design
question for its implementation PR to resolve, the same way this plan
has already done for the SourceGraphSummary-relocation and
compare()'s-own-public-surface-parameter questions elsewhere, rather than
asserting a fourth, unverified answer here.
This choice is not contained to Phase 2 — it determines whether Phase
3, as sequenced below (after Phase 2, before Phase 6), is buildable at
all. Phase 3's public-surface graph keys its declaration/type nodes
by EntityId (see that phase's own injective-key fix, above), which
needs a real, resolved EntityId for every declaration/type node the
graph builder visits. Under option (a), that identity is already sitting
on the model object by the time Phase 3 runs, same as every other field —
no conflict. Under option (b), no post-parse consumer has one yet
(resolution is deferred to Phase 6's SemanticIR assembly, which is
exactly why option (b) exists), and Phase 3's graph builder is a post-parse
consumer by construction — it walks an already-built AbiSnapshot, the
same position type_reachability.py's other consumers are in per the
finding above. Under option (b), Phase 3 therefore cannot be built as
sequenced: either its identity-dependent parts move to land with or
after Phase 6 (the same deferral option (b) already applies to every
other post-parse consumer, generalized to this one), or the Phase 2
implementation PR resolves the open question as option (a) before Phase 3
starts. Not decided here, for the same reason the question itself is
left open above — but the dependency is stated explicitly so Phase 3's
own implementation PR does not discover it mid-flight.
This phase explicitly targets, and closes, the specific collision bugs
AGENTS.md's "Known gaps" records as already-found-and-patched-locally:
opaque-type suppression keyed by bare RecordType.name
(diff_filtering._find_opaque_types), the dumper_clang.py tag-vs-
ordinary-namespace typedef collision, and type_reachability.py's
multi-round namespace-suffix/bare-alias collision history (eleven-plus
numbered findings in that one entry). Each of those local patches is
replaced by one ScopePath-based identity computation instead of being
kept as a parallel, narrower fix.
This is not the first EntityId/OccurrenceId in the repository, and a
first draft of this phase treated it as one. ADR-062 Phase 0 already
defined storage/entity_ids.py's own EntityId (kind: EntityKind,
qualified_name: str, discriminator: str) and OccurrenceId, complete
with a packed key property and to_dict()/from_dict() — inert today
(ADR-062 Phase 1's writer/reader doesn't exist yet, per that ADR's own
status), but a real, already-reviewed module, not a stub. Landing model/
identity.py's (ScopePath, kind, leaf_name, extra) shape as a second,
independent type would leave exactly two canonical identities once Phase 8
wires storage's writer/reader — the Governing Invariant's one forbidden
outcome. EntityKind/ObservationKind (genuinely domain vocabulary, not
a storage wire concern) relocate from storage/entity_ids.py into model/
identity.py alongside the new primitive, and model.identity.EntityId's
kind field is typed as the relocated EntityKind enum rather than a bare
string literal, closing that mismatch too.
Flattening ScopePath/extra into the existing bare
qualified_name/discriminator strings is not a lossless bridge, and a
first draft of this phase claimed it was without checking. ScopePath
is a typed tuple of segment kinds (Namespace, Record,
InlineNamespace, Anonymous, LocalToFunction) — rendering it to one
string, the way a display spelling does, discards which kind each segment
was. Two domain EntityIds whose ScopePaths differ only in segment kind
(a record nested in a record vs. the same names nested in a namespace; an
inline-namespace segment vs. an ordinary one) can render to the identical
qualified_name string, so from_dict() reconstructing a domain
EntityId from that string cannot recover which one it was — a save/load
round trip can silently collapse two distinct domain identities into one,
or change which declarations a reloaded EntityId is considered equal to.
A one-way render for display purposes is fine; claiming it as the wire
DTO's reversible encoding is not. The actual fix: storage/entity_ids.py's
EntityId/OccurrenceId DTOs gain a new wire-schema version (D8's "a
migration adapter per DTO version," applied here for the first time) whose
to_dict() encodes the ScopePath as an explicit list of typed segment
records — {"kind": "namespace" | "record" | "inline_namespace" |
"anonymous" | "local_to_function", "name": str, ...}, one entry per
segment, preserving exactly the structure ScopePath itself carries —
plus leaf_name and extra each kept as their own typed fields rather
than folded into one discriminator string. This is what makes
to_dto()/from_dto() an actual round trip rather than a one-way
projection: from_dto() reconstructs the identical ScopePath/leaf_name/
extra tuple to_dto() started from, with no string to parse back apart.
The old version-1 shape (kind/qualified_name/discriminator) stays
readable — a migration adapter maps a v1 document's bare strings into the
closest v2 domain EntityId it can (a single, untyped Namespace segment
per ::-separated component, since v1 never recorded which kind a segment
was) — but a v1-loaded EntityId is documented as potentially not equal
to the v2 EntityId the same declaration would produce today; this is an
accepted, one-time migration-boundary gap, not a property the wire format
promises going forward. This is the identical domain/DTO split D8 already
establishes for Phase 8's storage writer, applied one phase earlier because
the domain type it wraps already existed before this phase, not invented
by it. storage/identity.py's own re-export of both names is unaffected —
it already imports them from storage/entity_ids.py, and continues to.
Files. abicheck/model/identity.py (new, leaf — no dependency on
checker_types/diff_*, per ADR-063 D10; also receives the relocated
EntityKind/ObservationKind enums from storage/entity_ids.py, per the
note above). The direction of reuse with
finding_identity.resolve_function_identity matters and a first draft of
this phase had it backwards: finding_identity.py is comparison logic
that itself imports model entities and checker_types, so model/
identity.py calling into it would make the leaf module depend upward
on compare/-level code — reversing ADR-061's required compare -> model
direction and either failing the architecture gate outright or creating a
cycle the moment comparison code also starts consuming EntityId. The
corrected direction: the canonical signature-resolution algorithm itself
(mangled-name-primary, the normalized parameter-type/cv-qualifier fallback
tuple, the extern "C" exclusion) moves into model/identity.py as part
of EntityId's own function-identity constructor, and
finding_identity.resolve_function_identity becomes a thin wrapper
delegating to it — compare -> model is the allowed edge, so compare/
(where finding_identity.py lives) depends on the leaf, never the
reverse. This is the same generalization direction every other phase in
this plan already takes (the algorithm moves to the primitive; the
original call site becomes the wrapper), corrected here to actually match
it rather than stating it backwards. diff_filtering.py's
_find_opaque_types/_find_by_value_types/_root_type_name (consume
EntityId instead of bare t.name); dumper_clang.py/
dumper_castxml.py's parse_types() (produce ScopePath-derived
identity, replacing the ad hoc "::".join([*entry.scope, name]));
type_reachability.py (its multiple ambiguity-tracking helpers —
_spelling_index, _typedef_spelling_targets, _namespace_suffix_
spellings — collapse into one ScopePath-based resolver, deleting the
bespoke string-suffix machinery once the new resolver's test coverage
matches or exceeds the existing eleven-plus regression cases);
storage/entity_ids.py (trimmed to the wire-DTO EntityId/OccurrenceId
pair plus the new to_dto()/from_dto() bridge to model.identity.
EntityId, per the relocation note above — EntityKind/ObservationKind
move out, the packed-key DTO shape stays).
Tests. Every existing regression test named in the "Known gaps"
collision-history entries above is kept (they pin real, previously-found
counterexamples) and re-pointed at the new EntityId resolver rather than
deleted — a primitive-level property suite
(tests/test_entity_identity.py, per AGENTS.md's "Primitive-level
property tests" convention) states the contract directly: two distinct
declarations in different namespaces never collide regardless of bare-name
overlap; a using-declaration's EntityId always resolves to its target's,
never a sibling; namespace-suffix stripping is symmetric and never merges
two records whose full ScopePaths differ; and two distinct overloads
sharing one ScopePath (f(int) vs. f(double), and separately void
f() vs. void f() const) always produce distinct EntityIds, pinned
directly against the exact counterexample a reviewer raised for this
design, with an extern "C" sibling case confirming the deliberate
opposite rule (a changed parameter list there stays the same identity).
The carrier-vs-no-carrier tests below are conditional on which option
the open design question (above) actually resolves to — a first draft of
this phase stated both as unconditional requirements, which is
self-contradictory: the no-new-field test forbids option (a) outright,
and the pure-function-of-existing-fields test cannot pass for option (b)
at all, since a post-parse RecordType/Function's own flattened fields
are exactly what that open question found insufficient to reconstruct a
typed ScopePath from. Whichever option this phase's implementation PR
selects gets exactly one of these two test shapes, not both: if option
(a) (a real entity_id-shaped field, populated at parse time) is chosen,
the test suite asserts that field is populated for every declaration kind
immediately after parsing, round-trips through serialization.py
unchanged, and a static check confirms the resolver function itself is
never called on an already-parsed RecordType/Function outside the
parser (since by that option's own design, the field is read thereafter,
never recomputed). If option (b) (resolution deferred to Phase 6's
raw-fact capture) is chosen, this phase's own test suite is narrower: it
pins the resolver's contract against the raw facts Phase 6's normalizer
will eventually supply (not against already-parsed model objects), and
the "calling the resolver twice on separately-constructed field-identical
objects produces equal EntityIds" property and the "no new entity_id-
shaped field" static check both apply only to this narrower scope —
Phase 6's own implementation is where the real, structurally-sufficient
input actually gets threaded through, tested there, not asserted
prematurely here against a model shape this phase already found
insufficient.
A separate test on the relocation covers storage/entity_ids.py's new v2
wire schema: a primitive-level round-trip property test constructing
domain EntityIds across every ScopePath segment kind (including two
deliberately chosen so their rendered qualified_name strings would
collide under the old v1 flattening, pinning the exact counterexample this
finding raised) and asserting from_dto(to_dto(entity)) == entity — not
merely that some string comes back, but that the reconstructed domain
object is equal to the original, segment kinds included. A second test
covers the v1 migration adapter: every existing v1 fixture document still
loads without error, and the result is documented (in the test, not only
in prose) as a best-effort reconstruction rather than asserted equal to a
fresh v2 encoding of the same logical entity.
Acceptance criteria. diff_filtering.py/type_reachability.py's
string-based ambiguity-tracking helpers are deleted, not kept alongside
the new resolver — conditional on option (a) being the one this phase's
open design question resolves to, not unconditional. A first draft of
this criterion deleted them regardless of which option the implementation
PR picks, which review correctly caught as unsatisfiable under option
(b): that option's own premise (stated above) is that no post-parse
consumer — diff_filtering.py/type_reachability.py included, named there
by name — has a resolved EntityId until Phase 6's SemanticIR assembly
runs. Deleting their only working implementation in this phase under
option (b) would leave them with neither the old mechanism nor a usable
replacement for the several phases in between — worse than the
double-reporting/collision bugs this phase exists to close. So: under
option (a), this criterion holds exactly as stated, in this phase. Under
option (b), the deletion moves to land together with Phase 6's own
migration of these same consumers (already named as deferred to that
phase above), and this phase's acceptance bar for them is narrowed to "the
new model.identity resolver exists and is correct," not "every
consumer has migrated onto it yet." Exactly one EntityKind/ObservationKind definition
exists in the repository after this phase, in model/identity.py —
storage/entity_ids.py imports rather than redefines them. FP-rate gate
shows no regression (a net-new suppressed finding from the identity
change is a Phase 2 bug, not acceptable drift).
Landed (first slice, 2026-08-29), not the whole phase — read this before
assuming the Design section above is fully implemented.
abicheck/model/identity.py is real: the five ScopeSegment types
(Namespace, Record, InlineNamespace, Anonymous, LocalToFunction)
with exactly the identity-vs-payload field split the Design section states
(Record.access excluded from equality/hash via field(compare=False),
every other segment's fields all participate), the corrected EntityId
shape (scope, kind, leaf_name, extra), and entity_id_for_type/
_enum/_typedef/_constant/_variable/_function constructors
implementing the function/variable discriminator rules (mangled name first
when the caller has already established it is genuine, normalized
param-type/cv-qualifier fallback otherwise — both branches tagged
("mangled"/"sig") so they occupy disjoint regions of extra's value
space and can never collide with each other by coincidence).
EntityKind/ObservationKind are relocated from storage/entity_ids.py
into this module and re-exported from there under their original names, so
storage.entity_ids.EntityKind is model.identity.EntityKind — that one
acceptance criterion is closed now, ahead of the rest of the phase.
Primitive-level property tests in tests/test_model_identity.py (not
tests/test_entity_identity.py as this phase's own "Tests" section named —
that file was already taken, by the unrelated ADR-048/G31
buildsource.entity_identity L5 primitive; corrected here rather than
silently reusing the wrong name) pin: Record.access never affects
equality/hash; Anonymous/InlineNamespace/LocalToFunction are identity
on every field; a record nested in a record never collides with the same
bare names nested in a namespace (the exact collision a qualified_name-
only identity cannot distinguish, since both render to "A::B"); sibling
declarations of the same kind in one scope never collide (enums, typedefs,
constants, and the two function-overload shapes named in the Tests
section — f(int) vs. f(double), void f() vs. void f() const);
extern "C"-shaped input (a genuine mangled name present) ignores param
types, matching finding_identity.resolve_function_identity's own
documented rule; and the relocation itself, including a hashability check
per segment and a real AST-based (not substring) leaf-module-import check
per ADR-063 D10. Full fast unit suite green; storage's own existing test
suite (1421 tests) re-run clean against the relocated enums; mypy
abicheck/ and ruff both clean.
What this slice deliberately does not attempt, and why — both of Phase 2's own named open design questions are still open, on purpose:
- No
ScopePathis derived from a real parser yet. Everyentity_id_for_*constructor takes an already-builtScopePathas input; none of them reach intodumper_clang.py's/dumper_castxml.py'sentry.scope(still a barelist[str], exactly as insufficient as the Design section found it). Widening that parser state is real, separately-reviewable work of its own — this slice does not touch either dumper module. - The carrier-field question (option (a) vs. (b)) is still unresolved,
because nothing yet needs it resolved: with no real
ScopePathproducer wired up, there is nothing for a post-parse consumer (diff_filtering.py's_find_opaque_typesand siblings,type_reachability.py) to migrate onto yet, so this slice does not touch either of them, and their existing string-based ambiguity-tracking machinery is untouched and still the one working implementation. Per this phase's own acceptance criteria, deleting them now — before either option is chosen and actually wired — would be strictly worse, not a shortcut. finding_identity.pydoes not delegate to this module yet. The "direction of reuse" note above is correct about where the algorithm should end up, butfinding_identity.is_real_mangled_name/normalize_mangled_name(the ~450-line, independently multi-round- reviewed Itanium-mangling-validation machinery that decides whether a caller-suppliedmangled_nameis genuine) has not moved. This slice'sentity_id_for_function/entity_id_for_variableinstead take that determination as an already-resolvedmangled_name: str | Noneargument from the caller, documented in the module's own docstring as a deliberate scope boundary rather than an oversight. Migrating that validation logic, and turningresolve_function_identity/resolve_variable_identityinto thin wrappers, is real follow-on work this slice intentionally leaves for its own dedicated, independently reviewable pass — not attempted here to keep this slice small enough to review on its own, matching how every other phase's own first slice in this plan has been landed one piece at a time.- No storage v2 wire-schema bridge (
to_dto/from_dto) exists yet.storage.entity_ids.EntityId/OccurrenceIdare completely unchanged beyond the two enum imports —model.identity.EntityIdandstorage.entity_ids.EntityIdare two independent types today, which the Design section's own note names as the accepted, temporary state until a later slice does theScopePath-preserving v2 encoding it specifies.
Next slice, in order: (a) widen entry.scope in both dumper modules to a
typed segment list (this is the fork point for the carrier-field question —
see the Design section's own framing of why it can't be answered
independently of that work), then (b) migrate diff_filtering.py/
type_reachability.py once a real producer exists, then (c) the
finding_identity.py algorithm migration and the storage v2 wire bridge,
each as their own reviewable slice.
Correction (2026-08-29, same day, Codex review on PR #941): two real
discriminator gaps in entity_id_for_function fixed, both traced to the
same root cause -- the constructor accepted a strict subset of the
dimensions finding_identity.resolve_function_identity already treats as
load-bearing, so a caller reproducing that resolver's own contract through
this constructor could not. (1) An extern "C" producer follows
mangled_name's own documented contract and passes None for it (its raw
export spelling is not a genuine mangling) -- with no separate linkage
signal, that fell through to the signature-based fallback branch, so a
changed parameter list on a genuinely-non-overloadable C-linkage function
produced two ids instead of one modification, defeating the name-based
extern-C fallback the rest of the codebase relies on. Fixed by adding an
explicit is_extern_c: bool = False parameter that takes a third,
signature-free ("extern_c",) branch, mirroring resolve_function_
identity's own func.is_extern_c gate. (2) The signature fallback carried
only param_types/cv_qualifiers, missing the two dimensions
resolve_function_identity already folds in (ref_qualifier,
is_variadic) -- C::f() & vs. C::f() &&, and void f(int) vs.
void f(int, ...), both collided. Fixed by adding ref_qualifier: str = ""
and is_variadic: bool | None = None parameters, folded into the tagged
("sig", ...) tuple; is_variadic stays a tri-state (not defaulting to
False) for the same reason resolve_function_identity keeps it one --
"doesn't know" must not silently collapse onto "confirmed non-variadic".
Both fixes are additive keyword-only parameters with backward-compatible
defaults; no existing call site changes behavior (there are none outside
this module's own tests yet). Six new tests in tests/test_model_identity.py
pin each new dimension, the three-way tag disjointness, the tri-state
is_variadic, and that a genuine mangled_name still wins outright and
ignores every other dimension.
Correction (2026-08-29, same day, CodeRabbit review on PR #941): the
Design section above states LocalToFunction(owner) as a single-field
segment, but that is an under-specification the same review caught for
real -- fixed by adding block_ordinal, a required second field.
owner alone disambiguates two same-named locals declared in different
functions, but not two same-named locals in sibling compound blocks of
the same function (void f() { { struct A {}; } { struct A {}; } } --
two distinct declarations, identical owner, identical leaf name, so both
would collapse onto one EntityId under the Design section's original
single-field shape). block_ordinal closes this exactly the way
Anonymous.ordinal already closes the structurally identical
sibling-anonymous-scope case: a deterministic per-function sequence
number assigned at parse time, stable within one parse only -- the
identical accepted, documented limitation Anonymous already states (an
inserted or removed earlier local block shifts every later sibling's
ordinal and therefore its whole EntityId, even though nothing about
those later declarations changed). No default value, matching
Anonymous.ordinal, since nothing constructs a LocalToFunction outside
this module's own tests yet -- a caller must supply a real value rather
than silently under-specifying identity via an unwired default. The code
docstring's original "Both fields are identity" language was already
correct in intent (mirroring Anonymous's own phrasing) but described a
struct that, before this correction, had only one field -- a real
inconsistency between the stated contract and the actual shape, not just
prose.
Correction (2026-08-29, same day, second Codex review round on PR #941,
commit de0f23e): three more real gaps found and fixed, on top of the two
already closed above. (1) block_ordinal's own fix (just above) was
itself incomplete: LocalToFunction.owner stayed a bare str, which
still collides two overloads that each declare a same-named local in
their corresponding block (f(int) { struct A {}; } vs. f(double) {
struct A {}; } -- identical owner="f" string, identical leaf name).
Fixed by changing owner's type to EntityId -- the owning function's
own, already-overload-disambiguated identity -- rather than widening it
to yet another bespoke discriminator field; EntityId is frozen/hashable,
so the resulting recursive structure (an EntityId whose scope may
contain a LocalToFunction whose own owner is another EntityId) is
free, not a special case. (2) The is_extern_c branch added earlier
folded the caller-supplied scope into the returned EntityId unchanged,
which reintroduces exactly the evidence-tier fragmentation problem
resolve_symbol_identity deliberately avoids for this case: a header/
DWARF-derived observation of a namespaced extern "C" function may supply
a real ScopePath, while an export-table-only snapshot of the identical
binary symbol knows only the bare exported name (extern "C" linkage means
the symbol is that bare name at the ABI level -- no namespace is even
recoverable from an export table alone). Fixed by forcing scope=() for
the is_extern_c branch specifically, regardless of what scope argument
the caller passed; the mangled/sig branches are unaffected -- a real
mangled name already fully and deterministically encodes scope (no
evidence-tier divergence possible), and a DWARF-only, mangling-free
function has no comparable across-tier availability problem to guard
against. (3) The sig fallback's param_types were joined into extra
as raw, uncanonicalized strings, so CastXML's "char const*" and Clang's
"char const *" -- an otherwise-identical parameter type -- would
fragment identity across the two header-AST backends. Fixed by running
each param_types entry through name_classification.
canonicalize_type_name before joining, mirroring
resolve_function_identity's own canonicalization of func.params for
the identical reason; name_classification is itself a dependency-free
leaf module already imported by other model/ modules
(model/graph_identity.py, model/stdlib_surface.py), so this does not
violate this module's own leaf-module contract (ADR-063 D10) -- it is not
checker_types/diff_*/checker/compare/finding_identity. Nine new
tests in tests/test_model_identity.py pin all three fixes, including the
overload-disambiguated-owner case, the export-only-vs-namespaced
scope-independence case (and that it does not erase leaf_name's own
discriminating power), that the sig branch keeps scope as given, and
the cross-backend canonicalization. (This section originally also claimed
the mangled branch keeps scope as given, alongside sig -- see the next
correction below for why that claim was itself wrong and has been fixed
here rather than left standing.)
Correction (2026-08-29, same day, third Codex review round on PR #941,
commit e5cbdf2): the previous correction's own claim that the mangled
branch "keeps scope as given... redundant but harmless" was wrong, and
the identical evidence-tier-fragmentation bug the is_extern_c branch
was fixed for above turned out to reach two more places. (1)
entity_id_for_function's mangled branch folded the caller-supplied
scope into the returned EntityId unchanged. That is not harmless: a
header/DWARF-derived observation of a mangled function may supply a real
ScopePath, while an export-table-only snapshot of the identical binary
symbol knows only the bare mangled name -- exactly the same
evidence-tier-availability mismatch the is_extern_c fix already closed,
just for the mangled case instead of the extern-"C" case.
resolve_symbol_identity's own real primary id for a genuine mangling is
f"mangled:{real_mangled}" alone, with no scope folded in at all --
confirming the mangled branch should have matched the is_extern_c
branch's scope=() treatment from the start. Fixed by forcing scope=()
for the mangled branch too; only the sig fallback -- which has no
authoritative, scope-independent name to fall back on -- keeps scope as
given. (2) entity_id_for_variable had no is_extern_c parameter at all,
so a namespaced extern "C" variable (caller passes mangled_name=None
per that parameter's own contract) had no way to reach a scope-independent
identity the way the equivalent function case now does, and its own
mangled branch had the identical scope-folding bug as (1). Fixed by
adding the same is_extern_c parameter entity_id_for_function has, and
forcing scope=() for both its mangled and its is_extern_c branches.
Six new tests across both constructors pin all of this: mangled-branch
scope-independence for both functions and variables, the sig branch
still keeping scope as given, the new variable is_extern_c path and its
own tag-disjointness from the pre-existing mangling-free degenerate case,
and mangled-name-wins-over-is_extern_c precedence for variables
(mirroring the function constructor's own precedence, already tested).
Correction (2026-08-29, same day, fourth Codex review round on PR #941,
commit f22ecf7): a fourth, confirmed (not merely hypothetical) gap in
the mangled branch, found by reading the real producer code. The
previous correction made the mangled branch's EntityId.scope always
(), but left leaf_name folded in unchanged. Codex's finding named a
concrete, real code path proving this still fragments identity:
dumper_elf_fallback.py's ELF-only (export-table-only) path constructs
Function(name=sym, mangled=sym, ...) and Variable(name=sym,
mangled=sym, ...) -- the raw exported symbol reused for both fields,
confirmed by reading that module directly rather than assumed. A header/
DWARF observation of the identical symbol supplies the real demangled
short name for name (e.g. "f"), while the export-only observation's
name is the raw mangled spelling itself (e.g. "_Z1fv"). Since
leaf_name was still part of the mangled branch's EntityId, these two
observations of the same symbol -- agreeing on the mangled evidence --
would not merge. Fixed by ignoring the caller-supplied leaf_name
entirely in the mangled branch (both constructors), using "" instead:
the mangled spelling in extra already disambiguates every declaration
losslessly, so nothing is lost. Four new tests (two per constructor) pin
that a demangled-short-name observation and a raw-symbol-reused-as-name
observation of the identical mangled symbol now produce one EntityId.
Correction (2026-08-29, same day, fifth Codex review round on PR #941,
commit 4ad03da): two real gaps in the sig fallback's own signature
normalization, one requiring a genuinely new, carefully-scoped
primitive rather than reuse of an existing one. (1) param_types were
canonicalized only for cross-producer spelling (via
canonicalize_type_name), not for the C++-standard-mandated fact that a
top-level BY-VALUE cv-qualifier is dropped from a function's own type for
linkage/mangling purposes -- void f(int) and void f(const int) name
the identical function, but "int" and "const int" canonicalize to
different strings and collided as two overloads. The naive fix --
reusing this codebase's existing _strip_cv_qualifiers/
func_signature_cv_only_differ (the primitive an initial attempt reached
for, since Codex's own finding named it) -- was investigated and found to
be wrong, not merely reused: that helper is deliberately permissive at
the true top level, stripping a pointee cv-qualifier too ("const char
*" -> "char *"), because its actual job is "is this an already-
positionally-matched declaration's param change worth reporting as ABI-
breaking" (diff_symbols._params_differ's question) -- a different
question from this primitive's, "are these the same overload for identity
purposes." A pointee cv-qualifier on a pointer/reference parameter is a
genuine, standard-mandated overload discriminator (void f(char*) and
void f(const char*) are two simultaneously-declarable, independently-
mangled overloads), so reusing the permissive helper verbatim would have
silently merged two distinct functions into one identity -- reintroducing
exactly the sibling-overload-collision class this whole primitive exists
to prevent, while fixing the narrower problem Codex actually named. Fixed
correctly instead: a new, narrower public primitive,
name_classification.canonicalize_function_signature_type, strips a cv
token only when the canonicalized type carries no top-level pointer/
reference sigil at all (checked via the same _has_top_level_ptr_or_ref
helper cv_qualifiers_only_differ already uses) -- the one case where
every cv token found is unambiguously by-value, never pointee, cv. (2)
cv_qualifiers: tuple[str, ...] was itself the wrong representation:
unlike resolve_function_identity's own func.is_const/
func.is_volatile booleans, a caller-supplied token tuple invites order-
dependence for one member-cv qualification spelled two ways (("const",
"volatile") vs. ("volatile", "const") -- caught named literally as
f() const volatile vs. f() volatile const). Fixed by replacing the
parameter with is_const: bool = False, is_volatile: bool = False,
matching resolve_function_identity's own representation exactly and
eliminating the ordering question by construction rather than by
normalizing a tuple after the fact. Six new tests pin both fixes: by-value
top-level cv non-distinction, pointee cv's continued distinguishing power,
independent const/volatile discrimination, and that the two booleans'
call-site order cannot affect the resulting id.
Correction (2026-08-29, same day): the fifth correction's own new
primitive landed in the wrong module and had to be relocated before
push. name_classification.canonicalize_function_signature_type was
first added directly to abicheck/name_classification.py -- but that
file is a frozen, no-growth legacy module under ADR-061's debt ledger
(architecture/debt.yaml, baseline 1258 lines), and CI's
debt-no-growth architecture check (scripts/check_architecture.py)
correctly failed the push for exceeding it. Relocated the whole primitive
into abicheck/model/identity.py itself as a module-private
_canonicalize_function_signature_param_type, along with a
self-contained, deliberately-duplicated (not imported) reimplementation
of the top-level-pointer/reference-sigil detection algorithm
name_classification._has_top_level_ptr_or_ref already applies for a
different purpose -- duplication here is the correct outcome of the
no-growth constraint, not an oversight: the alternative (importing a
private helper from the frozen module) would be worse, and the algorithm
is small, stable, and independently tested in its new location. This is
exactly the kind of tension a genuinely completed ADR-061 migration of
name_classification.py would resolve (see that file's own debt-ledger
entry, "target: see ADR-061 ownership map") -- not attempted here, since
migrating the whole file is far outside this slice's scope. git diff
--stat/wc -l confirm name_classification.py is back to exactly 1258
lines (its original baseline) and scripts/check_architecture.py
reports 0 errors.
Correction (2026-08-29, same day, sixth Codex review round on PR #941,
commit 1b3575d): two more real gaps found, both requiring a genuine
extension rather than reuse. (1) The by-value-cv fix above left one
case unhandled: a function PARAMETER's array type always decays to a
pointer (int [] -> int *), so a cv-qualifier on the array's
element type is pointee-level, exactly like an explicit pointer -- void
f(int[]) and void f(const int[]) are two distinct, independently-
mangled overloads, not one. Neither spelling contains a */&
sigil, so _has_top_level_ptr_or_ref (module-private, in
model/identity.py) wrongly treated both as by-value and stripped the
const. Fixed by also treating a top-level [ as pointer-shaped for
this determination -- the fix landed in this correction's own new
model/identity.py primitive precisely because it was already the
no-growth-safe location the fifth correction relocated to; there was no
second relocation needed. (2) ScopePath names only the containing
scope, never the leaf declaration (this module's own stated design), so
two anonymous sibling records/enums both passing leaf_name="" collided
onto one identical EntityId regardless of which is meant --
Anonymous.ordinal (a ScopePath segment) disambiguates a descendant's
containing scope, not the anonymous declaration's own identity, so it
does not help here. Fixed by adding an opt-in anonymous_ordinal: int |
None = None keyword to entity_id_for_type/entity_id_for_enum (the
two kinds C++ actually allows to be anonymous), folded into extra as
("anonymous", str(ordinal)) only when leaf_name is empty -- mirroring
Anonymous.ordinal's own deterministic-per-parent-sequence-number
semantics and identical within-one-parse-only accepted limitation.
Omitting it (no wired producer yet, same scope boundary this module's own
docstring already states for LocalToFunction/Anonymous before their
producers existed) keeps the pre-existing degenerate-collision behavior,
not a silent, unrequested change. Six new tests pin both fixes.
Correction (2026-08-29, same day, seventh Codex review round on PR #941,
commit b767576): two more real gaps in the by-value/array-decay logic,
plus a new hard architectural gate the fix itself then had to satisfy.
(1) The pointer branch skipped all cv processing as soon as it saw any
top-level */&/[, so a cv-qualifier trailing the parameter's own
outermost pointer sigil (int * const -- the pointer VALUE itself is
const, not what it points to) survived unstripped, even though the
standard drops it exactly like any other top-level parameter qualifier:
void f(int *) and void f(int * const) name the same function. Fixed
by splitting the string at the LAST top-level */&: everything up to
and including it is kept verbatim (an intermediate pointer level's own
qualifier, e.g. int * const *'s middle const, is genuinely
distinguishing and must survive), only the suffix is stripped. (2) The
earlier [-is-pointer-shaped fix stopped int []/const int [] from
colliding, but never performed the actual array-to-pointer decay, so
int []/int [3]/int [4]/int * -- all one identical adjusted
parameter type -- still canonicalized to four different strings. Fixed
with _decay_top_level_array, deliberately narrow: multi-dimensional
arrays (T[][N]) and parenthesized declarators (int (*)[3], "pointer to
array", where the trailing [3] is the POINTEE's bound, not the
parameter's own shape) are left entirely unchanged rather than mis-decayed
-- an accepted, documented limitation, not a silent gap. Implementing
these two together, correctly, in model/identity.py pushed that file to
834 lines, over the AI-readiness gate's 800-line production maximum with
no debt-ledger entry possible for a brand-new file (debt-exemption
explicitly forbids that) -- a second, different hard architectural
constraint from the fifth correction's debt-no-growth one, caught before
push this time by running scripts/check_ai_readiness.py locally first.
Resolved by splitting the whole cv/array-decay block into a new sibling
leaf module, abicheck/model/signature_normalization.py (identity.py
now 586 lines, the new module 299) -- not a design change, purely a file
split, with identity.py importing the one public
canonicalize_function_signature_param_type from it. The new module also
got its own dedicated primitive-level test file,
tests/test_signature_normalization.py (per AGENTS.md's own convention
for a new reusable primitive), which caught a genuine bug of its own
before it shipped: the array-decay path bypassed
canonicalize_type_name's east-const normalization (its regex never
matches a bracket-containing string), so "const int [3]" and "const int
*" -- the identical adjusted type -- canonicalized to two different
strings until the decay result was re-canonicalized a second time. Fixed
in the same commit; test_element_cv_becomes_pointee_cv pins it.
Correction (2026-08-29, same day, eighth Codex review round on PR #941,
commit 572de14): a parenthesized declarator's own outer pointer skipped
by-value cv stripping entirely. The sigil-scan treated every ( as a
real nesting level, so a cv-qualifier written inside a declarator's own
grouping parens -- void (* const)(int) (a callback parameter: "const
pointer to a function taking int"), or int (* const)[3] (a const
pointer to an array of 3 ints) -- was never found at depth 0 and so never
stripped, even though it qualifies the parameter's own outermost pointer
exactly like an unparenthesized int * const does. Fixed by making an
opening ( "transparent" (not a real nesting level) whenever the next
non-space character is */& -- the one shape a real function-parameter-
list paren can never have, since a parameter list's first token is always
a type, never a bare sigil. This also reaches the previously-unchanged
pointer-to-array case (int (*)[3]) the same way: its own outermost
pointer's cv-qualifier, once one exists, is by-value too, while the
trailing array bound inside the parens (the pointee's own shape) stays
untouched exactly as before. Also removed _has_top_level_ptr_or_ref,
a leftover private helper from an earlier revision of this module that had
become genuinely dead code -- the array/pointer-shape detection it
described was already reimplemented inline at each of its two call sites
during earlier correction rounds, and nothing in this module or its tests
called it. Four new tests in tests/test_signature_normalization.py
(TestParenthesizedDeclaratorOwnCvIsDropped) pin both the fix and that the
callback/pointee's own inner types stay untouched.
A second finding from the same round -- that a raw, platform-decorated PE
export name (_foo@4 for an extern "C" foo) reaching this module's
is_extern_c branch as leaf_name would fragment identity against an
undecorated header/DWARF observation of the same symbol -- was investigated
and NOT fixed here. entity_id_for_function's docstring already states
this branch is deliberately built to mirror resolve_symbol_identity's own
representation, and that resolver has the identical property: it also
keys a non-_Z/? mangled value on its literal raw spelling, undecorated
(finding_identity.py's own normalized_basis = mangled if (mangled and
not real_mangled) else qn never strips a stdcall/cdecl decoration either).
This module's stated contract is to match that resolver's own chosen
representation for evidence-tier scope/leaf-name handling, not to invent a
new normalization capability the mirrored function doesn't have -- and no
PE-decoration-stripping helper exists anywhere else in the codebase to
reuse. Introducing one is a real, valid gap worth closing, but it is a
separate design question (most naturally pe_metadata.py's own extraction
layer, where the platform's decoration convention is already known, not
this identity primitive) and out of scope for this "purely additive,
mirrors the existing resolver" first slice.
Correction (2026-08-29, same day, ninth Codex review round on PR #941,
commit 0b7a80a): two more real gaps in the same paren-transparency logic,
both fixed with a general recursive treatment rather than another one-off
patch. (1) A pointer-to-member-function declarator (void (C::*
const)(int)) has its own outermost sigil preceded by the member's
qualified-name prefix (C::) rather than a bare sigil -- the eighth
round's transparency check ("next non-space character after ( is
*/&") never recognized this shape, so its own trailing cv-qualifier
stayed unstripped. Fixed by generalizing the transparency test to a regex
(_DECLARATOR_GROUP_RE) matching one or more identifier:: segments
before the sigil, not only a bare one. (2) A declarator's own trailing
parameter list (the (int) in void (*)(int)) is itself exactly as much
"a function's parameters" as this function's own top-level parameter --
C++ drops a nested callback parameter's top-level by-value cv from the
callback's own function type too, so void (*)(int) and
void (*)(const int) name the identical adjusted callback type, not two.
The eighth round's fix left every trailing parameter-list paren entirely
opaque, so this nested by-value cv was never stripped. Fixed with a real,
general recursive treatment rather than a narrow special case: every
top-level (...) group found after the declarator's own sigil is now
comma-split (_split_top_level_commas, depth-aware) and each of its own
parameters is recursively run back through
canonicalize_function_signature_param_type itself
(_normalize_nested_param_lists/_normalize_param_list_contents) --
unlike the array-decay limitations, this is not "unbounded C declarator
grammar": it is one already-correct function applied to a strictly
shorter substring at each level, which is what terminates it, and it
reaches a callback-of-a-callback automatically since each recursive call
performs the identical treatment on its own nested parameter lists. An
empty parameter list, a bare void, and a variadic ... marker are left
untouched (none is itself a parameter type). Ten new tests in
tests/test_signature_normalization.py
(TestPointerToMemberOwnCvIsDropped,
TestNestedCallbackParametersAreNormalizedRecursively, plus two new
idempotence cases) pin both fixes, including that a nested parameter's
genuine pointee cv still distinguishes, a doubly-nested callback
normalizes at its innermost level too, and the variadic/void/empty
special cases pass through unchanged.
Correction (2026-08-29, same day, tenth Codex review round on PR #941,
commit fd715cc): two more real gaps in the same declarator handling, plus
a self-caught correctness regression from fixing the second one. (1) The
paren-transparency regex recognized a bare sigil or a member-pointer's
qualified-name prefix, but not an MSVC/PE calling-convention keyword
(__cdecl, __stdcall, __fastcall, __thiscall, __vectorcall)
preceding the sigil, so void (__cdecl * const)(int) -- confirmed a real,
reachable shape (this codebase's own PE/PDB-decorated-export handling in
diff_platform.py already deals with __cdecl-decorated exports) --
never found its sigil at depth 0 either. Fixed by widening
_DECLARATOR_GROUP_RE to accept an optional calling-convention keyword
before the existing qualified-name-prefix loop; the keyword itself is
matched, not erased, so it stays genuine, distinguishing content in the
prefix (void (__cdecl *)(int) and void (__stdcall *)(int) correctly
stay two different types). (2) A pointer-to-member-function's own
TRAILING cv/ref-qualifiers -- the ones that follow its parameter list,
e.g. const in void (C::*)(int) const -- were wrongly blanket-stripped
by the same by-value logic that (correctly) strips the pointer's own
qualifier before the parameter list. These are not the same thing: a
trailing qualifier qualifies the POINTED-TO member function itself, a
genuine, standard-mandated discriminator (void (C::*)(int) const is a
different, non-interchangeable type from void (C::*)(int)), so
collapsing them together was a real over-merge, the same class of mistake
this whole primitive exists to prevent, not merely an incompleteness.
Fixed by splitting the suffix at the declarator's own trailing parameter
list (_split_at_trailing_param_list): everything before it stays the
pointer's own by-value region (stripped, as before); everything after it
now goes through a new _canonicalize_member_qualifiers, which -- mirroring
entity_id_for_function's own is_const/is_volatile two-boolean
treatment for the identical ordering problem one level up -- only
reorders const/volatile into a fixed order and normalizes a ref-
qualifier (&/&&), rather than dropping any of it.
Self-caught regression, found by this round's own new ref-qualifier
tests before push, not by a reviewer: implementing fix (2) initially
broke on void (C::*)(int) &&, canonicalizing it as void (C:: * )(int)
&& for & but corrupting the && case into void (C:: * )(int) & & --
worse, canon(...) != canon(...) on the SAME input, an outright identity
non-determinism bug. Root cause: canonicalize_type_name spells && as
"& &" (its own established sigil-spacing convention, the same source as
"int * const *" -> "int *const *" elsewhere in this module), and the
main sigil-scanning loop records the LAST */& found anywhere at depth
0 as last_top_level_sigil -- so the second & of the trailing "& &"
ref-qualifier, itself sitting at depth 0 well after the declarator's own
parameter list had already closed, wrongly overrode the member-pointer's
own already-found *, corrupting the entire prefix/suffix split. Fixed
at the root: the main loop now tracks whether it has already seen a
top-level opaque paren (the declarator's own trailing parameter list)
and stops updating last_top_level_sigil once it has -- nothing
*/&-shaped appearing after a declarator's parameter list closes can
ever be a NEW top-level sigil, only a ref-qualifier on what came before.
_canonicalize_member_qualifiers's own ref-qualifier detection was
likewise fixed to compare against a whitespace-collapsed form so "& &"
is still recognized as &&, not two separate & tokens. Eleven new tests
in tests/test_signature_normalization.py
(TestCallingConventionDeclaratorGroupIsRecognized,
TestPointerToMemberTrailingQualifiersPreserved, plus two new idempotence
cases) pin all of this, including the exact &&-vs-& & regression.
A third finding from the same round -- that a bare, unadjusted function
type used as a parameter (void(int), as opposed to the already-
adjusted void (*)(int)) is returned unchanged rather than decayed to a
pointer the way an array parameter is -- was investigated and NOT fixed.
Checked against this codebase's own real header-AST producer output
(tests/test_dumper_clang.py's and test_dumper_clang_vtable.py's fixed
qualType strings) and confirmed: a real castxml/clang dump always
already prints the ADJUSTED pointer form for a function-typed parameter
("void (*)(int)"), never the bare, unadjusted function type -- unlike
the array case, where the plan's own earlier correction explicitly notes
this primitive "makes no assumption about whether a given producer's own
parsed representation already reflects the decay" because an unadjusted
array spelling genuinely does reach this function from real callers.
There is no equivalent confirmed real code path supplying an unadjusted
bare function type here, so implementing a heuristic "is this really an
unadjusted function type, not some other paren-containing spelling" guess
would be solving a hypothetical shape, with the attendant risk of a false
positive on a legitimate class-type spelling that happens to end in a
parenthesized group. Recorded here as an accepted, documented limitation
rather than silently dropped.
Correction (2026-08-29, same day, eleventh Codex review round on PR #941,
commit ea44356): one more real gap in the declarator-transparency test,
plus a second, more serious self-caught over-merge in the tenth round's
own trailing-qualifier fix. (1) _DECLARATOR_GROUP_RE's qualified-name
loop matched only plain identifier:: segments, so a template-qualified
nested-name-specifier -- void (C<int>::* const)(int), a real, common
C++ shape -- was never recognized as a declarator group, and its own
by-value cv stayed unstripped. Fixed by replacing the single regex with a
manual scanner, _is_declarator_group: a template-argument list can nest
arbitrarily deep (Box<Pair<int, int>>::), which re's non-recursive
matching cannot balance, so this needed real character-by-character
</> depth tracking, not a regex extension. (2) The tenth round's
_canonicalize_member_qualifiers fixed the over-merge on trailing
const/&/&&, but did so by RECONSTRUCTING the whole trailing region
from only those three tokens -- silently dropping anything else found
there. The practically important case: a noexcept-specifier, which
C++17 made part of the function type, so void (*)(int) noexcept and
void (*)(int) are two different, non-interchangeable types -- and this
primitive was merging them into one identity, the exact same over-merge
class the tenth round's own fix existed to close, just relocated rather
than eliminated. Fixed generally: instead of naming every possible
trailing specifier and reconstructing from a fixed list, the function now
only ever removes and reorders the const/volatile WORDS themselves
(via _CV_WORD_RE, the same primitive _strip_cv_tokens_outside_nesting
already uses) and passes everything else through verbatim, in its
original relative order -- dcl.fct's own grammar already fixes
cv-qualifier-seq first among a function's trailing specifiers, so a real
producer's placement of ref-qualifier/noexcept/anything else never
needs inferring, only cv needs reordering relative to itself. This also
means the function no longer special-cases the ref-qualifier's "&&" ->
"& &" spelling quirk (canonicalize_type_name already normalizes that
upstream, before this function ever sees the string, so passing the
leftover text through verbatim is already consistent). Eight new tests in
tests/test_signature_normalization.py
(TestTemplateQualifiedMemberPointerOwnCvIsDropped, four new cases in
TestPointerToMemberTrailingQualifiersPreserved for noexcept, plus two
new idempotence cases) pin both fixes, including a nested template
argument and noexcept combined with order-independent cv.
Correction (2026-08-29, same day, twelfth Codex review round on PR #941,
commit 509e3f3): the eleventh round's noexcept fix preserved the
specifier verbatim, which was necessary but not sufficient -- preserving
its exact SPELLING is not the same as canonicalizing it. Since C++17, a
function type's exception specification collapses to exactly two kinds
for TYPE purposes: "non-throwing" (bare noexcept or noexcept(true))
and "potentially-throwing" (no specifier at all, or noexcept(false)).
Those pairs are the SAME type and must canonicalize identically, but the
eleventh round's fix passed the raw text through unchanged, so
void (*)(int) noexcept and void (*)(int) noexcept(true) -- one
identical adjusted type -- canonicalized to two different strings (an
under-merge this time, the milder sibling of the over-merges every
earlier round fixed, but the same underlying principle: "verbatim
preservation" is not "canonicalization"). Fixed with a new
_canonicalize_noexcept: only the two literal, constant-expression
spellings (true/false) are recognized and normalized -- bare
noexcept/noexcept(true) collapse to the single canonical spelling
"noexcept", noexcept(false) is dropped entirely (equivalent, for type
purposes, to an already-absent specifier) -- while any other,
non-literal noexcept(expr) is left completely untouched, since
evaluating an arbitrary constant expression is out of scope, the same
"don't solve the fully general grammar" limit this module already draws
for multi-dimensional arrays and non-literal declarator shapes elsewhere.
Nine new tests in tests/test_signature_normalization.py
(TestNoexceptSpellingsCanonicalized, plus two new idempotence cases)
pin this, including that a non-literal expression stays genuinely
distinguishing rather than being silently (and wrongly) folded into
either canonical form.
**Correction (2026-08-29, same day, thirteenth Codex review round on PR
941, commit 45fd050): one more real under-merge, plus one more real¶
over-merge -- this time an actively corrupting one -- in the trailing-
qualifier and nested-parameter-list handling.** (1)
_normalize_param_list_contents returned an empty parameter list and a
bare void unchanged rather than unifying them, so void (*)() and
void (*)(void) -- the identical "no parameters" adjusted type --
canonicalized to two different strings. Fixed by collapsing both to the
same canonical empty form. (2) _canonicalize_member_qualifiers's
const/volatile extraction used a plain, depth-blind re.search/
re.sub over the WHOLE trailing region, which wrongly reached inside a
non-literal noexcept(expr)'s own argument too -- a const/volatile
token that is part of THAT expression's own text (e.g.
noexcept(Foo<const int>)) is not this declarator's own cv-qualifier at
all. This was worse than the earlier over-merges: it didn't just misjudge
which type two spellings belonged to, it actively MUTATED the nested
expression's own text (moving the found "const" out to the front,
leaving the argument reading Foo< int>), and could merge two genuinely
different overloads that happen to share a nested "const" by coincidence.
Fixed with a depth-aware scan, _extract_top_level_cv, mirroring
_strip_cv_tokens_outside_nesting's own outside-nesting discipline: only
a cv word sitting at nesting depth 0 -- outside any (...)/<...>/
[...] -- is ever this declarator's own trailing qualifier. Thirteen new
tests in tests/test_signature_normalization.py
(TestCvInsideNoexceptExpressionIsNotExtracted, a corrected
test_empty_and_void_param_lists_unify, plus two new idempotence cases)
pin both fixes, including that the nested const neither merges with nor
is corrupted by a real leading const.
Correction (2026-08-29, same day, fourteenth CodeRabbit review round on
PR #941, commit 45fd050): one more real, serious over-merge in the
tenth round's seen_top_level_opaque_paren flag itself, plus two
minor cleanups. (1) That flag was armed on ANY top-level opaque paren,
regardless of whether the declarator's own sigil had already been found.
This is wrong for a real, observed producer spelling this codebase
already handles elsewhere (tests/test_diff_templates.py's own
"(anonymous namespace)::T"): (anonymous namespace)::Foo const *
starts with an opaque paren (_is_declarator_group correctly rejects it
-- it's not a declarator group) that precedes the parameter's own actual
pointer sigil entirely, not a trailing parameter list. Arming the flag
there locked out that LATER real sigil, so the whole type fell through to
the by-value-strip branch and wrongly merged Foo const * with Foo *
-- pointer-to-const vs. pointer-to-non-const, two genuinely
non-interchangeable types. Fixed by only arming the flag once
last_top_level_sigil != -1 -- an opaque paren before the declarator's
own sigil is never its trailing parameter list, so it must never lock
sigil detection out. (2) A tautological test assertion
(canon(x) == canon(x), which cannot fail for a pure function and pinned
nothing) was replaced with a real idempotence check. (3) The twelfth
round's own changelog wording claimed a non-literal noexcept(expr)
"stays genuinely distinguishing", which overclaims: C++ evaluates a
constant expression's actual boolean VALUE, not its literal spelling, so
e.g. noexcept(sizeof(int)) is genuinely type-equivalent to
noexcept(true) while this primitive (which only recognizes the two
literal true/false spellings, per its own docstring's already-correct
hedge) treats them as different -- a real, documented over-splitting
limitation, corrected in the changelog fragment's wording rather than the
code (evaluating an arbitrary constant expression remains out of scope).
Two new tests in tests/test_signature_normalization.py pin the
anonymous-namespace fix, including idempotence.
**Correction (2026-08-29, same day, fifteenth Codex review round on PR
941): a bare data-member-pointer's pointee cv-qualifier was misplaced¶
across the ClassName:: infix, plus the fix's own line-count growth
forced a second module split.** canonicalize_type_name's own east-const
regex -- unmodifiable, since name_classification.py is a frozen,
no-growth legacy file -- does not know how to normalize a leading
cv-qualifier across a bare (non-parenthesized) data-member-pointer's own
ClassName:: infix, and MISPLACES it depending on which side it started
on: "int const C::*" stays "int const C:: *" (matching this
primitive's own canonical output), but "const int C::*" becomes
"int C:: const *" -- the cv-word shoved in between the qualifier and
the sigil, indistinguishable in that position from the pointer's own
by-value qualifier. Both spell the identical pointer-to-const-int-member
type and must canonicalize identically; a first, narrower fix attempt
(scanning only immediately before the sigil) caught the first
manifestation but not the second, discovered via direct testing. Fixed
with a more robust "reliable marker" strategy in the new
_find_member_pointer_qualifier: a member pointer's own qualifier is
always followed by a single space before whatever comes next
("C:: *"), unlike ordinary namespace qualification within a type's own
spelling, which never has a space after :: ("ns::Foo") -- so the LAST
":: "marks the qualifier's end reliably, and the caller re-collects
any cv word found on either side of the qualifier (base-side and
tail-side) before re-canonicalizing the combined base. A related,
broader limitation was found and deliberately left out of scope:canonicalize_type_namenever reordersvolatileat all (onlyconst), a pre-existing gap affecting every pointee-volatile position in
this module, not specific to member pointers -- fixing it is a
substantially larger undertaking than this round's actual Codex finding
(which namedconstspecifically), so it is documented directly in_find_member_pointer_qualifier`'s own docstring and the corresponding
doctest as an accepted, out-of-scope residual gap rather than silently
patched over.
Implementing this pushed signature_normalization.py to 908 lines, over
the AI-readiness gate's 800-line production maximum (which has no
debt-ledger workaround for a file this new) -- the identical situation
the seventh round's own identity.py -> signature_normalization.py
split addressed one level up. Resolved the same way: a second sibling
leaf module, abicheck/model/declarator_qualifiers.py, now holds every
piece of this module's own machinery that has no recursive dependency
back into canonicalize_function_signature_param_type itself --
_is_declarator_group (+ its _CALLING_CONVENTIONS/_IDENTIFIER_RE
supporting constants), _extract_top_level_cv, the new
_find_member_pointer_qualifier (+ _MEMBER_POINTER_TAIL_RE),
_split_at_trailing_param_list, _canonicalize_member_qualifiers, and
_canonicalize_noexcept (+ _NOEXCEPT_RE). What stays behind in
signature_normalization.py is exactly the machinery that DOES recurse
back into canonicalize_function_signature_param_type
(_normalize_param_list_contents/_normalize_nested_param_lists, for a
callback/member-function-pointer's own nested parameter list) plus the
top-level by-value cv/array-decay logic and the main function itself --
so the import is strictly one-way (signature_normalization ->
declarator_qualifiers, never the reverse), avoiding a cycle between the
two sibling leaf modules. _CV_WORD_RE (a trivial one-line regex
constant) is duplicated in both modules rather than shared, since sharing
it would need an import in the direction that would create that cycle.
Both files stay comfortably under the cap after the split (~600 and ~370
lines respectively). The new module gets its own dedicated
primitive-level test file, tests/test_declarator_qualifiers.py
(mirroring test_signature_normalization.py's own rationale and
including its own leaf-module-contract test), alongside new tests in
tests/test_signature_normalization.py pinning the member-pointer fix
itself (both cv-spelling orderings unifying, an ordinary namespace-
qualified pointer staying unaffected, the parenthesized member-function-
pointer case staying unaffected, and idempotence).
**Correction (2026-08-29, same day, sixteenth Codex review round on PR
941, commit 932814b): restrict/__restrict/__restrict__ was left¶
completely unstripped, fragmenting identity across a compatible
parameter-qualifier change.** int * and int *restrict canonicalized
to two different strings, even though restrict has NO effect on the
Itanium C++ ABI's mangling at all -- this repository already tracks the
fact separately (Param.is_restrict; dumper_castxml.py's own
extraction comments state the identical thing), so treating a restrict
addition/removal as an EntityId change turned that dedicated compatible
change into a spurious removal/addition pair instead. The finding as
reported scoped a fix to only the parameter's own outermost pointer,
mirroring how a genuine BY-VALUE cv-qualifier is scoped (per this
primitive's own established, deliberately narrow-scoped-to-review-
evidence discipline) -- but unlike cv, restrict's absence from mangling
does not depend on WHERE in the declarator it sits (the C/C++ grammar
only ever allows it directly after the * it qualifies, at any nesting
depth), so a fix scoped to "outermost only" would leave e.g.
int * restrict * and int * * still wrongly distinguishing. Fixed more
generally than the reported scope: a new _RESTRICT_WORD_RE strips every
restrict/__restrict/__restrict__ token from the canonical type
string unconditionally, before any of the position-sensitive cv/pointer
logic runs -- so a nested callback parameter's own restrict-qualified
pointer is covered too, through this function's existing recursion into
itself for a trailing parameter list's own parameters, with no special-
casing needed for that case. New tests:
TestRestrictQualifierIsAlwaysStripped in
tests/test_signature_normalization.py (all three spellings, a
non-outermost-pointer restrict, a callback parameter's own restrict, a
sanity check that a genuine pointee cv-qualifier still distinguishes, and
idempotence).
**Correction (2026-08-29, same day, seventeenth Codex review round on PR
941, commit 2264423): Clang's own trailing calling-convention attribute¶
spelling was not recognized, fragmenting identity across header-AST
backends.** Clang's ParmVarDecl.type.qualType renders a calling-
convention-decorated function-pointer declarator as e.g.
void (*)(int) __attribute__((cdecl)) -- the attribute trails the
parameter list -- rather than MSVC/castxml's leading-keyword spelling,
void (__cdecl *)(int), which _is_declarator_group/
_CALLING_CONVENTIONS already recognized. The two spellings of one
identical type canonicalized differently, so a template or DWARF-only
function observed through both backends would fragment. Fixed with a new
_CALLING_CONVENTION_ATTR_RE, matching the same five attribute-node
kinds dumper_clang_attributes.py's own _CLANG_ATTR_TOKENS mapping
already recognizes for a FunctionDecl's own top-level contract
attributes (this is the textual, parameter-type-spelling equivalent, for
the mangling-free signature-fallback identity this module computes): when
found in the trailing region after a declarator's own parameter list, the
attribute text is removed and, if the leading prefix doesn't already
carry a calling-convention keyword, the equivalent __cdecl-style
keyword is injected at the same position _is_declarator_group looks for
it -- so both spellings converge on one prefix before the ordinary
member-qualifier canonicalization runs on whatever remains of the
trailing text. Verified this is a real, internally-corroborated Clang
spelling (not merely the finding's own claim) via
dumper_clang_attributes.py's existing, independent handling of the
identical attribute-node kinds for the top-level FunctionDecl case. New
tests: TestClangTrailingCallingConventionAttributeUnifies in
tests/test_signature_normalization.py (cdecl and stdcall unifying with
their leading-keyword equivalents, the member-function-pointer case,
no-attribute staying unaffected, two different conventions still
distinguishing, coexistence with a trailing noexcept, idempotence, and
a defensive redundant-both-spellings-at-once case).
**Correction (2026-08-29, same day, eighteenth Codex review round on PR
941, commit 06255fc): the sixteenth round's own "restrict never affects¶
mangling, strip it everywhere" fix was itself wrong -- a real
generalization mistake, not a narrow miss.** Fresh evidence, confirmed by
direct compilation rather than mere assertion (g++ -c, inspecting the
real mangled symbols): void f(int **) and void f(int * restrict *)
mangle to two DIFFERENT, simultaneously-declarable Itanium symbols
(_Z1fPPi vs. _Z1fPrPi), so restrict on a NON-outermost pointer level
genuinely IS a standard-mandated, mangling-relevant overload
discriminator -- the sixteenth round's own unconditional strip wrongly
collapsed that pair into one identity. Only restrict on the parameter's
own OUTERMOST, by-value pointer position drops from the mangled name
(void f(int *)/void f(int * restrict) mangle identically, and GCC
even refuses to compile them together as an overload set -- the same
"can't be a real overload pair" signal a genuine by-value cv-qualifier
gives). This is exactly the position-sensitivity the SIXTEENTH round's
own docstring dismissed ("restrict's absence from mangling does not
depend on where in the declarator it sits") -- which was the actual bug:
restrict behaves like cv, not like a pure no-op token. Fixed by
REVERTING the standalone _RESTRICT_WORD_RE-based unconditional strip
entirely and instead folding restrict's three spellings into
_CV_WORD_RE itself, so it goes through the exact same outermost-vs-pointee
position discipline const/volatile already have throughout this
module (_strip_cv_tokens_outside_nesting's existing depth-aware scan,
applied at exactly the same two safe positions as always -- no new
logic needed, since the position discipline was already correct for cv
and restrict needed nothing different from it). Updated tests: TestRestrictQualifierIsAlwaysStripped renamed to
TestRestrictQualifierSharesCvPositionDiscipline
in tests/test_signature_normalization.py, with the previously-wrong
test_restrict_on_non_outermost_pointer_also_stripped (asserting
equality) replaced by
test_restrict_on_non_outermost_pointer_still_distinguishes (asserting
INequality) plus a new idempotence case for the inner-restrict spelling.
This is the second consecutive round to revise the SAME primitive's
restrict handling (sixteenth introduced it, this one corrects its own
overreach) -- direct compiler verification, not just re-reading the
finding text, is now the standard for any future ABI-mangling-behavior
claim in this file before implementing a fix, the same discipline that
already caught this round's own error.
**Correction (2026-08-29, same day, nineteenth Codex review round on PR
941, commit 3aeccc2): an explicit leading :: (forced global-namespace¶
lookup) fragmented identity across producers. ::dep::Thing * and
dep::Thing * -- the identical type, since dep is unambiguous --
canonicalized differently, because nothing stripped the leading
qualifier. Confirmed as a real, already-documented producer spelling
rather than a hypothetical: direct-clang's own qualType preserves an
explicit global-scope qualifier verbatim, the identical fact
tests/test_dumper_scoping_dependency_retention.py's own twenty-sixth-
round docstring already states from the type-matching side of this
codebase (test_globally_qualified_signature_spelling_still_matches).
Fixed with a new _GLOBAL_SCOPE_RE, applied unconditionally (not
position-sensitive the way restrict/cv are, since global-vs-unqualified
lookup never changes which entity a name denotes) up front, before any
of the position-sensitive machinery runs. The match is anchored to
SPECIFIC preceding characters (string start, or immediately after
whitespace/(/</,/*/& -- everywhere a fresh type-name token can
begin) rather than a blanket "not preceded by an identifier character"
negative lookbehind: a first draft used exactly that blanket test and
would have wrongly stripped the genuine, load-bearing :: in
(anonymous namespace)::Foo too (that separator follows a ), not an
identifier character, so the naive test can't tell it apart from a
genuine leading global-scope marker) -- caught before implementing by
walking through the anonymous-namespace case as a check, not discovered
after the fact. New tests:
TestLeadingGlobalScopeQualifierIsStripped in
tests/test_signature_normalization.py (the bare case, the qualifier
surviving after stripping, inside a template argument, inside a callback
parameter, an ordinary namespace qualifier staying unaffected, the
anonymous-namespace regression pin, the member-pointer qualified-name-
prefix case, idempotence).
**Correction (2026-08-29, same day, twentieth Codex review round on PR
941, commit 2a14e2e): the seventeenth round's own "does prefix already¶
carry a calling-convention keyword" check used a bare substring test,
which a coincidentally-named return type defeats.** any(cc in prefix
for cc in _CALLING_CONVENTIONS) matches whenever any of the five
keyword strings appears ANYWHERE in prefix -- but prefix also
contains the return type, which can legitimately CONTAIN one of these
keywords as a substring of an unrelated identifier, e.g.
my__cdecl_result. For my__cdecl_result (*)(int)
__attribute__((stdcall)), the substring test wrongly concluded a
convention keyword was already present (matching __cdecl inside
my__cdecl_result), which both skipped injecting the REAL __stdcall
keyword AND still stripped the trailing attribute text -- silently
merging two distinct callback types (one genuinely __stdcall, one with
no calling convention at all) into one identity. Fixed with a new
_CALLING_CONVENTION_KEYWORD_RE, matched as a genuine whole token
positioned immediately after the declarator's own opening paren
(allowing leading whitespace, mirroring exactly where
_is_declarator_group/_CALLING_CONVENTIONS themselves look for a
convention keyword) rather than a substring test anywhere in prefix.
New tests: a doctest plus
test_return_type_containing_convention_keyword_as_substring in
TestClangTrailingCallingConventionAttributeUnifies (asserting both the
correct injected keyword AND non-collision with the convention-free
form).
Correction (2026-08-29, same day, twenty-first Codex review round on PR #941, commit 05091da): two findings -- one real fix (noexcept integer literals), one real REGRESSION in the nineteenth round's own global-scope-stripping fix, reverted after direct-compilation falsification.
Fix 1 (real, implemented): noexcept's argument is contextually
converted to bool, so noexcept(1)/noexcept(0) are the identical
types as noexcept(true)/noexcept(false) -- confirmed both by direct
compilation (g++: redefinition errors for each pair) and by Clang's
own qualType, which genuinely emits these integer-literal spellings
verbatim (clang -Xclang -ast-dump=json: void (int) noexcept(1)).
_canonicalize_noexcept now recognizes "1"/"0" alongside
"true"/"false", deliberately narrow to exactly these two literals
(a non-0/1 integer constant in this position triggers a
narrowing-conversion diagnostic in real compilers and is not a spelling
this module has confirmed evidence for).
Fix 2 (self-correction, REVERTED): the nineteenth round's own
_GLOBAL_SCOPE_RE-based unconditional stripping of a leading :: was
itself wrong, falsified by direct compilation before reverting rather
than accepted on the strength of a plausible-sounding C++ semantic
argument. Given
namespace dep { struct Thing; }
namespace local { namespace dep { struct Thing; } void f(dep::Thing*); }
qualType for f's parameter prints the BARE, unqualified
dep::Thing * (no leading ::) even though it resolves to
local::dep::Thing -- a type DISTINCT from the true global
::dep::Thing a sibling declaration in the SAME namespace prints WITH
the leading :: (verified via clang -Xclang -ast-dump=json on both
declarations side by side). This module has no scope-tree information;
it operates purely on already-printed type-name text, so it cannot tell
"a genuinely global entity, spelled either way" apart from "an
unqualified name that happens to resolve to a DIFFERENT, locally-
shadowing entity of the same spelling" -- Clang's own choice to include
or omit the leading :: IS exactly that distinguishing signal, so
erasing it can silently merge two non-interchangeable types. Separately:
the nineteenth round's own motivating evidence
(tests/test_dumper_scoping_dependency_retention.py::
test_globally_qualified_signature_spelling_still_matches) turns out to
be for a DIFFERENT subsystem entirely -- matching a signature's type
reference against an already-KNOWN declared type's own qualified_name
within ONE snapshot's dependency-scoping pass, not comparing two
independently-observed SIGNATURES for cross-producer/cross-revision
identity (this function's own job) -- and does not establish that
castxml and Clang ever disagree on this leading :: for one identical
real declaration. _GLOBAL_SCOPE_RE is removed entirely (not merely
disabled); TestLeadingGlobalScopeQualifierIsStripped in
tests/test_signature_normalization.py is renamed to
TestLeadingGlobalScopeQualifierIsPreserved with every assertion
flipped from equality to inequality. This is the SECOND self-correction
on this primitive (the first was the restrict handling, sixteenth ->
eighteenth rounds) -- the common thread both times is that a plausible-
sounding cross-producer-normalization generalization was accepted
without direct-compilation verification first; every future claim about
what two spellings "mean the same thing" on this primitive gets checked
against real compiler/AST-dumper output before implementing, not merely
re-derived from C++ semantics on paper.
Landed (second slice, 2026-08-29): the first slice's own "Next slice"
item (a) -- both dumper modules now track scope as typed segments. The
parser-internal scope state in dumper_clang.py and dumper_castxml.py is
no longer only a bare tuple[str, ...]/context-chain-of-names: each
containing scope is now ALSO recorded as a model.identity.ScopeSegment,
built at the exact point that scope is determined, which is the only point
the AST node kind and its kind-specific payload are still in hand (the
Design section's own reasoning for why a later reconstruction from the
flattened string cannot work). Three new leaf modules under extract/
(extract -> model, ADR-061): extract/headers/scope_segments.py -- the
ONE construction primitive both backends share, so two producers cannot
independently spell the same construct two ways, plus flat_names(), the
parity helper that renders a typed path back to exactly the flat spelling
each backend already built; extract/headers/clang/scope.py -- clang JSON
node inspection (scope_segment_for/anonymous_scope_kind/
anonymous_scope_key); and extract/headers/castxml/scope.py --
scope_path(ctx, el), the structural counterpart of
location.qualified_name's own context-chain walk. On the clang side
_walk threads a scope_path alongside the untouched scope, and
_Decl grew an optional scope_path field (defaulted, so every direct
_Decl construction elsewhere is unaffected); on the castxml side
_CastxmlParser._scope_path sits beside the untouched _qualified_name.
Backward compatibility is structural, not merely tested: the flat
representation is not derived from the typed one and is not modified at
all, and flat_names(typed) == flat is asserted over every categorized
declaration in both backends' tests (tests/test_typed_scope_paths.py),
with qualified_name reconstructed end-to-end from the typed path plus
the element's own name as the castxml oracle rather than a re-implemented
parent walk.
Verified against the real producers, per this section's own
direct-verification standard -- not inferred from plausible AST semantics.
Running clang -Xclang -ast-dump=json and castxml --castxml-output=1
over the same headers established: an inline namespace is a
NamespaceDecl with isInline: true (clang) and does not exist as an
element at all in castxml output (a declaration inside inline namespace
v1 is attributed directly to the enclosing named namespace, confirmed on
both a hand-written header and libstdc++'s own std::__cxx11), so
InlineNamespace is structurally unproducible from castxml -- a backend
capability gap the flat spelling already had, now documented rather than
papered over; an anonymous namespace/record carries no name at all in
either producer; a class-scope castxml record carries an explicit access
attribute while a namespace-scope one carries none (mapped to the one
shared "public" spelling _walk already threads, and non-identity
payload regardless); and only <Namespace>/<Struct>/<Class>/<Union>
are ever referenced as another element's castxml context (checked across
a full <string>/<vector>-including dump), so no other tag needs a
guessed segment kind.
One real over-split was found this way and fixed before it shipped,
which is exactly what the direct-verification standard is for. Two
namespace { ... } blocks in one translation unit REOPEN the same unnamed
namespace -- C++ merges them -- but clang's JSON AST emits one
NamespaceDecl node per block, so a naive positional counter handed
declarations in the first and second blocks different Anonymous.ordinals
and split one real scope into two identities. That is the mirror image of
the sibling collision ordinal exists to prevent, and it was also a
cross-backend divergence: castxml emits a single merged <Namespace>
element for both blocks (verified directly). Fixed by keying the per-parent
ordinal on the entity rather than the block, via clang's own
originalNamespace/previousDecl link (anonymous_scope_key), with the
"no id at all" case (a hand-built test AST) deliberately reported as
"cannot be merged" rather than "same as the last one that also had no id".
Ordinals are counted per parent scope and across all anonymous kinds at
once, so two siblings never share one ordinal even before kind is
consulted; a named LinkageSpecDecl and a ClassTemplateSpecializationDecl
each deliberately produce NO segment from the shared node inspector (the
former is unreachable in real clang output and would need a guessed segment
kind; the latter's trimmed A<double>-style spelling is owned by _walk's
own specialization branch, which must not be given a second opinion).
One architectural constraint found mid-implementation, recorded rather
than worked around. InlineNamespace.version_tag is left empty by both
producers. This repository has exactly one definition of "what an
inline-namespace version tag is" -- qualified_name_segments.version_suffix,
the signal ADR-025's own versioned-inline-namespace-alias handling already
keys on -- and that module belongs to the compare layer, which extract
may not import under ADR-061's dependency direction (scripts/
check_architecture.py failed on exactly that edge, which is how this was
caught, before push). Re-deriving the rule inside extract would create a
second, independently-drifting notion of a version tag -- the precise
duplication this plan's own Governing Invariant forbids -- and relocating
the existing one into model is a real compare-layer migration of its
own, not a drive-by inside this slice. Left empty and documented in the
constructor's own docstring and pinned by a test, so a later slice
populating it has to do so consciously. Nothing is lost meanwhile:
InlineNamespace is identity on name too, so v1 and v2 are already
distinct segments and the tag is a convenience payload, not a
discriminator, at this point in the phase.
What this slice deliberately still does not do. (1) The carrier-field
question (option (a) vs. (b)) remains open, and this slice did not force
it. The typed path is parser-internal state only -- a _walk recursion
parameter and a _Decl field, both alive only during and immediately after
the walk -- attached to no persisted AbiSnapshot/RecordType/Function,
with no schema bump and no serialization change. (2) No
model.identity.EntityId is constructed from real parser output anywhere.
Neither dumper calls any entity_id_for_* constructor; this slice produces
the typed scope data and stops there. (3) diff_filtering.py,
type_reachability.py and finding_identity.py are untouched, and their
existing string-based ambiguity machinery is still the one working
implementation, exactly as the first slice left it. (4) No storage v2 wire
bridge. Two smaller, producer-specific limitations are recorded in the new
modules' own docstrings rather than left implicit: InlineNamespace is
unproducible from castxml (above), and LocalToFunction is unproducible
from EITHER backend today -- clang's _walk stops at a function node by
design (a body is not an ABI declaration surface) and castxml emits no
function-local declarations at all (verified: a struct declared inside a
function body is absent from its output entirely) -- which also means this
slice never had to construct a LocalToFunction.owner, i.e. an EntityId,
which would itself have forced the carrier question.
Next slice, in order (superseding the first slice's own list above, whose
item (a) is what this slice landed): (b) migrate diff_filtering.py/
type_reachability.py -- which still requires answering the carrier-field
question first, since both are post-parse consumers and the typed data this
slice produces does not outlive the parse; then (c) the
finding_identity.py algorithm migration and the storage v2 wire bridge,
each as their own reviewable slice.
Correction (2026-08-29, same day, Codex review on PR #943): the
per-parent anonymous-ordinal counter this same slice introduced was itself
a call-frame-local variable, and a NAMED namespace reopened across two
separate blocks defeats exactly that. namespace N { struct { ... } a; }
... namespace N { struct { ... } b; } walks the second block as a
SEPARATE _walk call from the first -- confirmed directly (clang -Xclang
-ast-dump=json): a reopening block carries originalNamespace/
previousDecl pointing at the first block's node, identical linkage to
the anonymous-namespace-reopening case this slice's own "Landed" paragraph
above already handles, just one level up (the container being reopened,
not an anonymous child of it). A call-local anonymous_ordinals dict
resets between the two calls, so block two's first anonymous child gets
ordinal 0 again -- the SAME ordinal already given to block one's first
anonymous child -- silently merging two genuinely distinct anonymous
scopes under one ScopePath, the exact collision this whole primitive
exists to prevent. Fixed by moving the ordinal state off the call stack
entirely: _ClangAstParser now holds
self._anonymous_ordinal_state: dict[str, dict[str, Any]], keyed by
_clang_scope.anonymous_scope_key computed on the CONTAINER node itself
(the one whose children are about to be numbered) rather than only ever
being called on an anonymous child -- that function's own merge logic
(originalNamespace/previousDecl, falling back to the node's own id) is
generic to any node kind, and was already exactly what this fix needed,
not a new mechanism. A node with no id at all (the root call, or a
hand-built test AST) falls back to its own Python object identity
(f"objid:{id(node)}"), so two such calls never accidentally share
state. New regression coverage: test_live_clang_typed_scope_paths
reopens a NAMED namespace with a distinct anonymous struct in each block
and asserts the two get DIFFERENT ordinals under the same Namespace(...)
segment (confirmed to fail with the pre-fix call-local dict: both reported
Anonymous("struct", 0), verified by reverting the fix locally before
committing it).
A second Codex finding on the same PR, investigated and left open
rather than fixed here. Y in template<class T> struct X<T, int> {
struct Y {}; }; is, per real clang output, nested directly under a
ClassTemplatePartialSpecializationDecl -- a node kind neither in
_RECORD_NODE_KINDS/_SCOPE_NODE_KINDS nor covered by _walk's own
ClassTemplateSpecializationDecl (FULL-specialization-only) spelling
-reconstruction branch, so Y.scope_path (and, checked directly:
Y's pre-existing flat scope/qualified_name too) omits the
partial specialization's own containing scope entirely. This is
confirmed, by direct compilation, to be a PRE-EXISTING gap in the flat
representation this slice's own parity contract requires reproducing
byte-for-byte -- not a regression this slice's typed scope_path
introduces on top of a previously-correct flat spelling. Building a real
partial-specialization scope segment needs a spelling reconstruction
comparable to _specialization_spelling's existing full-specialization
handling (matching template arguments against the partial pattern's own
parameter list, e.g. X<T, int> rather than a fully-concrete X<double,
int>), which is real, independently-reviewable work of its own, not a
small fix bundled into this slice -- left for a follow-on slice rather
than attempted under review pressure here.
Correction (2026-08-29, same day, a SECOND Codex review round on
PR #943): the previous correction's own fix was itself incomplete -- keying
ordinal state by the walked node's identity is wrong whenever a
TRANSPARENT AST wrapper sits between a scope and its anonymous
children. extern "C" { ... } (a LinkageSpecDecl) contributes NO
ScopePath segment of its own (confirmed above: scope_segment_for
returns None for it, so child_scope_path for its children is exactly
scope_path, unchanged) -- but it IS a separate AST node and a separate
_walk call. namespace N { extern "C" { struct { struct X {}; } a; }
struct { struct Y {}; } b; }: confirmed directly (clang -Xclang
-ast-dump=json) that the LinkageSpecDecl sits directly between N and
X's anonymous struct, while Y's anonymous struct is a direct child of
N. Both are, from ScopePath's own perspective, direct children of the
IDENTICAL logical scope (Namespace("N"),) -- but the previous fix's
node-identity key gave them separate _anonymous_ordinal_state entries
anyway (one keyed by the LinkageSpecDecl's own id, one by N's), so both
got ordinal 0. The reopened-namespace fix and this one share a root cause:
ordinal state was keyed by which AST node produced it, when the actual
invariant is about which logical scope the children are entering --
child_scope_path IS that answer, computed once per _walk call already
for an unrelated reason (threading the typed path to children), and using
it as the ordinal-state key closes BOTH cases with one rule instead of two
special-cased identity fallbacks: a reopened namespace's two blocks
compute the identical segment tuple (construction depends only on
name/isInline, never on which node/block produced it), and a transparent
wrapper's child_scope_path is by definition identical to its own
scope_path. ScopeSegments are frozen/hashable, so the tuple is a valid
dict key directly -- no string-identity/objid fallback needed at all, which
also means the previous fix's "objid:{id(node)}" fallback branch (for a
node with no id) is gone, not merely untested: child_scope_path is
always a real, comparable value regardless of whether the node producing
it happens to carry an id. New regression coverage: test_live_clang_
typed_scope_paths adds an extern "C" block beside a plain anonymous
struct in the same namespace and asserts distinct ordinals (confirmed to
fail against the node-identity-keyed fix, both reporting ordinal 0,
verified by reverting to that version locally before committing the real
fix). This is the same "verify before generalizing, and verify again when
generalizing a second time" discipline the earlier restrict/calling-
convention/global-scope corrections on this same plan established --
two review rounds in a row found the previous round's fix real but
incomplete, which is exactly what direct-compilation verification is for.
Decision (2026-08-29, PR #943): the carrier-field question is CLOSED as
option (a) -- EntityId is computed once, at parse time, and carried on
the model object. The Design section above deliberately left this open
("this plan does not pick one under continued review pressure a third
time") and stated the consequences of each branch; this entry records the
choice its implementation PR made, and why, so no later slice has to
re-litigate it. Three reasons, each checked against this plan's own text
rather than asserted:
- Option (b) leaves two live representations of one concept standing
indefinitely. Its own premise is that
diff_filtering.py/type_reachability.pykeep their bespoke string-based ambiguity trackers until Phase 6'sSemanticIRassembly -- a phase with no scheduled start, several phases out. This plan's Governing Invariant is "one concept, one representation, everywhere it is used, never two," with an explicit "delete after consolidating -- same PR or the very next one, never eventually" rule. An unscheduled deferral is precisely the "eventually" that rule names. - Option (b) blocks Phase 3 as sequenced. The "not contained to Phase 2"
paragraph above already establishes this: Phase 3's public-surface graph
keys
declaration/typenodes byEntityId, and its graph builder is a post-parse consumer by construction. Under option (b) Phase 3 must move to land with or after Phase 6, cascading the deferral into Phases 3/4/5. Option (a) keeps the stated phase ordering intact. - Option (a) is the only branch where "computed once" is literally true.
The typed
ScopePathexists only during the AST walk (the second slice's own finding). Recomputing identity later is not merely inconvenient, it is structurally impossible from an already-parsed model object -- so a carrier is not a cache of something re-derivable, it is the only place the answer can live at all.
Consequently, the conditionals this plan deliberately wrote in both
branches now resolve to their option-(a) halves: the "Tests" section's
carrier-vs-no-carrier pair resolves to the first shape (the field is
populated for every declaration kind immediately after parsing, and a
static check confirms the resolver is never called on an already-parsed
RecordType/Function outside the parser); the "Acceptance criteria"
section's deletion of diff_filtering.py/type_reachability.py's
string-based helpers is in scope for this phase rather than moving to
Phase 6; and Phase 6's own conditional paragraph ("If Phase 2's
implementation PR resolves ... as option (b)") is now unreachable and
carries no work for that phase. One clause of the option-(a) test shape
is NOT satisfied by this slice and is not claimed to be -- "round-trips
through serialization.py unchanged"; see the finding immediately below.
Finding (2026-08-29, same day, found while implementing the decision
above): option (a) is necessary but NOT sufficient for the consumer
migration, and this plan's Phase 2 sequencing assumed it was. The
Acceptance-criteria text above reads as though choosing option (a) makes
diff_filtering.py's migration land in the same slice. Reading the three
actual call sites, it does not -- three independent blockers, none of them
about the carrier field itself:
- A carrier that is not persisted is not available to a post-parse
consumer either.
compare old.json new.jsonis a first-class, documented invocation; those two snapshots come back throughsnapshot_from_dict, which cannot reconstruct anEntityIdwithout theScopePath-preserving storage v2 wire DTO this phase's own Design section specifies (and which the second slice's "Next slice" list already scopes as slice (c)). A consumer keyed onentity_idwould therefore answer one way for two live binaries and another for two dumped snapshots OF those same binaries -- a worse defect than the bare-name collision it set out to close, and one that no amount of care insidediff_filtering.pycan prevent. Encoding it with a stopgap flattening is not an option: this section's own "FlatteningScopePath/extrainto the existing barequalified_name/discriminatorstrings is not a lossless bridge" paragraph already rejected exactly that, by name. _root_type_namedoes not operate on a model object at all. It takes aChange(checker_types) and slices itssymbolSTRING; the set_find_opaque_typesreturns is matched against that string in_downgrade_opaque_type_changes. Keying the set onEntityIdtherefore requiresChangeitself to carry one -- which is thefinding_identity. pyalgorithm migration this phase already scopes as slice (c), not part of slice (b)._find_by_value_typesis not a name-keyed index; it is a substring scan. It asks whether an opaque type's spelling occurs inside a parameter's or return value's own type text (tname in rt). AnEntityIdcannot be substring-matched inside a type spelling: turning that into an identity join requires resolving a type REFERENCE to a declaration, which istype_reachability.py's job -- the module this phase's own text (and this slice's task) explicitly defers on the strength of its eleven-plus-finding regression history.
So the correct sequencing, recorded here rather than discovered again mid-slice: option (a)'s carrier lands first (this slice), persistence via the storage v2 wire bridge second, and the post-parse consumer migrations -- with the string-based helper deletion the Acceptance criteria require -- only after BOTH, since neither is optional for a correct migration. The Acceptance criteria's "deleted, not kept alongside" therefore still holds for this phase; it does not hold for this phase's third slice, which has no working replacement to switch those consumers onto yet. Deleting them now would leave the same "neither the old mechanism nor a usable replacement" hole the criteria themselves already reject for option (b).
Landed (third slice, 2026-08-29): the carrier field, populated by both
header-AST backends. No consumer migration -- see the finding above for
why that is a separate slice, not an omission.
RecordType/EnumType (model/entities.py) and Function/Variable
(model/declarations.py) each gained one field, entity_id: EntityId |
None = field(default=None, kw_only=True, compare=False). Each property is
deliberate: None is the honest answer for the direct constructor call an
external consumer of these public dataclasses makes (never a fabricated
identity reconstructed from qualified_name, which this section already
established is structurally insufficient); kw_only=True, appended last,
means no existing positional argument's slot moves (the same reasoning
Function.hidden_friend_owner already records); and compare=False keeps
__eq__ bit-for-bit what it was, since identity is derived from the
declaration and folding it into equality would make two otherwise-identical
model objects differ purely on whether a producer wired it -- the same
identity-vs-payload split identity.Record.access applies one level down.
typedefs/constants get no carrier, and this is a real limitation
rather than an oversight: both are dict[str, str] on AbiSnapshot, not
dataclasses, so entity_id_for_typedef/entity_id_for_constant have
nowhere to write to. Giving them one means giving each a model dataclass
first, which is its own change to a persisted snapshot shape.
Populated at every construction site of the four carriers, in both
backends: extract/headers/clang/records.py (both the opaque and the
complete branch), clang/enums.py, clang/functions.py,
dumper_clang.py's parse_variables, and the four castxml counterparts
(castxml/records.py, castxml/enums.py, castxml/functions.py,
dumper_castxml.py's parse_variables) -- each reading the typed
scope_path the second slice threaded, never re-deriving one. Two
supporting changes were needed and are worth not rediscovering: both
function builders now hoist is_extern_c/is_const/is_volatile/
ref_qualifier/is_variadic into locals shared by the model object and
the identity constructor, so the two cannot form independently-computed
opinions about the same fact; and the clang function builder keeps the
declaration's own unqualified leaf_name separately from name, which
that builder requalifies for a template specialization's member (a
display/owner-matching spelling, not a leaf identity).
The mangled-vs-extern "C" routing is producer-local, by design.
entity_id_for_function/_variable require the caller to have already
established that mangled_name is a GENUINE mangling; that determination
lives in finding_identity.is_real_mangled_name, which model/extract
may not import (a compare-layer module; ADR-061/ADR-063 D10). Both
backends already answer the same question locally for the model's own
is_extern_c field, so the identity call reuses exactly that answer:
mangled_name=None if is_extern_c else mangled, is_extern_c=is_extern_c
-- the same order resolve_function_identity itself applies (linkage
first, mangling second). Verified end-to-end against BOTH real producers on
one probe header rather than inferred: clang -Xclang -ast-dump=json and
castxml --castxml-output=1 agree exactly on every kind -- ns::Outer,
ns::Outer::Inner (whose scope is (Namespace("ns"), Record("Outer")),
the Record-vs-Namespace distinction a flat spelling cannot carry),
ns::Color, the f(int)/f(double) overload pair (two distinct
("mangled", ...) ids), c_fn/c_var (both ("extern_c",)), and
ns::gVar (("mangled", "_ZN2ns4gVarE")).
One real cross-backend divergence found by that verification, confirmed
PRE-EXISTING and left as-is rather than papered over in this primitive.
For extern "C" int c_var; with no export evidence supplied, castxml emits
a bogus pseudo-Itanium mangled="_Z5c_var" (confirmed by reading the raw
XML) while clang emits the bare c_var, so the two backends' EntityIds
disagree -- but they disagree because the two backends' own
Variable.mangled fields already disagree, before this slice and
independently of it. castxml offers no local signal for a C-linkage
variable at all (its extern="1" attribute marks the extern storage
class: the C++ ns::gVar carries it and the extern "C" c_var does
not, verified directly), which is why the pre-existing real-ELF-export
override is the mitigation -- and with real export evidence supplied, the
normal dump path, both backends produce the identical
((), VARIABLE, "c_var", ("extern_c",)), confirmed by re-running the same
parse with the symbol present. Inventing a castxml-only C-linkage
heuristic inside an identity constructor would be exactly the "solve a
shape the producer never gave us" overreach the eighth round's PE-
decoration finding already declined, one layer lower.
Not persisted, deliberately, and this is the one clause of the Design
section's option-(a) test shape this slice does not satisfy.
serialization.snapshot_to_dict drops the field
(storage/entity_id_codec.py, a new storage-owned leaf module mirroring
fact_codec.py/enum_codec.py's role), SCHEMA_VERSION does not move,
and a reloaded snapshot carries entity_id=None for every declaration.
The alternative was a stopgap flattening this section already rejected by
name as a silently-collapsing round trip; the real encoding is the storage
v2 wire DTO, still slice (c). Three tests pin the drop as a deliberate
property rather than an accident -- including that the snapshot stays
json.dumps-able at all, which is the concrete regression at stake:
EntityId.kind is a plain Enum (not a (str, Enum)) and scope is a
tuple of dataclasses, so an asdict()-ed carrier reaching json.dumps
raises TypeError outright. Until persistence lands, no diff/report
consumer may read this field: it is present on an in-memory dump and
absent from a reloaded one, so a consumer keyed on it would answer
differently for the same two libraries depending only on whether a
snapshot was written to disk in between.
Tests. tests/test_entity_id_carrier.py (new): the dataclass contract
(default None, kw_only, excluded from equality -- parameterized over
all four carriers rather than asserted for one); the non-persistence
properties above; the live-backend population tests for every declaration
kind on clang and castxml, sharing one assertion helper so the two
producers must AGREE rather than each merely be self-consistent; the
record-vs-namespace collision closed by construction (struct B { struct C
{}; }; and namespace B { struct C {}; } parsed as two separate headers
-- the real shape of the bug, one spelling on each side of a comparison --
producing the identical qualified_name "B::C" and two different
EntityIds, on both backends); and the static check the Design section's
option-(a) branch asks for, as a real AST scan for entity_id_for_* CALL
nodes across all of abicheck/ (attribute-style calls included, docstring
mentions excluded), asserting every call site is a header-AST producer.
That check carries its own guard -- an assertion that the scan actually
finds the four real producer modules -- since an emptiness assertion alone
passes just as happily against a broken scanner.
Correction (2026-08-29, same day, found by the third slice's own full
local suite before review): adding the carrier broke every dump of a
lambda-bearing library outright, and the cause was not the carrier.
qualified_name_segments.renumber_anonymous_closure_identities -- the pass
that replaces a closure marker's raw :line:col with a snapshot-stable
ordinal, i.e. the machinery identity.environment_taint rests on -- walks
every dataclass reachable from functions/variables/types/enums and
assigns rewritten strings back with setattr. EntityId is frozen, so the
first lambda marker anywhere in a snapshot raised FrozenInstanceError and
aborted the dump; ten end-to-end tests in
tests/test_identity_taint_end_to_end.py failed at once, none of which the
targeted per-module runs this slice had done so far would have reached.
Fixed at the root rather than by excluding the new field from the walk: a
frozen dataclass is now rebuilt via dataclasses.replace. Excluding it was
considered and rejected -- a closure marker surviving unrewritten inside
an identity carrier is a path/line-tainted identity sitting next to the
normalized ones, which is precisely the taint class that pass exists to
remove, so the crash would have been traded for a silent version of the
same bug. The regression coverage is stated about the walk primitive
itself over six independently-chosen frozen shapes (nested in a tuple, a
list, a dict value, two levels deep, under a mutable parent, under a
frozen parent) plus an init=False field and an unchanged-value
object-identity case, alongside the concrete entity_id case through the
public entry point, so the next frozen model field is covered by
construction rather than by someone remembering; all nine were confirmed
to fail against the pre-fix walk by reverting it locally. Worth stating
plainly as the transferable lesson: a purely additive model field is not
automatically inert, because a generic object-graph walk elsewhere in
the codebase makes every new field its input.
Correction (2026-08-29, same day, Codex and CodeRabbit independently on
PR #943): the third slice wired the clang producer's own is_extern_c
into the identity constructor, and that field is broader than the
constructor's is_extern_c parameter means. parse_functions'
mangled local falls back to the bare name whenever clang emits no
mangledName, and the long-standing mangled == name heuristic reads
that fallback as C linkage. Confirmed by direct compilation that clang
omits mangledName entirely for three real shapes -- an uninstantiated
function template, an uninstantiated class-template method, and a
class-template pattern's static member -- and then reproduced the
consequence through the real parser: A::f/B::f collapsed onto one
EntityId, as did C::S::v/D::S::v through the variable path, because
the extern_c branch deliberately forces scope=() and drops the
signature. Fixed by gating the heuristic on the AST key's own presence
rather than the fallback-widened value's truthiness, in both the function
and the variable path; verified the narrowing cannot break real C linkage
(clang -x c on a plain C header and clang -x c++ on an extern "C"
block both emit an explicit mangledName equal to the name, functions and
variables alike), and pinned that as a negative control. castxml needs no
counterpart -- verified directly that it emits nothing at all for an
uninstantiated template, so its own absent-mangled attribute genuinely
does mean C linkage.
The transferable point, which generalizes past this one field: every
evidence-tier branch this constructor added that ERASES a discriminator
(scope=() for mangled/extern_c, leaf_name="" for mangled) turns
any over-broad predicate feeding it into an identity collapse, silently.
Those erasures are correct -- they exist to stop one entity fragmenting
across evidence tiers, per the third/fourth Codex rounds above -- but they
invert the usual failure mode: a wrong answer here merges two entities
rather than splitting one, and merges are the direction no downstream
consumer can recover from. A producer wiring a new caller into these
constructors owes the same check this correction had to make after the
fact: is the linkage/mangling signal I am passing authoritative, or is
it a display fallback that happens to be true? The second slice's own
is_extern_c field had been an inert inaccuracy for as long as it
existed; only the carrier made it load-bearing.
Correction (2026-08-29, same day, Codex review on PR #943): the carrier
also stopped being inert for a second, unrelated reason -- dumper_
hybrid.py rewrites a declaration's mangled field in two places AFTER a
producer already resolved its entity_id from the ORIGINAL spelling, and
neither rewrite touched the carrier. Reconciling a castxml ctor/dtor
synthetic placeholder key to clang's real mangled name (_merge_functions)
left the reconciled declaration's own mangled field correct while its
entity_id still carried the synthetic, no-such-symbol-exists placeholder
inside its own "mangled" tag; normalizing a Mach-O linker symbol's
leading underscore (merge_snapshots) had the identical shape, just for a
real-not-synthetic spelling. Before this slice, the same rewrite site
already existed but touched an inert field -- nothing kept it in sync with
anything, because nothing downstream read it as an identity. Fixed two
ways, matching which side of the rewrite has a matching declaration to
adopt: the ctor/dtor case adopts match's entire, already-correctly-
resolved entity_id wholesale (the clang-side declaration for the SAME
real symbol); the Mach-O case has no "other side" to borrow from, so it
rebuilds only the "mangled" tag's spelling via a new, general
model.identity.with_mangled_name(entity_id, new_mangled_name) helper,
which is a no-op on an extern_c/sig-tagged identity (or None) since
neither was derived from the mangled spelling in the first place. Regression
tests for both live in tests/test_entity_id_carrier.py (not tests/
test_dumper_hybrid.py, despite testing that module's own merge logic --
they're pinning THIS module's identity-carrier contract, the same reasoning
the file's own module docstring already states for keeping every carrier
test in one place), and both were confirmed to fail against the pre-fix
dumper_hybrid.py before landing this correction.
Next slice, in order (superseding the second slice's list, whose item (a)
that slice landed and whose item (b) this slice's own finding above
re-sequences): (c1) the storage v2 wire bridge -- storage/entity_ids.py's
ScopePath-preserving to_dto()/from_dto() plus the v1 migration
adapter -- and, on top of it, real persistence of the entity_id carrier
with a schema bump and the round-trip test the Design section names; then
(c2) the finding_identity.py algorithm migration, which is also what
gives Change an EntityId to be keyed on; then (b) the post-parse
consumer migrations, now unblocked: diff_filtering.py's
_find_opaque_types/_find_by_value_types/_root_type_name and
type_reachability.py's eleven-plus-case ambiguity machinery, with their
string-based helpers deleted in the same PR per the Acceptance criteria.
Landed (fourth slice, 2026-08-30): (c1)'s bridge half, not its
persistence half. storage/entity_ids.py gains
domain_entity_id_to_dto/domain_entity_id_from_dto, a wire-schema-
versioned (DOMAIN_ENTITY_ID_SCHEMA_VERSION = 2) pair operating on
model.identity.EntityId directly -- kept deliberately separate from this
module's own pre-existing EntityId/OccurrenceId packed-key wire DTO
(unchanged, still the shape a caller that only needs a flat, orderable,
key-producing identity reaches for; storage/identity.py's re-export is
therefore unaffected, exactly as the Design section's own note predicted).
domain_entity_id_to_dto encodes ScopePath as an explicit list of typed
segment records ({"kind": "namespace"|"record"|"inline_namespace"|
"anonymous"|"local_to_function", ...}, one entry per segment, each
carrying that segment's own fields -- Record.access, InlineNamespace.
version_tag, Anonymous.{kind,ordinal}, and LocalToFunction's own
owner/block_ordinal, with owner itself recursively encoded as a full
nested EntityId document, not a bare string, since LocalToFunction.owner
is a whole EntityId by the primitive's own design (see that segment's own
docstring in model/identity.py) -- rather than one rendered
qualified_name string, which is exactly the lossy encoding the Design
section's own finding named (a record nested in a record vs. the same
bare names nested in a namespace both render "A::B", and a
from_dict-style reconstruction from that string alone cannot recover
which one it was). domain_entity_id_from_dto dispatches on
schema_version: an absent version or an explicit 1 routes through
_domain_entity_id_from_v1_dto, a best-effort migration adapter matching
the Design section's own stated shape (each ::-separated
qualified_name component becomes an untyped Namespace segment, the
last becomes leaf_name, and a non-empty discriminator becomes a
single-entry extra tuple) -- documented, in the test suite and not only
in prose, as a lossy reconstruction never asserted equal to a fresh v2
encoding of the same logical declaration. Tests in
tests/unit/storage/test_entity_id_domain_bridge.py: a Hypothesis
round-trip property (from_dto(to_dto(x)) == x) generating arbitrary
EntityIds across every ScopePath segment kind, plus dedicated cases for
the exact Record-vs-Namespace and InlineNamespace-vs-Namespace
counterexamples the Design section's own finding raised (confirmed
distinct both before and after the round trip, and confirmed to encode to
different wire documents -- not merely to reconstruct correctly, which
alone would not rule out two different domain objects sharing one
document by coincidence), a case pinning that Record.access stays
non-identity across the bridge the same way it is in-memory (the wire
document itself still records the real access level; only equality
ignores it), recursive LocalToFunction.owner round-trips (including an
owner that is itself local to another function), the mangled-variable
branch's degenerate scope=()/leaf_name="" shape, and malformed-input
refusals (an unknown schema_version, an unrecognized segment "kind"
tag, a non-mapping document, an unrecognized ScopeSegment type passed
directly to to_dto). Full fast unit suite green; mypy abicheck/ and
ruff both clean; tests/unit/storage/test_landed_surface.py's table
check (this plan's storage-v2 sibling document,
docs/contribute/plans/storage-format-v2.md) updated with the three new
names.
Correction (2026-08-30, same day, Codex review on PR #949): three
malformed-document leniency gaps in the reader half, all sharing one root
cause -- a boundary door that coerced or defaulted instead of refusing, the
same class of defect this module's own pre-existing EntityId/
OccurrenceId guards exist to close, just not yet extended to the new v2
bridge. (1) schema_version was dispatched by == against 1; since
bool subclasses int and compares across int/float in Python,
True == 1 and 2.0 == 2, so a document carrying either silently
dispatched to a parser for a version it never declared (a true
schema_version alongside real v2 scope/extra fields ran the lossy
v1 adapter, discarding data the v1 shape cannot even express). Fixed by
_entity_id_schema_version, which rejects any non-int-excluding-bool
value outright rather than comparing it. (2) ordinal/block_ordinal
were read via the existing _instance_of(value, int, ...) guard, which
accepts bool for the identical subclassing reason -- so a document with
ordinal: true and one with ordinal: 1 reconstructed to equal,
same-hash Anonymous/LocalToFunction segments, two structurally
different wire values collapsing onto one identity. Fixed by
_ordinal_field, a strict integer guard excluding bool for both fields.
(3) access, version_tag, and extra were read via .get(key, default)
even though the writer (_domain_segment_to_dict/domain_entity_id_to_dto)
always emits every one of them regardless of value -- so a v2 document
truncated or hand-edited to omit one of these silently read as "the
producer had nothing to say" rather than "this document is malformed,"
exactly the shape storage.guards.row_sequence's own docstring already
names as this package's central invariant. Fixed by reading all three via
_required_field, matching how every other field in this bridge (scope,
kind, leaf_name) was already read. New tests for all three in
tests/unit/storage/test_entity_id_domain_bridge.py
(TestMalformedDocuments), each starting from a real domain_entity_id_to_dto
output and mutating exactly the field under test rather than a hand-typed
fixture, confirmed to fail against the pre-fix reader.
Correction (2026-08-30, same day): the paragraph originally here
overclaimed that no entity_id carrier field existed yet and that the
carrier-field question was still unresolved -- both wrong. The carrier
field (RecordType/EnumType/Function/Variable.entity_id) was added
by the THIRD slice, the day before this one, resolving the open question as
option (a); this fourth slice's own opening line ("populated by both header-
AST backends... schema-inert at the time: dropped before reaching the
wire") already said so correctly two sections up. The error was writing
this closing paragraph from the SECOND slice's own stale "what it doesn't
do yet" framing instead of the current state. What this slice actually
still does not attempt: "real persistence of the entity_id carrier with a
schema bump," the second half of (c1) as originally scoped, stays open --
that is what actually wires this bridge into AbiSnapshot serialization (a
SCHEMA_VERSION bump and a real snapshot-level round-trip test). (c2) (the
finding_identity.py algorithm migration) and (b) (the post-parse consumer
migrations) remain open exactly as stated above. See the fifth slice below
for the persistence half.
Landed (fifth slice, 2026-08-30): (c1)'s persistence half. The
entity_id carrier now round-trips through serialization.py
(SCHEMA_VERSION 27 -> 28). storage/entity_id_codec.py gained
encode_entity_ids(d, snap)/decode_entity_id(raw), replacing the interim
drop_entity_ids this section originally landed: encode_entity_ids
cannot operate on the already-dataclasses.asdict()-ed snapshot dict alone
the way this package's other codecs (fact_codec.encode_fact_fields) do,
because asdict() recurses into a ScopeSegment's own fields but loses
which dataclass (Namespace/Record/...) produced them -- exactly the
type tag domain_entity_id_to_dto needs. So it is given both the
asdict()-ed dict and the original, still-typed AbiSnapshot, pairs each
declaration list by position (asdict() preserves list order), and
re-derives each declaration's wire-encoded entity_id from the ORIGINAL
typed object rather than from what asdict() already flattened. A
declaration with entity_id is None gets no key at all (sparse, matching
OccurrenceId.to_dict's existing convention), rather than an explicit
null -- so a pre-persistence snapshot and a persisted one whose
declarations never resolved an identity serialize identically.
snapshot_from_dict wires entity_id=decode_entity_id(f.get("entity_id"))
into all four reconstruction sites (Function, Variable, RecordType via
decode_record_facts's neighbor, and _enum_type_from_dict).
decode_entity_id needs no schema-version-gated distinction the way
fact_codec.decode_fact does (Fact[T]'s "predates the carrier" vs.
"malformed" split): a missing/None entity_id is genuinely optional even
on a snapshot written by the current build (not every declaration resolves
one), so absence always means the same thing regardless of which side of
v28 the snapshot was written on.
One AI-readiness hard-cap consequence, worth recording so it isn't
rediscovered. serialization.py was already at this repo's 2000-line
hard cap before this slice (load_bundle_facts/save_bundle_facts are
kept as long, single, ruff format-noncompliant lines specifically to stay
under it -- a pre-existing, deliberate trade-off this slice found and
preserved rather than "fixed": running ruff format on the whole file
would reflow those two functions across ~20 more lines and push the file
over the hard cap). New entity_id encode/decode call sites and the
SCHEMA_VERSION history-comment entry were kept as terse as the existing
convention allows, and ruff format/ruff check were run scoped to confirm
no NEW formatting drift was introduced beyond that pre-existing, known
exception -- not to "fix" it, which would have been a scope-creeping,
cap-violating change bundled into an unrelated slice.
Tests. tests/test_entity_id_carrier.py's TestCarrierIsNotPersisted
(third slice) is replaced by TestCarrierIsPersisted: a real
json.dumps/json.loads round trip (not just the dict form) reconstructs
an identical EntityId for all four carriers; the Record-vs-Namespace
counterexample from the Design section's own finding survives the round
trip at the WHOLE-SNAPSHOT level (not only in the storage-layer bridge's
own unit tests from the fourth slice); a declaration with no resolved
identity reloads as None, not a reconstructed guess; and a snapshot whose
schema_version is pinned to 27 with entity_id keys stripped (simulating
a genuine pre-v28 document, not merely an old in-memory object) reloads
every carrier as None rather than raising. tests/test_baseline_pinning.py
and tests/test_serialization_roundtrip.py's own SCHEMA_VERSION ==
27 pins updated to 28. Full fast unit suite green; mypy abicheck/ and
ruff check clean; golden-marked tests green (a v27-stamped fixture
snapshot reloads with entity_id=None for every declaration, exactly the
backward-compatible degradation this slice's own design states -- no golden
fixture needed regeneration).
Correction (2026-08-30, same day, Codex review on PR #949): the claim
below that snapshot_cache.py needs no bump was wrong -- _SNAPSHOT_CACHE_
VERSION bumped to "22". The original reasoning ("it caches parsed
AbiSnapshot objects... no consumer reads this field yet") missed that
store_key/lookup_key round-trip through write_snapshot/load_snapshot
-- a real on-disk JSON write, not an in-memory object cache surviving one
process's lifetime -- so a cache entry written before this slice really
does have entity_id=None for every declaration, the same way an on-disk
v27 snapshot does. The consequence the "no consumer reads it yet" argument
missed: snapshot_to_dict stamps converted["schema_version"] =
SCHEMA_VERSION unconditionally on every write, including a warm cache
entry re-saved to a caller-visible file -- so a stale pre-this-slice cache
entry would re-serialize claiming schema_version 28 while never having had
the chance to resolve identities a genuine v28 extraction would. Exactly
the same staleness shape the file's own pre-existing v21 bump (Function.
is_compiler_generated, schema v27) already documents and exists to
prevent -- this slice missed applying its own established precedent to
itself. Fixed by bumping _SNAPSHOT_CACHE_VERSION with a matching
v22 comment entry.
What this slice deliberately does not attempt. (c2) (the finding_identity.py algorithm
migration, which is also what gives Change an EntityId to key on) and
(b) (the post-parse consumer migrations -- diff_filtering.py's
_find_opaque_types/_find_by_value_types/_root_type_name and
type_reachability.py's ambiguity machinery) remain the two open items
before Phase 2 can be considered complete.
Correction (2026-08-30, same day, Codex review on PR #949): three
findings, fixed in the same PR. (1) decode_entity_id used if not raw:
return None -- truthiness, not raw is None -- so a genuinely malformed
wire value that happens to be falsy ({}, [], "", False, 0) was
silently read as an honest "never resolved an identity" rather than
reaching domain_entity_id_from_dto's own validation to be refused. Fixed
by testing raw is None specifically; regression test parametrized over
all five bogus values in tests/test_entity_id_carrier.py. (2) and (3) were
both scripts/check_architecture.py violations from incidental
ruff format reflow of pre-existing, already-ruff format-noncompliant
code the initial push happened to touch: serialization.py's
debt-no-growth gate (a stricter, separately-tracked adoption-debt ceiling
of 1985 lines -- below the 2000-line AI-readiness hard cap this file was
already sitting at) and a tests/test_serialization_roundtrip.py
new-test-size violation (1200-line test-file cap) from unrelated
multi-line reflows of long assertion lines that carried no functional
change. Both fixed by reverting the incidental reflow (restoring the exact
pre-existing single-line forms outside this PR's own touched lines) rather
than "fixing" the formatting, which would have been unrelated scope creep
bundled into this slice -- and, for serialization.py specifically, by a
genuine simplification Codex's own finding prompted looking for:
decode_entity_id's four separate entity_id=decode_entity_id(...)
keyword arguments (one inline at each of Function's, Variable's,
RecordType's, and EnumType's own reconstruction site) collapsed into
one new decode_entity_ids(d, functions=funcs, variables=variables,
types=types, enums=enums) call, set post-construction (none of the four
carrier-bearing dataclasses are frozen) -- a real reduction in
serialization.py's own footprint, not merely a line-count-driven
rearrangement, since the four sites' near-identical one-line additions are
exactly the kind of small, repeated wiring a single codec-owned helper
should absorb. encode_entity_ids also gained a return value (returns d)
so its own call site could fold back into the original single-statement
_sets_to_lists(encode_entity_ids(d, snap)) shape the pre-existing
_sets_to_lists(drop_entity_ids(d)) line already used, instead of a
separate statement.
Correction (2026-08-29, same day, Codex review on PR #943): the
over-broad-extern-C fix above closed one collision but left a sibling one
open -- two uninstantiated function/method templates that share scope, leaf
name, and an identical (possibly empty) ordinary parameter list, differing
ONLY in template-parameter kind. template<class T> void f(); and
template<int N> void f(); are two distinct, legally-overloaded
declarations (real C++ disambiguates them via explicit template arguments,
e.g. f<5>()), but neither is is_extern_c (both are genuinely mangled by
any real instantiation) nor do they differ in param_types/qualifiers --
so entity_id_for_function's "sig" fallback tuple, built only from the
ordinary parameter list, still collapsed them (confirmed by direct
compilation: distinct: 1 of 2 declarations before this fix). Fixed by
extracting each FunctionTemplateDecl's own per-position
parameter-KIND signature (function_template_param_kinds in
extract/headers/clang/functions.py -- "type"/"template"/
"nontype:<type-spelling>" per position, stopping at the first
non-parameter child) at parse time (threaded through _Decl/_walk/
_categorize in dumper_clang.py, set only on the direct FunctionDecl
child of a FunctionTemplateDecl, never inherited further down) and
folding it into entity_id_for_function's new template_param_kinds
parameter -- tagged ("tmpl", *template_param_kinds) and appended only
when non-empty, so an ordinary non-template function's extra tuple is
unchanged byte-for-byte. castxml needs no counterpart: it does not emit
uninstantiated function/method template declarations as parseable
snapshot entries at all (a different, pre-existing asymmetry between the
two backends, not one this fix introduces or has to reconcile). Regression
test in tests/test_entity_id_carrier.py
(test_live_clang_template_param_kind_discriminates_overloaded_templates),
confirmed to fail (collapsing to one EntityId) against the pre-fix
entity_id_for_function.
Correction (2026-08-29, same day, Codex review on PR #943): the
template-parameter-kind fix above still missed a second, sibling
collision -- two legal overloads differing only in parameter
packness. template<class T> void f(); and template<class... T>
void f(); share scope, leaf name, and an identical ordinary parameter
list, and the first version of function_template_param_kinds recorded
only parameter kind ("type"/"template"/"nontype:..."), so both
still reduced to the identical ("type",) -- reproduced end to end:
distinct: 1 of 2 declarations. Confirmed by direct compilation that
clang tags a pack parameter with isParameterPack: true on all three
parameter-node kinds (TemplateTypeParmDecl, NonTypeTemplateParmDecl,
TemplateTemplateParmDecl alike), omitting the key entirely otherwise.
Fixed by appending a trailing "..." to each discriminator entry when
that flag is set, giving e.g. "type" vs. "type...",
"nontype:int" vs. "nontype...:int". Regression test in
tests/test_entity_id_carrier.py
(test_live_clang_template_param_packness_discriminates_overloaded_templates),
confirmed to fail against the pre-fix (kind-only) version.
Correction (2026-08-29, same day, Codex review on PR #943): the opposite
hazard -- a non-semantic template-parameter RENAME changing identity.
template<class T, T N> void f(); and template<class U, U N> void
f(); are the identical declaration under a pure rename, but clang's own
qualType for the non-type parameter N spells its dependent type
literally as the type parameter's own name (confirmed by direct
compilation: "T" vs. "U" respectively), and the discriminator built
directly from that spelling changed too -- reproduced end to end: the two
revisions produced unequal EntityIds. Fixed by canonicalizing a
non-type parameter's own declared type against the PRECEDING type
parameters' names before joining into the "nontype:" entry, replacing
each type-parameter name with its 0-based position
(_canonicalize_dependent_type_param_spelling, a whole-word
regex substitution so a name that is a substring of another, e.g. T
inside TT, is never partially replaced) -- ("type", "nontype:T") and
("type", "nontype:U") both become ("type", "nontype:type-param-0").
Verified this does not collapse a genuinely different non-type parameter
type (template<class T, int N> still keeps nontype:int, distinct from
either rename revision). Regression test in
tests/test_entity_id_carrier.py
(test_live_clang_template_param_rename_does_not_change_identity).
A second finding from the same review round -- two overloads
distinguished only by a requires-clause (e.g. template<class T>
requires C1<T> void f(); vs. the same constrained by C2<T>) still
collide, since clang's own ConceptSpecializationExpr node (confirmed by
direct compilation to appear as a FunctionTemplateDecl child right
after its TemplateTypeParmDecl) carries no concept name or a
resolvable reference to one anywhere in its own JSON subtree -- verified
directly, not assumed: every key on the node and its
ImplicitConceptSpecializationDecl child was inspected, and neither
carries anything but synthetic AST ids and dependent-type placeholders
(type-parameter-0-0). Recovering the concept's actual name would need
either a different clang AST-dump mode/flag or the raw header source
text sliced at the node's own range offsets -- and _ClangAstParser
deliberately consumes only an already-parsed JSON tree (this module's own
docstring), with no source text available to it. Recorded here as a
known gap rather than attempted as a fragile source-offset hack: a
requires-clause-only overload pair is the one template-overload shape
this discriminator still cannot distinguish.
Correction (2026-08-29, same day, CodeRabbit review on PR #943): a third
sibling collision -- a template-TEMPLATE parameter's own NESTED arity.
template<template<class> class TT> void f(); and
template<template<class, class> class TT> void f(); are two more legal
overloads sharing scope, leaf name, and an identical ordinary parameter
list; the TemplateTemplateParmDecl branch recorded only the bare
"template"/"template..." tag, with nothing encoding ITS OWN nested
parameter list -- reproduced end to end: distinct: 1 of 2 declarations.
Confirmed by direct compilation that clang shapes a
TemplateTemplateParmDecl's own inner exactly like a top-level
parameter list (its own TemplateTypeParmDecl/NonTypeTemplateParmDecl/
nested TemplateTemplateParmDecl children), so the fix is a genuine
recursion rather than a special case: function_template_param_kinds now
delegates to a shared _template_param_kinds_from_node helper that runs
identically over the top-level FunctionTemplateDecl and any nested
TemplateTemplateParmDecl, each level getting its own independent
type_param_names scope (a template-template parameter's own type
parameters live at a different depth than the enclosing list's). A
template-template parameter's entry now reads e.g.
"template(type,type)" for two nested type parameters. Regression test
in tests/test_entity_id_carrier.py
(test_live_clang_template_template_param_nested_arity_discriminates).
While landing that fix, also found and fixed an unrelated, pre-existing
test bug in the SAME file: _clang_parser's subprocess invocation named
no --target, so it silently compiled for whichever platform the test
happened to run on -- on a macOS CI runner, clang's own default target
(Darwin/Mach-O) bakes an extra leading underscore directly into
mangledName (__ZN2ns4gVarE rather than the Linux-target
_ZN2ns4gVarE), which every assertion in this file was written against.
Confirmed reproducible via cross-compilation (--target=x86_64-apple-
darwin on this Linux sandbox reproduces the exact double-underscore
spelling a real macOS CI run hit). Fixed by pinning
--target=x86_64-unknown-linux-gnu on the one clang invocation this
file's helpers share -- this module tests entity-identity logic, not
host-linker-convention accidents, so every live-clang probe here now
compiles for one fixed target regardless of which OS runs the test.
Correction (2026-08-29, same day, Codex review on PR #943): the
dependent-rename canonicalization from two corrections ago missed a
sibling case -- a non-type parameter dependent on a preceding
template-TEMPLATE parameter, not just a type parameter.
template<template<class> class TT, TT<int>* N> void f(); renamed to
UU are the identical declaration, but a bare reference to a
template-template parameter is not itself legal C++ (confirmed by direct
compilation: it fails to parse) -- a real non-type parameter dependent on
one always names a concrete instantiation like TT<int>, and clang's own
qualType for N spells that instantiation using the template-template
parameter's own name literally ("TT<int> *"/"UU<int> *"). The
canonicalization loop tracked only TemplateTypeParmDecl names, so a
template-template parameter's own name was never added to the
substitution map -- reproduced end to end: the two renamed revisions
produced unequal EntityIds. Fixed by appending a TemplateTemplateParmDecl's
own name to the same substitution list its type-parameter siblings use,
so either kind at a given declaration position resolves to the identical
"type-param-N" token. Regression test in tests/test_entity_id_carrier.py
(test_live_clang_template_template_param_rename_does_not_change_identity).
Correction (2026-08-29, same day, CodeRabbit review on PR #943): an
unrelated, pre-existing primitive bug in qualified_name_segments.
_walk_rewrite_strings (the general closure-marker rewrite walk, not
this phase's own discriminator work) -- a changed init=False field on
a frozen dataclass was computed but then silently discarded. Only
init=True fields can be handed to dataclasses.replace, so a changed
init=False field's rewritten value was dropped on the floor -- most
visibly when it was the ONLY field on that dataclass to change (replace
is then never even called, since the replacements dict stays empty),
leaving that field holding stale, un-normalized :line:col content
indefinitely. Fixed by applying a changed init=False field via
object.__setattr__ -- the same escape hatch a frozen dataclass's own
__post_init__ uses to set a derived field, and an established
convention elsewhere in this codebase (compatibility_evaluation_
config.py) for the identical need -- applied AFTER replace rebuilds
the init=True fields, so a rewrite touching both kinds of field in one
dataclass lands on the object this function actually returns. Regression
test in tests/test_lambda_identity_ordinal.py
(test_a_changed_non_init_field_is_itself_rewritten), confirmed to fail
against the pre-fix walk (verified via git stash on just the source
file).
Correction (2026-08-29, same day, Codex review on PR #943): the
dependent-rename hazard also reached the ORDINARY parameter list, not
just non-type template parameters. template<class T> void f(T); and
template<class U> void f(U); are the identical declaration, but an
ordinary parameter's raw spelling names the template parameter literally
too (confirmed by direct compilation), and entity_id_for_function's
param_types were never canonicalized against the enclosing template's
own type parameter names -- reproduced end to end: the two renamed
revisions produced unequal EntityIds. Fixed by generalizing the
canonicalization helper (renamed canonicalize_type_param_references,
relocated from extract/headers/clang/functions.py to model/identity.py
since both a model-layer function and an extract-layer one now need it,
and ADR-061's import direction is extract -> model, never the reverse)
and adding a new type_param_names parameter to entity_id_for_function,
applied to each param_types entry before the existing cross-backend
signature canonicalization. The producer side threads the enclosing
function template's own TOP-LEVEL type/template-template parameter names
through a new function_template_type_param_names (sharing
_template_param_kinds_from_node's exact walk with
function_template_param_kinds, now returning (kinds,
type_param_names)) and a new _Decl.template_type_param_names field,
threaded through _walk/_categorize in dumper_clang.py identically to
the existing template_param_kinds field. Regression test in
tests/test_entity_id_carrier.py
(test_live_clang_template_param_rename_in_ordinary_param_does_not_change_identity).
A second finding from the same review round, in the castxml backend
rather than this discriminator: castxml's own C-linkage recovery for a
bogus pseudo-Itanium mangled name (the "case141" class of issue) checked
only exported_dynamic, so a C API observed exclusively through a static
archive's own export set (exported_static) left the bogus guess
standing -- confirmed by direct compilation (a plain, unmarked int
foo(int); gets mangled="_Z3fooi" from castxml's ambiguous-language-mode
default) and by inspecting dumper_castxml.py's own sibling
variable-level override, which already checks the
exported_dynamic | exported_static union. Fixed by extending
extract/headers/castxml/functions.py's function-level override to the
identical union. Regression test in tests/test_entity_id_carrier.py
(test_live_castxml_honors_static_export_evidence_for_c_linkage),
confirmed to fail against the pre-fix override via git stash on just
the source file.
Correction (2026-08-29, same day, Codex review on PR #943): two more sibling collisions in the SAME rename-canonicalization mechanism.
First: a non-type parameter's own NAME had never been added to the
substitution list, so a later non-type parameter referencing an earlier
one (decltype(N) for a preceding int N) was left uncanonicalized
-- confirmed by direct compilation and reproduced end to end: renaming
N to M changed the EntityId. Fixed by appending a non-type
parameter's own name to the shared substitution list too, once its own
spelling is canonicalized (so its name is visible to LATER parameters,
never to itself).
Second, and more fundamental: the sequential (one name, one re.sub
call, applied to the progressively-mutated string) substitution scheme
itself had a self-inflicted collision -- if an earlier name substitution
produces "type-param-0", and a LATER parameter happens to be named
literally type (a legal, unremarkable C++ identifier that just happens
to match this function's own generated marker's prefix), that later
substitution's \btype\b pattern matches the "type" INSIDE the
already-generated token, corrupting it into "type-param-1-param-0" --
so renaming an entirely unrelated, unused parameter changed the
EntityId too. Reproduced end to end
(template<class T, class type, T x> vs. the unused second parameter
renamed to U). Fixed by replacing the sequential per-name scheme with
ONE combined-alternation regex pass: Python's re.sub resumes scanning
immediately after each match in the ORIGINAL string, never inside what
it just substituted, so this class of collision cannot occur regardless
of what any parameter happens to be named. The helper (renamed
canonicalize_type_param_references, already relocated to
model/identity.py in the correction above) now builds one
name -> index dict and one compiled alternation pattern per call,
rather than looping re.sub once per name.
Regression tests for both in tests/test_entity_id_carrier.py
(test_live_clang_nontype_param_dependent_rename_does_not_change_identity,
test_live_clang_rename_of_param_named_type_does_not_corrupt_a_prior_marker),
both confirmed to fail against the pre-fix (sequential, non-type-name-
excluding) canonicalization via git stash on just the source files.
Correction (2026-08-29, same day, Codex review on PR #943): a
pre-existing bug in dumper_clang.py's own linkage-state propagation
(not new code from this phase, but only newly OBSERVABLE through the
entity_id carrier this phase adds) -- a nested extern "C++" block
inside extern "C" was misidentified as C linkage. Confirmed by direct
compilation that extern "C" { extern "C++" { void cppfun(); } } places
a language="C++" LinkageSpecDecl directly inside a language="C"
one, and clang genuinely mangles cppfun normally (_Z6cppfunv) -- real
C++ linkage nested inside a C block, a legal and real shape. _walk's
child_extern_c computation was extern_c or (kind == "LinkageSpecDecl"
and node.get("language") == "C") -- a sticky OR that never resets back
to False for the inner block, so cppfun got is_extern_c=True and
collapsed onto the bare ("extern_c",) EntityId, colliding with every
other C-linkage declaration. Verified this line is unchanged from
origin/main (a genuinely pre-existing defect, not something this PR's
own diff introduced), but fixed it here anyway since it directly affects
identity correctness -- the exact class of thing this phase is about --
and the fix is small and self-contained. Fixed by having a
LinkageSpecDecl RESET the linkage state to its own declared language
rather than only ever OR-ing a True in -- linkage specs don't stack,
the innermost one wins, so a non-LinkageSpecDecl node simply inherits
whatever is already in effect. Verified plain extern "C" and plain
top-level C++ (no enclosing linkage spec at all) both still resolve
correctly. Regression test in tests/test_entity_id_carrier.py
(test_live_clang_nested_cpp_linkage_inside_extern_c_is_not_extern_c),
confirmed to fail against the pre-fix sticky-OR propagation via git
stash on just the source file.
Correction (2026-08-29, same day, Codex review on PR #943): a hidden
friend function/function-template's EntityId was resolved in the
befriending class's scope, not the enclosing namespace it is actually
injected into. Per [namespace.memdef], a hidden friend (one with no
prior declaration reachable by ordinary lookup) is a member of the
nearest enclosing namespace, not of the class it is lexically written
inside. Confirmed by direct compilation: clang rejects struct A {
template<class T> friend void f(T) {} }; struct B { template<class T>
friend void f(T) {} }; as a redefinition of f -- proof the two
hidden friend templates are the identical entity in the global namespace,
not two distinct class-scoped ones. The clang backend's own scope-building
walk is lexical, though: it pushes a Record segment for the enclosing
class regardless of whether a nested declaration is a genuine member or a
hidden friend, so each declaration's scope_path (and therefore its sig
fallback EntityId, since neither gets a real mangled name as an
uninstantiated template) wrongly still named its own befriending class
(Record("A") vs. Record("B")) -- an identity collision in the opposite
direction from every other fix in this phase: two genuinely identical
declarations wrongly compared unequal, rather than two distinct ones
wrongly compared equal. castxml does not have this bug: its context
attribute for a hidden friend already points at the enclosing namespace
element directly (confirmed with real castxml output), with the
befriending class recorded only via a separate befriending attribute --
exactly the split this fix brings the clang backend to. Fixed by adding
strip_record_scopes() to extract/headers/scope_segments.py (drops
every Record segment and every record-kind Anonymous segment -- ALL of
them, not just the innermost, since a friend hidden inside nested classes
is injected past every enclosing class -- keeping Namespace/
InlineNamespace/an anonymous-namespace Anonymous/LocalToFunction
unchanged) and applying it only to the scope argument
entity_id_for_function receives for a hidden friend in
extract/headers/clang/functions.py; hidden_friend_owner and every
other use of entry.scope_path (display qualified name included) are
untouched. Regression tests: tests/test_scope_segments.py's
TestStripRecordScopes (the primitive, in isolation) and
tests/test_entity_id_carrier.py's
test_live_clang_hidden_friend_template_resolves_in_namespace_scope (the
live-clang end-to-end case, confirmed to fail pre-fix via git stash on
just the source files).
Correction (2026-08-29, same day, Codex review on PR #943): the
unmangled function-TEMPLATE identity fallback ignored a dependent return
type, so two templates differing only in it collapsed onto one
EntityId. A function template's return type can itself depend on its
own template parameter (template<class T> typename T::x f(T);) --
confirmed by direct compilation that clang accepts both that declaration
and its typename T::y sibling with no redefinition error, two real
FunctionTemplateDecls. entity_id_for_function's sig fallback already
folded ordinary parameters, const/volatile/ref-qualifier/variadic, and
template-parameter kinds into extra, but never the return type, so the
two templates shared every dimension and collided. Fixed by adding a
return_type parameter, folded into extra as ("ret", <canonicalized
spelling>) -- placed BEFORE the existing "tmpl" block so it doesn't
shift the fixed extra[-1]/extra[-2] tail position every existing
template_param_kinds consumer already reads -- and ONLY when
template_param_kinds is non-empty: an ORDINARY function can never
legally overload solely by return type, so this is a no-op there,
identical to why finding_identity.normalized_signature never folds
return type in at all. Canonicalized identically to a dependent ordinary
parameter type (canonicalize_function_signature_param_type then
canonicalize_type_param_references), so a pure template-parameter
rename reflected only in the return type still resolves to the same id.
Regression test in tests/test_entity_id_carrier.py
(test_live_clang_dependent_return_type_discriminates_overloaded_templates),
confirmed to fail pre-fix via git stash on just the source files.
Correction (2026-08-29, same day, CodeRabbit review on PR #943): the
frozen-dataclass rebuild qualified_name_segments._walk_rewrite_strings
added for an init=False field's own rewrite mutated the caller's
ORIGINAL object when no init=True field also changed. value =
_dataclasses.replace(value, **replacements) ran only if replacements,
so when only frozen_field_updates was non-empty (the exact motivating
case this rebuild exists for -- a changed init=True field with no
sibling init=False change is the common path, but the reverse is what
this mechanism was added to handle), value still named the caller's own
original frozen instance, and the following object.__setattr__ loop
mutated it in place instead of a fresh copy -- a frozen dataclass silently
becoming not-actually-immutable to its own caller. Fixed by rebuilding
whenever EITHER dict is non-empty (if replacements or
frozen_field_updates); dataclasses.replace(value) with no overrides
still constructs a genuinely new instance. Regression test in
tests/test_lambda_identity_ordinal.py's
test_a_changed_non_init_field_is_itself_rewritten, extended to assert
result is not original and that original's own field is unchanged
after the rewrite, confirmed to fail pre-fix via git stash on just the
source file.
Correction (2026-08-29, same day, Codex review on PR #943): the
return-type canonicalization the previous correction added dropped a
top-level by-value cv-qualifier that is a real overload discriminator for
a function TEMPLATE. entity_id_for_function's new return_type
handling reused canonicalize_function_signature_param_type, which
deliberately drops a top-level by-value cv-qualifier because that
qualifier is dropped from an ORDINARY function's own type (confirmed:
int f(int); const int f(int); is a redefinition error). But confirmed
by direct compilation that template<class T> T f(T); and template
<class T> const T f(T); are two more real, legally-coexisting
FunctionTemplateDecls (T (T) vs. const T (T)) -- the opposite rule
for a template's return type, so the reused function collapsed both onto
type-param-0. Fixed by canonicalizing the return type through
canonicalize_type_name instead (cross-producer spelling normalization
only, keeps every cv-qualifier) before the same
canonicalize_type_param_references rename-blind substitution.
Regression test in tests/test_entity_id_carrier.py
(test_live_clang_return_type_top_level_cv_discriminates_overloaded_templates),
confirmed to fail pre-fix via git stash on just the source file; the
prior dependent-return-type test
(test_live_clang_dependent_return_type_discriminates_overloaded_templates)
and its rename-invariance case both still pass unchanged.
Correction (2026-08-29, same day, Codex review on PR #943): an ordinary
member function nested inside a class template still changed identity on
a pure rename of the ENCLOSING class template's own parameter. Every
prior correction in this phase threaded a directly-templated FUNCTION's
own type-parameter names into entity_id_for_function, but _walk
computed child_template_type_param_names as own-names-if-
FunctionTemplateDecl-else-() -- resetting to empty for every other node
kind, including a ClassTemplateDecl. So template<class T> struct A {
void f(T); }; renamed to template<class U> struct A { void f(U); };
-- the identical declaration -- fingerprinted as a remove+add: f is an
ordinary (non-template) CXXMethodDecl, never itself a
FunctionTemplateDecl, so it never received the enclosing class
template's own parameter names to canonicalize its dependent ordinary
parameter type against. Confirmed by direct compilation.
ClassTemplatePartialSpecializationDecl carries the identical shape
(its own parameter list, then its own member pattern) and needs the same
treatment -- also confirmed by direct compilation. Fixed by making
child_template_type_param_names an ACCUMULATING value (like
in_template), not a per-node reset: a ClassTemplateDecl/
ClassTemplatePartialSpecializationDecl node's own names (via a new
extract.headers.clang.functions.class_template_type_param_names,
sharing function_template_type_param_names's exact node-shape walk) are
now APPENDED to whatever was already accumulated, and every other node
kind simply inherits the incoming value unchanged instead of discarding
it. A member function template nested inside a class template sees BOTH
levels' names (verified: template<class T> struct B { template<class U>
void g(T, U); }; renamed at both levels still resolves to one id).
Regression test in the new tests/test_entity_id_template_discriminators.py
(split out of tests/test_entity_id_carrier.py, which had grown past the
architecture gate's 1200-line test-file cap, purely to keep both files
under it -- no test content or behavior changed by the split itself)
(test_live_clang_enclosing_class_template_param_rename_does_not_change_identity),
confirmed to fail pre-fix via git stash on just the source files.
Known gap raised (2026-08-29, same day, Codex review on PR #943, NOT
fixed): an out-of-line member (function or static data member) template
definition gets a different EntityId scope than its in-class
declaration, colliding one entity into two. Confirmed by direct
compilation that clang emits two FunctionTemplateDecl nodes for
struct A { template<class T> void f(T); }; template<class T> void
A::f(T) {} -- one lexically nested inside A, one at the enclosing
namespace's own lexical level carrying a parentDeclContextId pointing
back at A's node id, clang's own signal for the out-of-line
definition's real semantic owner. _walk computes scope/scope_path
purely from lexical nesting, with no parentDeclContextId handling
anywhere in this codebase, so the two nodes get scope=() and
scope=(Record("A"),) respectively for what is one entity;
parse_variables has the analogous gap for an out-of-line class-template
static data member. Unlike every other correction in this phase (each a
small, local addition to one already-threaded parameter), this needs a
NEW general-purpose facility: a typed, ScopePath-valued sibling of the
existing dumper_clang_expr._index_decl_id_qualified_names (which
already indexes every decl id to a flat qualified-name STRING for a
different consumer -- a flat string cannot be losslessly converted back
into a typed ScopePath, exactly the ambiguity ScopePath exists to
prevent), threaded through both parse_functions and parse_variables,
with further edge cases (an out-of-line member of a nested class, of a
class-template specialization, and whether castxml has the identical
gap) needing their own verification before landing. Recorded as a known
gap in docs/contribute/known-gaps.md rather than attempted as a
narrow, risky patch under this PR's own scope; the corresponding review
thread was left open by design, the same as the requires-clause gap
above.
Correction (2026-08-29, same day, real Windows CI failure on PR #943):
the castxml C-linkage export-evidence override never fired on the
Windows CI leg, since it was gated on Itanium's own "_Z" mangling
prefix. Both extract.headers.castxml.functions.parse_function_element
and dumper_castxml.parse_variables's override (recovering extern "C"
identity when castxml's own guess is ambiguous, matching a confirmed
bare-name export) checked mangled.startswith("_Z") before considering
the export evidence. A real Windows CI run failed two tests
(test_live_castxml_populates_every_kind, test_live_castxml_honors_
static_export_evidence_for_c_linkage) because the Windows-targeting
castxml install decorates a guessed C-linkage function/variable with its
own MSVC "?...@@..." prefix instead -- extern "C" int c_var; got
mangled="?c_var@@3HA", and a plain (no-extern) int foo(int x); got
"?foo@@YAHH@Z" -- so the "_Z" check silently never matched on that
platform, leaving the bogus MSVC-decorated symbol standing even though
exported_dynamic/exported_static already confirmed the bare name.
Nothing else in the override's condition is ABI-specific: mangled not
in (exported_dynamic|exported_static) already means "not itself a real
observed export" regardless of what guessed prefix produced it. Fixed by
dropping the "_Z"-prefix gate from both overrides entirely, making
them recognize the identical evidence on every mangling scheme
castxml's underlying compiler can guess. Regression test in
tests/test_entity_id_carrier.py
(test_live_castxml_export_override_recognizes_non_itanium_mangling_prefixes)
-- since this sandbox has no MSVC-targeting castxml to reproduce the
real failure directly, it runs a real (Linux) castxml dump and then
rewrites its own mangled attributes to the exact MSVC-decorated
strings the Windows CI log showed, confirmed to fail pre-fix via git
stash on just the source files. Both originally-failing tests
(unmodified) were re-verified to still pass on Linux.
Correction (2026-08-29, same day, Codex review on PR #943): the
rename-blind substitution treated an EXPLICITLY globally-qualified name
as a reference to a template parameter merely because it collided in
spelling. canonicalize_type_param_references's whole-word
substitution had no way to distinguish a genuine dependent reference
from a name that only happens to share a spelling with the template
parameter but is actually disambiguated to the global namespace by a
leading ::. Confirmed by direct compilation: namespace T { struct X
{}; } template<class T> void f(::T::X); keeps ::T::X verbatim in
clang's own qualType (it does not resolve to the parameter), yet the
substitution rewrote it to ::type-param-0::X regardless -- so renaming
the (here, unused) parameter to U left the second revision's ::T::X
untouched (since "U" doesn't match), producing unequal EntityIds for
the identical declaration. Fixed by adding a (?<!::) negative
lookbehind to the substitution pattern -- a bare :: prefix (global
scope, no preceding name) is the only legal spelling a name can appear
after in this position, so no alternation is needed. Regression test in
tests/test_entity_id_template_discriminators.py
(test_live_clang_globally_qualified_name_is_not_canonicalized_as_a_param_ref),
confirmed to fail pre-fix via git stash on just the source file; every
existing rename-invariance test in that module (ordinary dependent
references, none globally-qualified) still passes unchanged.
Correction (2026-08-29, same day, Codex review on PR #943): the same
whole-word substitution also mistook a MEMBER-ACCESS expression's member
name for a template-parameter reference merely because it collided in
spelling. struct S { int N; }; template<int N> void
f(decltype(S{}.N)); (and the pointer form ((S*)0)->N) keeps the
member name N verbatim in clang's own qualType regardless of what
the (here, unused) non-type template parameter is actually named --
confirmed by direct compilation for both . and -> forms -- yet the
substitution rewrote it to type-param-0 anyway, so renaming the
parameter to M left the second revision's S{}.N/((S*)0)->N
untouched, producing unequal EntityIds for the identical declaration.
Same collision shape as the globally-qualified-name correction directly
above, just for ./-> instead of ::. Fixed by extending the
substitution pattern with (?<!\.)(?<!->) negative lookbehinds
alongside the existing (?<!::) one. Regression test in
tests/test_entity_id_template_discriminators.py
(test_live_clang_member_access_name_is_not_canonicalized_as_a_param_ref,
parametrized over both the . and -> forms), confirmed to fail
pre-fix via git stash on just the source file for both parametrized
cases; every existing test in that module still passes unchanged.
abicheck/model/identity.py sits exactly at the AI-readiness
production-file soft cap (800 lines) after this fix -- the two related
lookbehind comments were condensed to fit rather than granted a debt-
ledger entry, since both fixes are genuinely small and the file was
already at 797 lines before this correction.
Correction (2026-08-29, same day, Codex review on PR #943): a function
template's TRAILING return type was discarded entirely, collapsing two
legal, coexisting overloads onto one identity. _return_type
(extract.headers.clang.functions) scanned a function qualType for
the first top-level ( (the start of the parameter list) and returned
everything before it -- correct for an ordinary leading return type, but
clang spells a trailing-return-type function's leading part as the bare
placeholder "auto", confirmed by direct compilation: template<class
T> auto f(T) -> typename T::x; and the sibling overload returning
typename T::y both carry the literal qualType "auto (T) -> typename
T::x"/"auto (T) -> typename T::y". So both the model's own
Function.return_type field and (for an uninstantiated template with no
mangled name) the EntityId's "sig" fallback read "auto" for both
overloads, discarding the one thing that actually distinguishes them.
Fixed by resuming the same bracket-depth scan past the parameter list's
matching ) and, when a top-level -> follows (never inside
<...>/[...], so a nested std::function<T(int)->int>-shaped alias
cannot be mistaken for one), returning the spelling after it instead of
the leading placeholder. Regression test in
tests/test_entity_id_template_discriminators.py
(test_live_clang_trailing_return_type_discriminates_overloaded_templates),
confirmed to fail pre-fix via git stash on just the source file;
non-template, non-trailing, and function-pointer-return-type cases were
independently checked to keep their existing spellings (int g(int)
still reads "int", auto f(int) -> int* still reports
return_pointer_depth=1).
Correction (2026-08-29, same day, Codex review on PR #943): a nested
template-template parameter's own non-type parameter could reference an
ENCLOSING parameter's name, but the recursive descent started that
nested scope's substitution empty. template<class T, template<T>
class TT> void f(); is valid C++ -- confirmed by direct compilation:
clang parses it without error, and the nested, unnamed
NonTypeTemplateParmDecl inside TT's own parameter list carries
qualType "T", the literal enclosing type parameter's name. But
_template_param_kinds_from_node's recursive call for a
TemplateTemplateParmDecl (extract.headers.clang.functions) passed no
enclosing names into the nested walk, so
canonicalize_type_param_references had nothing to substitute "T"
against inside that nested scope -- renaming the enclosing parameter to
U produced "template(nontype:T)" vs. "template(nontype:U)" for the
otherwise-identical declaration, the same collision shape the earlier
enclosing-class-template-parameter correction fixed for an ordinary
member, just one level further inward. Fixed by threading an
enclosing_type_param_names parameter through the recursive descent,
seeded once ahead of the nested list's own accumulated names so a nested
parameter never shadows an enclosing one's substitution position.
Regression test in tests/test_entity_id_template_discriminators.py
(test_live_clang_nested_template_template_param_sees_enclosing_names),
confirmed to fail pre-fix via git stash on just the source file; every
existing template-template-parameter discriminator test in that module
(kind/packness/nested-arity collisions, the TT/UU rename case) still
passes unchanged.
Correction (2026-08-29, same day, Codex review on PR #943): two
templates differing only in the DECLARATOR SHAPE of a dependent return
type still collapsed onto one identity, after the trailing-return-type
fix above. template<class T> typename T::x f(T); and template<class
T> typename T::x (*f(T))(T); both compile with no redefinition error
(confirmed by direct compilation) -- the second returns a pointer to a
function, and clang spells this as a SPIRAL declarator, typename T::x
(*(T))(T): the first top-level parenthesized group ((*(T))) is not
itself the function's parameter list -- it wraps a pointer declarator
around the real one, (T), nested one level deeper; the trailing (T)
belongs to the RETURNED function type, not to the outer function. The
fixed _return_type (from the trailing-return-type correction above)
still treated the first top-level group as the parameter list
unconditionally, so both overloads' return type -- and, for an
uninstantiated template, EntityId -- collapsed onto the identical
"typename T::x". Fixed by detecting when a function's qualType has
MORE than one top-level parenthesized group (the ordinary,
single-group case is left byte-for-byte unchanged): in that case the
real parameter list is located by recursing into the first group's own
interior until a nesting level is reached with no further top-level
group following it (the recursion bottoms out there, generalizing to
arbitrarily deep pointer/reference-to-function nesting, not just one
level), and its contents are excised (parens kept, as an empty marker)
so the remaining text -- including the */& and any further wrapping
groups -- becomes the return-type discriminator. Verified the pointer
and reference forms stay distinct from each other and from the ordinary
case: typename T::x (*f(T))(T) -> "typename T::x (*())(T)",
typename T::x (&f(T))(T) -> "typename T::x (&())(T)", typename T::x
f(T) -> "typename T::x" (unchanged). Regression test in
tests/test_entity_id_template_discriminators.py
(test_live_clang_function_pointer_return_declarator_discriminates_overloaded_templates),
confirmed to fail pre-fix via git stash on just the source file; every
existing return-type/trailing-return-type discriminator test in that
module still passes unchanged, and no other clang-backend test in the
unit suite regressed.
Correction (2026-08-29, same day, Codex review on PR #943): the
spiral-declarator fix directly above was itself too general -- assuming
EVERY function qualType with more than one top-level parenthesized
group was a function-pointer/reference-returning spiral broke two other
real, confirmed cases. (1) A dependent return type containing its OWN
parenthesized sub-expression, decltype((T::x)): template<class T>
decltype((T::x)) f(T); and the T::y sibling both compile with no
redefinition error, but the fix's first-group recursion treated
((T::x)) as a spiral wrapper and excised its interior, discarding the
operand entirely and collapsing both overloads onto one EntityId. (2)
An ordinary function's exception specification, noexcept(expr): int
f() noexcept(cond());'s qualType is genuinely
"int () noexcept(cond())" -- a real two-top-level-group spelling -- and
the fix appended the whole noexcept(cond()) group onto return_type
as if it were return-type text, which would fabricate a spurious
return-type-changed finding whenever only the exception-specification
condition changes. Both confirmed by direct compilation. Root cause: the
FIRST top-level group is not reliably identifiable as "the parameter
list" or "not the parameter list" from local shape alone (a leading */
& sigil test was considered and rejected -- it would have fixed the
noexcept case but not the decltype one, since neither of that case's
groups starts with a sigil). Replaced with a simpler, more general rule:
the function's own real top-level parameter list is the LAST top-level
parenthesized group that is not itself an exception-specification group
(checked via a noexcept/throw keyword immediately preceding it) --
everything before that group, verbatim, is the return type; the earlier
recursive _excise_own_param_list helper is no longer needed at all and
was deleted. Verified this single rule now resolves all four cases
together: ordinary (int (int) -> "int"), trailing-return-type
(auto (T) -> typename T::x -> "typename T::x"), spiral pointer/
reference (typename T::x (*(T))(T) -> "typename T::x (*(T))", the
(&... form distinctly "typename T::x (&(T))"), dependent-parens
(decltype((T::x)) (T) -> "decltype((T::x))"), and noexcept
(int () noexcept(cond()) -> "int"). Regression tests added in
tests/test_entity_id_template_discriminators.py
(test_live_clang_dependent_return_type_own_parens_discriminates_overloaded_templates,
test_live_clang_noexcept_expression_group_is_not_mistaken_for_return_type),
both confirmed to fail against the prior commit's code via git stash
and pass post-fix; every existing return-type discriminator test in that
module (including the spiral-declarator one directly above, whose exact
expected string this correction also updates -- "typename T::x (*(T))"
rather than the previous fix's excised "typename T::x (*())(T)", since
the new rule no longer excises the nested parameter list at all) still
passes.
Correction (2026-08-29, same day, real Windows CI failures on
PR #943): two more real Windows-only failures in the EntityId carrier
test suite itself, this time in the test infrastructure rather than the
production code. (1) TestResolverIsOnlyCalledByAProducer/
TestMangledRewritesKeepTheCarrierInSync (both new AST-scanning tests
this phase's own second slice added) read every abicheck/**/*.py file
via Path.read_text() with no explicit encoding -- str.decode/
open()'s platform default on Windows is the host codepage, not UTF-8,
so this raised UnicodeDecodeError on the first source file containing
a byte sequence the codepage can't decode; a real Windows CI run failed
both scanner test classes this way. Fixed by passing
encoding="utf-8" explicitly at both call sites. (2)
test_live_castxml_populates_every_kind failed for the identical
root-cause class the earlier castxml-mangling-prefix correction fixed in
production code, but here in the test's OWN castxml invocation:
_castxml_parser (unlike its _clang_parser sibling, which already
pins --target=x86_64-unknown-linux-gnu for exactly this reason) passed
no target at all, so on Windows CI the underlying castxml install
targeted the host by default and mangled the probe header's gVar as
MSVC's "?gVar@ns@@3HA" instead of the Itanium "_ZN2ns4gVarE" the
test's assertion is hardcoded against -- confirmed by a real Windows CI
failure log. Fixed by pinning the identical --target=... flag on the
castxml invocation too (castxml forwards an unrecognized flag straight
to its internal clang compiler, so it is the same flag reached a
different way). Both fixes verified by re-running the affected tests
locally (Linux, where the bug was invisible either way -- UTF-8 is
already the default codepage there, and this sandbox's castxml already
targets Itanium by default) and by confirming this repository's own
mypy/ruff/architecture gates and the full clang/castxml-marked test
subset stay green.
Correction (2026-08-29, same day, Codex and CodeRabbit independently
on PR #943): two more real return-type collisions the "scan from the
end" fix above still missed. (1) Codex: a TRAILING return type that
itself contains parentheses, auto f(T) -> decltype((T::x)) and the
T::y sibling, both compile with no redefinition error, but locating
"the" parameter-list group BEFORE ever checking for a top-level ->
(the previous fix's own order of operations) picked the decltype's own
((T::x)) group -- the LAST top-level group, and not preceded by
noexcept/throw -- as if it were the parameter list, reducing both
overloads to the identical "auto (T) -> decltype" and discarding the
dependent operand entirely. (2) CodeRabbit: a function-pointer/reference
return type's OWN parameter list is real, distinguishing content, but
"scan from the end, pick the last non-exception-spec group" picked the
RETURNED function's parameter list ((int)/(double) in typename S::x
(*f(T))(int) vs. ...(double)) as if it were f's own, discarding it
entirely and reducing both overloads to the identical "typename S::x
(*(T))" -- the exact distinguishing content the ORIGINAL (pre-"scan
from the end") recursive-excision design preserved, before that design
was replaced to fix the noexcept/decltype cases. Both confirmed by
direct compilation. Root cause, in both cases: "scan from the end" is
not a single correct rule for every shape -- a trailing return type must
never be group-parsed at all (its own parentheses are never a parameter
list), and a spiral function-pointer/reference return type's REAL
parameter list is always nested ONE LEVEL INSIDE the first group, not
the last group at the top level. Fixed by re-introducing a three-step
resolution, each step checked in order and never falling through once
one matches: (1) a top-level -> (checked FIRST, before any group
scan) takes everything after it verbatim, so a trailing return type's
own parentheses are never mistaken for a parameter list; (2) a spiral
declarator (detected by the first top-level group's own interior
starting with a */& sigil) uses the ORIGINAL recursive
_excise_own_param_list design, which correctly preserves the returned
function type's own parameter list while excising only the nested,
duplicate copy of f's own; (3) otherwise, scan from the end for the
last non-exception-spec group, exactly as the previous fix already
established for the ordinary/dependent-parens/noexcept cases. Verified
all eight cases together (ordinary, trailing-return, spiral pointer,
spiral reference, dependent-parens, noexcept, spiral-with-differing-
returned-parameters, and trailing-return-containing-parens). Regression
tests added in tests/test_entity_id_template_discriminators.py
(test_live_clang_trailing_return_type_containing_parens_discriminates_overloaded_templates,
test_live_clang_spiral_declarator_preserves_returned_function_parameter_list),
both confirmed to fail against the prior commit's code via git stash
and pass post-fix; the spiral-declarator test from the earlier round had
its own expected string updated back to the excised form
("typename T::x (*())(T)") since excision is back. This correction
also split _return_type (now return_type) and its three private
helpers out of extract/headers/clang/functions.py into a new sibling
leaf module, extract/headers/clang/return_type.py -- the accumulated
docstrings pushed functions.py past the AI-readiness gate's 800-line
production soft cap, and this primitive was already self-contained with
exactly one external call site, matching this package's own established
"prefer extending a split-out module over growing the parent toward the
cap" convention.
Correction (2026-08-29, same day, Codex review on PR #943): a QUOTED
LITERAL sharing a template parameter's spelling was rewritten by the
same rename-blind substitution as if it were a reference to that
parameter. template<char C> struct Literal {}; template<int N> void
f(Literal<'N'>); keeps the char literal 'N' verbatim in clang's own
qualType -- confirmed by direct compilation -- it does not resolve to
the (here, unused) non-type parameter, yet
canonicalize_type_param_references's whole-word substitution rewrote
it to Literal<'type-param-0'> anyway, so renaming the parameter to M
left the second revision's Literal<'N'> untouched, producing unequal
EntityIds for the identical declaration -- the identical collision
shape the globally-qualified-name and member-access corrections earlier
in this file fixed, just for a quoted literal instead of ::/./->.
Fixed by adding quoted_literal_spans (a new leaf module,
model/identity_literals.py -- identity.py was already at the
800-line cap after the return-type corrections above, so this stayed a
sibling module rather than inline) and skipping any substitution match
whose start falls inside a single- or double-quoted literal span
(backslash-escapes honored). Regression test in
tests/test_entity_id_template_discriminators.py
(test_live_clang_quoted_literal_is_not_canonicalized_as_a_param_ref),
confirmed to fail pre-fix via git stash on just the source file; every
existing rename-invariance test in that module still passes unchanged.
Correction (2026-08-29, same day, Codex review on PR #943): a MEMBER
FUNCTION TEMPLATE's own non-type parameter could reference an ENCLOSING
CLASS template's parameter, one level further out than the earlier
nested-template-template correction, and function_template_param_kinds
had no seeding for that scope at all. template<class T> struct A {
template<T N> void f(); }; is valid C++ -- confirmed by direct
compilation -- and clang's qualType for the member template's own
non-type parameter N spells its type literally as the enclosing class
template's own parameter name, "T". But function_template_param_kinds
took no enclosing_type_param_names parameter at all (unlike
class_template_type_param_names, which already exists to fix the
identical hazard for an ORDINARY, non-template member), so renaming the
enclosing parameter to U produced "nontype:T" vs. "nontype:U" for
the otherwise-identical declaration. Fixed by adding an
enclosing_type_param_names parameter (mirroring the nested-template-
template-parameter fix already threaded through
_template_param_kinds_from_node's own recursion) and passing
dumper_clang.py's already-accumulated template_type_param_names
(the enclosing scope, available at the exact point a FunctionTemplateDecl
node is visited, before its own names are folded in) into the call.
Regression test in tests/test_entity_id_template_discriminators.py
(test_live_clang_enclosing_class_template_param_rename_does_not_change_member_template_identity),
confirmed to fail pre-fix via git stash on just the source files;
every existing template-parameter discriminator test in that module
still passes unchanged.
Correction (2026-08-29, same day, Codex review on PR #943): a
POINTER-TO-MEMBER-FUNCTION return type is another spiral declarator, but
its wrapper prefix is a qualified C::*, not a bare */&, and the
spiral-detection sigil check from two corrections ago only recognized
the latter. template<class T> int (C::*f(T))(int); and the sibling
returning a pointer to a member function taking double instead both
compile with no redefinition error -- confirmed by direct compilation
that clang spells this as int (C::*(T))(int), whose first group's
interior, C::*(T), does not start with a bare */&. The
leading-sigil check (first_interior[:1] in ("*", "&")) missed this
shape entirely, falling through to the scan-from-the-end branch and
discarding the returned member function's own parameter list -- the
identical hazard the ordinary pointer/reference spiral fix already
closed, just for a class-qualified sigil. Fixed by replacing the
single-character check with _is_spiral_wrapper_prefix: it locates the
first group's own nested top-level group (if any) and checks whether
the text BEFORE it is exactly */&/&&, or ends with ::* (any
qualified, possibly-templated class name followed by the pointer-to-
member sigil) -- both confirmed correct against the full existing case
set (ordinary, spiral pointer/reference, dependent-parens, noexcept,
trailing-return, member-pointer). Regression test in
tests/test_entity_id_template_discriminators.py
(test_live_clang_member_pointer_spiral_return_declarator_preserves_returned_function_parameter_list),
confirmed to fail pre-fix via git stash on just the source file; every
existing return-type discriminator test in that module still passes
unchanged.
Correction (2026-08-29, same day, Codex review on PR #943): a
function-pointer-returning function's own OUTER exception specification
leaked into return_type through the SPIRAL branch's trailing group.
template<class T> int (*f(T))(int) noexcept(noexcept(T()));'s
qualType is "int (*(T))(int) noexcept(noexcept(T()))" -- confirmed
by direct compilation -- and the spiral branch appends
qualtype[first_end:] (everything after the wrapper group) verbatim,
which here includes the trailing noexcept(noexcept(T())) text on top
of the returned function's own real (int) parameter list. Left
unstripped this pollutes return_type with exception-specification
text, which would fabricate a spurious return-type-changed finding
whenever only the exception-specification condition changes, not the
actual return type -- the identical hazard the ordinary, non-spiral
noexcept correction closed, here for the spiral branch's own trailing
group instead of its parameter-list-selection logic. Fixed by adding
_strip_trailing_exception_spec, applied to the spiral branch's tail
before appending it: it finds the last top-level group in the tail and,
if that group is immediately preceded by noexcept/throw AND sits at
the very end of the string, truncates the tail to remove it (falling
back to a plain regex strip for a bare, parenthesis-free noexcept).
Regression test in tests/test_entity_id_template_discriminators.py
(test_live_clang_spiral_return_strips_outer_exception_spec), confirmed
to fail pre-fix via git stash on just the source file; every existing
return-type discriminator test in that module still passes unchanged.
Correction (2026-08-29, same day, Codex review on PR #943): the
previous correction's own premise was backwards, and it left a distinct,
still-real leak unfixed. Direct compilation (static_assert(!noexcept(
f(0))) plus static_assert(noexcept((*(decltype(f(0)))(0)))) for
template<class T> int (*f(T))(int) noexcept(noexcept(T()));) proves the
trailing noexcept(noexcept(T())) in that qualType belongs to the
RETURNED function pointer type, not to f itself -- the previous
correction's _strip_trailing_exception_spec removed it anyway, silently
collapsing two overloads differing only in that condition onto the same
reported return type (the identical class of hazard the arrow-first and
scan-from-end corrections above exist to prevent, just reintroduced here
by this fix). Separately, fresh review on this same head found the
genuinely outer case Codex originally raised had never actually been
fixed: for template<class T> int (*g(T) noexcept(noexcept(T())))(int);
(qualType "int (*(T) noexcept(noexcept(T())))(int)", confirmed by
direct compilation that g(0) itself IS noexcept there), the exception
specification sits INSIDE the wrapper's own first top-level group --
_excise_own_param_list's len(spans) == 1 base case -- not in the
tail _strip_trailing_exception_spec was applied to; a complex condition
is itself parenthesized, so it produces a SECOND top-level span in that
same interior exactly like a genuine further-nested spiral level does
(compare int (*(*h(T))(T))(T)'s first_interior, *(*(T))(T), whose
second span (T) is real, kept return-type content) -- a span-count-only
rule cannot tell the two apart, so g's complex own condition took the
"further wrapper, recurse" branch and both the spec AND g's own
parameter list leaked through verbatim. Fixed by removing
_strip_trailing_exception_spec entirely (the spiral branch's tail is
now kept exactly as qualtype[first_end:], unmodified -- confirmed real
return-type content, never something to strip) and instead
discriminating inside _excise_own_param_list on what immediately
follows the wrapper's own nested parameter list: a new
_LEADING_EXCEPTION_SPEC_RE match (a leading noexcept/throw keyword,
regardless of how many parenthesized groups its own condition
introduces) discards that remainder outright as the CURRENT level's own
spec, while anything else that starts with ( is real further-nested
return-type content, recursed into as before. Two regression tests in
tests/test_entity_id_template_discriminators.py
(test_live_clang_spiral_trailing_exception_spec_belongs_to_returned_function,
replacing the now-incorrect test_live_clang_spiral_return_strips_outer_
exception_spec, and test_live_clang_spiral_return_own_exception_spec_
excised), each confirmed to fail pre-fix via git stash on just the
source file; every existing return-type discriminator test in that
module (including the double-spiral and member-pointer-spiral cases
above) still passes unchanged. mypy abicheck/ clean (515 files),
ruff check/ruff format --check clean on both touched files,
check_architecture.py 0 errors, check_ai_readiness.py 0 new errors.
Correction (2026-08-29, same day, Codex review on PR #943): a function
returning a Clang Blocks-extension block pointer collapsed the same way
the pointer/reference and pointer-to-member spiral cases above once did,
for a sigil _is_spiral_wrapper_prefix never covered. Confirmed by
direct compilation (clang -fblocks -x c++ -Xclang -ast-dump=json):
int (^f(int))(int); is spelled "int (^(int))(int)", structurally
identical to the pointer-declarator spiral case but with ^ instead of
* -- _is_spiral_wrapper_prefix only recognized */&/&& and a
qualified <class>::* prefix, so this fell through to the scan-from-end
branch and discarded the returned block's own parameter list, exactly
the hazard the member-pointer-spiral fix above already closed for a
different sigil: int (^f(int))(int); and int (^g(int))(double); both
reduced to the identical "int (^(int))" before this fix. Fixed by
adding ^ to _is_spiral_wrapper_prefix's recognized prefix set.
Regression test in tests/test_entity_id_template_discriminators.py
(test_live_clang_block_pointer_spiral_return_declarator_preserves_returned_function_parameter_list,
using a local _clang_blocks_parser helper since this is the only case
in this module needing -fblocks enabled), confirmed to fail pre-fix via
git stash on just the source file and pass post-fix; every existing
return-type discriminator test in that module still passes unchanged.
mypy abicheck/ clean, ruff check/ruff format --check clean,
check_architecture.py 0 errors.
Correction (2026-08-29, same day, CodeRabbit review on PR #943): a
dependent return type containing a quoted literal with an unbalanced
paren character corrupted the top-level paren scan itself. Confirmed
by direct compilation: template<class T> decltype("(") f(T);'s
qualType is 'decltype("(") (T)' -- the literal's own ( character
was counted as real declarator structure by _top_level_paren_spans's
naive bracket-depth counter (which has no concept of quoted text), so
its running depth count from the opening ( at decltype( never
returned to zero before the string ended, swallowing the real trailing
(T) parameter-list group along with everything else and reducing
return_type to the bare "decltype" -- a ")"-literal sibling
(decltype(")") f(T);) happened to produce a different, but equally
wrong, truncation, so the two were also distinguishable by accident,
which is why this needed a repository review to surface rather than
being caught by any existing test asserting an exact expected value. A
second claim in the same review round -- that
canonicalize_type_param_references's quoted_literal_spans helper
mishandles a raw string literal's R"(...)" delimiters -- was
investigated and found NOT reproducible against real clang output:
direct compilation of template<class T, decltype(R"(T)") N> void f();
shows clang's own printed qualType for N is "decltype(\"T\")" --
the raw-string prefix/delimiter never survives into the printed type at
all, since clang's type printer always re-renders a string literal's
value using ordinary quoted-string syntax, not its original source
spelling. Every call site of quoted_literal_spans/
canonicalize_type_param_references operates only on clang-derived
qualType text, so a raw-string delimiter is not a real input this code
path can ever see; no fix applied for that half, and no regression test
added for an input that cannot occur. Fixed the confirmed half by adding
quoted_literal_spans (abicheck/model/identity_literals.py, extract
-> model, ADR-061's allowed direction) as a skip-list to both
_top_level_paren_spans and _find_top_level_arrow: a literal span is
now hopped over entirely when tracking bracket/paren depth, rather than
having its contents inspected character-by-character. Regression test in
tests/test_entity_id_template_discriminators.py
(test_live_clang_string_literal_in_return_type_does_not_confuse_paren_scan),
confirmed to fail pre-fix via git stash on just the source file and
pass post-fix; every existing return-type discriminator test in that
module still passes unchanged. mypy abicheck/ clean (515 files),
ruff check/ruff format --check clean, check_architecture.py 0
errors (confirming the new extract -> model import introduces no
cycle).
Correction (2026-08-29, same day, Codex review on PR #943): the
spiral-declarator sigil check alone was too permissive, matching a
decltype operand's own unrelated expression syntax. Confirmed by
direct compilation: template<class T> decltype(*(typename T::x *)0)
f(T);'s qualType is "decltype(*(typename T::x *)0) (T)" -- a
dereferenced C-style cast, not a declarator, whose first top-level
group's interior (*(typename T::x *)0) nonetheless starts with the
bare sigil * that _is_spiral_wrapper_prefix treats as a
function-pointer-return marker. The check fired, and
_excise_own_param_list discarded the entire dependent operand,
collapsing this and a legal T::y sibling onto the identical
"decltype (*()0) (T)". Fixed by requiring _is_spiral_wrapper_prefix
to also validate what follows the sigil's first nested group: nothing,
the original function's own exception specification, or a
further-nested spiral level's own parameter list (verbatim (...)) are
all genuine declarator continuations; anything else (here, the bare
token 0) is expression text, not a declarator, and the check now
returns False for it, falling through correctly to the scan-from-end
branch, which keeps the entire dependent operand verbatim as
distinguishing return-type content. Regression test in
tests/test_entity_id_template_discriminators.py
(test_live_clang_decltype_dereferenced_cast_is_not_mistaken_for_spiral_declarator),
confirmed to fail pre-fix via git stash on just the source file and
pass post-fix; every existing return-type discriminator test in that
module (spiral pointer/reference/member-pointer/block-pointer cases
included) still passes unchanged. mypy abicheck/ clean, ruff check/
ruff format --check clean, check_architecture.py 0 errors.
Correction (2026-08-29, same day, Codex review on PR #943): a trailing
GNU __attribute__((...)) clause was mistaken for the function's real
parameter list, and separately leaked verbatim into a spiral
declarator's return type. Confirmed by direct compilation:
int f(int) __attribute__((sysv_abi));'s qualType is "int (int)
__attribute__((sysv_abi))" -- the attribute clause's own argument
group (((sysv_abi))) is a second top-level span, and the fallback
branch's scan-from-end picked it as "the last group not preceded by
noexcept/throw", swallowing the real parameter list (int) into
what was reported as the return type ("int (int) __attribute__"
instead of "int"). Separately, template<class T> int (*f(T))(int)
__attribute__((sysv_abi));'s spiral-branch tail (kept verbatim
otherwise, since an exception specification there is real,
distinguishing return-type content per the earlier correction above)
carried the attribute text straight through too ("int (*())(int)
__attribute__((sysv_abi))" instead of "int (*())(int)"). Unlike an
exception specification -- part of the function's TYPE since C++17 --
a GNU attribute is never part of the type: it doesn't affect overload
resolution or type identity, so both hazards needed fixing, and neither
the same way as an exception spec. Fixed by (1) extending
_EXCEPTION_SPEC_KEYWORD_RE (the fallback branch's own group-exclusion
check) to also exclude a group immediately preceded by
__attribute__, and (2) adding _strip_trailing_gnu_attribute, applied
to the spiral branch's tail, which repeatedly strips one or more
trailing __attribute__((...)) clauses outright (never keeping them,
unlike the exception-spec tail case). Regression test in
tests/test_entity_id_template_discriminators.py
(test_live_clang_trailing_gnu_attribute_does_not_leak_into_return_type),
covering both the ordinary scan-from-end case and the spiral case,
confirmed to fail pre-fix via git stash on just the source file and
pass post-fix; every existing return-type discriminator test in that
module still passes unchanged. mypy abicheck/ clean, ruff check/
ruff format --check clean, check_architecture.py 0 errors.
Correction (2026-08-29, same day, Codex review on PR #943): the
previous correction's own "strip every trailing attribute" rule was
itself wrong -- it can silently erase a real, ABI-breaking
calling-convention difference on the RETURNED function of a spiral
declarator. Confirmed by direct compilation on an i386 target
(where the distinction is observable; both collapse to the platform
default on x86-64, which is why the previous correction's own
sysv_abi probe never caught this): int (__attribute__((stdcall))
*h())(); and int (*hc())(); produce DIFFERENT qualTypes --
"int (*())() __attribute__((stdcall))" vs. "int (*())()" -- for a
genuine ABI difference (stdcall vs. cdecl disagree on stack-cleanup
responsibility), which the previous correction's
_strip_trailing_gnu_attribute, applied unconditionally to the spiral
branch's tail, silently erased. Worse, direct compilation also confirms
this attribute CANNOT be reliably attributed to "the outer function" by
position alone: writing the identical attribute at the very END of the
whole declaration instead of on the returned pointer explicitly
(int (*h())() __attribute__((stdcall));) produces the BYTE-IDENTICAL
qualType, so clang's own printer offers no textual way to tell which
function a trailing spiral attribute binds to. Given that ambiguity,
fixed by reverting the spiral branch's tail to being kept fully
verbatim again (no attribute stripping at all) -- the same treatment
exception specifications already get there, and for the same reason:
erring toward reporting a difference that turns out to be the outer
function's own is a strictly safer failure mode for an ABI checker than
silently erasing a real one. The FALLBACK (scan-from-end) branch's own
fix -- excluding an attribute-preceded group from being mistaken for
the real parameter list -- remains correct and unchanged; an ordinary
(non-spiral) function's own trailing attribute was never ambiguous (it
sits in the group's own suffix, never the prefix that becomes
return_type) and continues to be excluded naturally, with no special
stripping needed. _strip_trailing_gnu_attribute and its
_TRAILING_ATTRIBUTE_KEYWORD_RE are removed entirely as dead code.
Regression tests in tests/test_entity_id_template_discriminators.py:
the previous single test is split into
test_live_clang_ordinary_functions_own_trailing_attribute_does_not_leak_into_return_type
(unchanged behavior, kept as its own regression) and
test_live_clang_spiral_returns_own_calling_convention_attribute_is_preserved
(new, using a local _clang_i386_parser helper since the distinction
needs a 32-bit x86 target), the latter confirmed to fail against the
previous (over-eager stripping) commit via git stash on just the
source file and pass post-fix; every existing return-type discriminator
test in that module still passes unchanged. mypy abicheck/ clean,
ruff check/ruff format --check clean, check_architecture.py 0
errors.
Correction (2026-08-29, same day, CodeRabbit review on PR #943): a
relational/shift operator inside a paren-wrapped non-type template
argument permanently corrupted the bracket-depth counter both scanners
share. Confirmed by direct compilation: template<class T>
std::enable_if_t<(sizeof(T) < 4), int> f(T);'s qualType is
"std::enable_if_t<(sizeof(T) < 4), int> (T)" -- the OLD
_top_level_paren_spans tracked bracket depth only at the outer
scanning level (switching to a paren-only inner loop once a "(" was
seen with bracket already 0, never touching bracket state again until
that inner loop's own parens closed): the relational < here was
reached with bracket already at 1 (from enable_if_t<'s own opening),
so it never entered that inner loop at all and instead incremented the
SAME bracket counter to 2, which the qualType's one remaining > could
only ever bring back down to 1 -- permanently stuck above zero, so the
real trailing (T) was never recognized as a top-level group at all,
and return_type retained the whole string verbatim (including a
sibling declaration's trailing noexcept), corrupting both functions'
identity. The identical corruption affected _find_top_level_arrow when
the relational operator appeared in a PARAMETER instead: auto
f(std::enable_if_t<(sizeof(T) < 4), int>) -> T;'s arrow was never
found, falling back to the bare placeholder "auto". Fixed by
unifying bracket-depth and paren-depth tracking into ONE pass in both
scanners: a bare </> is only ever counted as a bracket while paren
depth is ALSO zero (i.e. not currently inside any already-open
parenthesized group) -- this matters because a non-type template
argument containing a relational/shift operator must, per the grammar,
be wrapped in its own parens to disambiguate it from the closing >,
so once such a paren group has opened, any </> inside it can only be
that operator, never a genuine template bracket, regardless of what
unrelated <...> surrounds the whole expression -- while a paren group
already open for an UNRELATED reason (e.g. the relational operator's own
wrapping parens) still correctly tracks its OWN "(" and ")" balance via
paren depth, independent of bracket state, so it closes correctly and
returns bracket-tracking control to the outer scan afterward. Regression
test in the new tests/test_entity_id_return_type_discriminators.py
(split out of tests/test_entity_id_template_discriminators.py, which
had grown past the architecture gate's 1200-line test-file cap -- no
test content changed by the split itself) --
test_live_clang_relational_operator_in_template_argument_does_not_corrupt_bracket_depth,
covering both the paren-scan and arrow-scan cases, confirmed to fail
against the pre-fix code via git stash on just the source file and
pass post-fix; every existing return-type discriminator test still
passes unchanged. mypy abicheck/ clean, ruff check/ruff format
--check clean, check_architecture.py 0 errors.
Correction (2026-08-29, same day, Codex review on PR #943): the
remainder-based spiral-wrapper check has an irreducible blind spot for
an EMPTY remainder, closed with a more general rule instead of another
remainder heuristic. Confirmed by direct compilation:
decltype(&(S::x)) f();'s qualType is "decltype(&(S::x)) ()" -- an
address-of a parenthesized member-access expression, whose leading &
sigil and EMPTY remainder after its own nested group ((S::x)) is
BYTE-FOR-BYTE indistinguishable, by the remainder check alone, from a
genuine reference-returning spiral declarator with no parameters (int
(&f())();, qualType "int (&())()") -- both are <sigil>(<content>)
with nothing following. No refinement of "what follows the group" can
ever close this specific shape, since there is nothing following in
either case; the two are only distinguishable by what's INSIDE the
group (a type vs. an arbitrary expression), which plain text scanning
cannot determine in general. Fixed with a different, SOUND signal
instead of attempting to refine the remainder check further: a group
whose immediately preceding text (no space, exactly as clang always
prints it) is the bare token decltype is ALWAYS that operator's own
parenthesized operand, never a declarator wrapper, regardless of what
the operand's own text starts with -- this closes the address-of case
here, the earlier dereferenced-cast case, and (by the same reasoning)
any future construct sharing the identical shape, in one general rule
rather than adding a third sigil- or remainder-specific patch.
_is_spiral_wrapper_prefix now takes the group's own preceding text as
a second parameter to check this before its existing sigil/remainder
checks (which remain, as a second line of defense, for any construct
other than decltype this hasn't been confirmed to need). Regression
test in tests/test_entity_id_return_type_discriminators.py
(test_live_clang_decltype_address_of_expression_is_not_mistaken_for_spiral_declarator),
confirmed to fail against the pre-fix code via git stash on just the
source file and pass post-fix; every existing return-type discriminator
test still passes unchanged. mypy abicheck/ clean, ruff check/ruff
format --check clean, check_architecture.py 0 errors.
Correction (2026-08-29, same day, Codex review on PR #943): a
QUALIFIED MEMBER-TEMPLATE name is one qualifier keyword past what the
existing ::/./-> exclusions reach. Confirmed by direct
compilation: struct Base { template<class T> static int N(); };
template<int N, class S> void f(decltype(S::template N<int>())); keeps
S::template N<int>() verbatim in clang's own qualType regardless of
the non-type parameter N's own name -- but canonicalize_type_param_
references's existing (?<!::) lookbehind (which already protects the
plain S::N shape) doesn't reach this N, since it's separated from
:: by the literal template keyword and a space, not adjacent to it.
Substituting it anyway fingerprinted a pure rename of the unrelated
outer non-type parameter (N -> M) as a remove+add, since the
member-template call's own raw text is identical either way. Fixed by
adding a fourth fixed-width negative lookbehind, (?<!::template ), to
the same combined pattern the ::/./-> exclusions already share.
Regression test in tests/test_entity_id_template_discriminators.py
(test_live_clang_qualified_member_template_name_is_not_canonicalized_as_a_param_ref),
confirmed to fail against the pre-fix code via git stash on just the
source file and pass post-fix; every existing rename-invariance test
still passes unchanged. identity.py's own docstrings condensed further
to stay at the AI-readiness gate's 800-line production cap after this
addition (no wording removed beyond redundant restatement). mypy
abicheck/ clean, ruff check/ruff format --check clean,
check_architecture.py 0 errors.
Correction (2026-08-30, same day, Codex review on PR #943): a
USER-DEFINED LITERAL SUFFIX ('x'_tag) is a fifth hazard the existing
literal exclusion doesn't reach, since it sits OUTSIDE the quoted
literal span rather than inside it. Confirmed by direct compilation:
struct X { char v; }; constexpr X operator""_tag(char c) { return
{c}; }; template<int _tag> void f(decltype('x'_tag)); keeps 'x'_tag
verbatim in clang's own qualType regardless of the non-type parameter
_tag's own name -- the existing quoted-literal exclusion in
canonicalize_type_param_references checks whether a candidate
match's start falls INSIDE a literal span ('x'), but a UDL suffix
(_tag) is the token immediately AFTER the literal's closing quote,
outside every recorded span, so it wasn't excluded and got substituted
like an ordinary identifier. Fixed by also excluding any match whose
start coincides exactly with a literal span's own end index (a UDL
suffix always attaches with no space directly after the closing quote,
per the language grammar, so this is a precise, not heuristic, test).
Regression test in tests/test_entity_id_template_discriminators.py
(test_live_clang_user_defined_literal_suffix_is_not_canonicalized_as_a_param_ref),
confirmed to fail against the pre-fix code via git stash on just the
source file and pass post-fix; every existing rename-invariance test
still passes unchanged. identity.py's own docstrings condensed
further still to stay at the 800-line production cap after this
addition. mypy abicheck/ clean, ruff check/ruff format --check
clean, check_architecture.py 0 errors.
Landed (sixth slice, 2026-08-30): a first, bounded piece of (c2), not
all of it. (c2) has two stated deliverables (see the fourth slice's own
"Next slice" note above): the finding_identity.py algorithm migration
itself, and giving Change an EntityId to key on. This slice lands only
the first half of the first deliverable. finding_identity.
resolve_function_identity's per-parameter canonicalization -- previously
a second, independently-maintained canonicalize_type_name(p.type) call,
duplicating rather than reusing the cross-producer-spelling/cv-qualifier
logic entity_id_for_function's own "sig" fallback branch already
established -- now calls model.signature_normalization.
canonicalize_function_signature_param_type directly, the same primitive.
Not a pure refactor: that primitive additionally drops a top-level
BY-VALUE cv-qualifier (void f(int)/void f(const int) name the same
function per the C++ standard's own linkage rules), which
canonicalize_type_name deliberately does not -- so a DWARF-only,
non-mangled function whose parameter merely gained or lost a top-level
const no longer fragments into two NORMALIZED-tier identities. A
pointee cv-qualifier (char * vs. const char *) remains genuinely
distinguishing either way; both directions are pinned by new tests in
tests/test_finding_identity.py
(test_by_value_cv_qualifier_no_longer_distinguishes_overloads,
test_pointee_cv_qualifier_still_distinguishes_overloads). The
mangled-name-is-genuine determination (is_real_mangled_name/
normalize_mangled_name) stays owned by finding_identity.py, unchanged
-- model/identity.py's own docstring already records why moving it would
reverse the required compare -> model import direction (entity_id_for_
function's contract only ever needs an already-vetted mangled name as
input, never finding_identity's own validation logic). Full fast unit
suite green; mypy abicheck/ and ruff check/ruff format --check
clean.
What this slice deliberately does not attempt. resolve_variable_
identity takes no param_types argument at all, so it has nothing
equivalent to delegate (variables have no signature-shaped discriminator
to canonicalize). Giving Change an EntityId to key on -- (c2)'s other,
larger deliverable, which touches checker_types.Change,
resolve_change_identity, and the diff_symbols.py/diff_filtering.py
call sites that construct a Change -- is not attempted here; neither is
(b), the post-parse consumer migrations (diff_filtering.py's
_find_opaque_types/_find_by_value_types/_root_type_name and
type_reachability.py's ambiguity machinery). Both remain open, and
no consumer may read the entity_id carrier field itself yet --
this slice recomputes the signature discriminator from Function's own
raw fields on every call, the same "one algorithm, not a cached value"
discipline the carrier's own design note states, rather than reading the
carrier Function.entity_id may already hold. Next slice, in order:
give Change an EntityId (closing (c2)), then the post-parse consumer
migrations (b), per the fourth slice's own sequencing note above, which
this slice does not otherwise revise.
Landed (seventh slice, 2026-08-30): Change gains its own entity_id
carrier, populated at every diff_symbols.py function-diff call site --
not yet the exhaustive-population or consumer halves of (c2)/(b).
checker_types.Change gains entity_id: EntityId | None = field(default=
None, kw_only=True), appended last per the file's own established
per-field-kw_only convention (Change is public API; see that
convention's own comment, condensed in the same commit to make room under
checker_types.py's zero-slack debt.yaml no_growth pin). Semantics:
the OLD side's Function.entity_id when it exists, else the NEW side's --
mirroring Change.symbol_binding's own already-documented old-side
convention, rather than inventing a new rule. Wired at every
function-level call site in diff_symbols.py where the producing
Function object(s) are already in scope: _check_removed_function (both
branches), _check_return_type_change, _check_params_change,
_check_ref_qualifier_change, _check_linkage_change, the
_check_contract_attributes_change's three sites, _check_exception_spec_
change, _check_vtable_index_change, _check_inline_transitions's two
sites, the FUNC_ADDED site in _diff_functions (new side, since there is
no old side), and _detect_newly_deleted_functions (old side's, via
f_old_any.entity_id or f_new.entity_id, since f_old_any -- present only
when the symbol persisted before gaining = delete -- can be None).
diff_helpers.bool_transition (backing the noexcept/virtual/explicit/
variadic checks) gains a matching entity_id: EntityId | None = None
parameter, passed through to both Change(...) constructions unchanged --
it has no declaration of its own to derive one from, so resolving it stays
the caller's job.
What this slice deliberately does not attempt, named explicitly rather
than left to be discovered: (1) the other family of Change(...)/
make_change(...) construction sites -- variable-diff, type/enum/platform/
versioning detectors -- even though several of those already construct
Change from a RecordType/EnumType/Variable that also carries its
own .entity_id (a ~400-site total population, per the scoping
investigation this slice's own PR ran before implementing); (2)
finding_identity.resolve_change_identity/_change_discriminator reading
Change.entity_id at all -- both functions still key entirely on
change.symbol/change.qualified_name/flat value fields, unchanged by
this slice, so no consumer may read this carrier field yet, the
identical staging the model-layer entity_id carriers (Function.
entity_id et al.) already went through. Verified via a new, real
compare()-level regression suite (tests/test_change_entity_id_carrier.py)
rather than only unit-testing the private _check_* helpers directly:
FUNC_RETURN_CHANGED/FUNC_REMOVED/FUNC_ADDED/FUNC_NOEXCEPT_ADDED each pin the
carrier's value end to end, plus a no-producer-entity_id case confirming
Change.entity_id stays honestly None rather than fabricating one. Full
fast unit suite green; mypy abicheck/ and ruff check/ruff format
--check clean; check_architecture.py 0 new errors (checker_types.py
and diff_symbols.py both land at or under their existing debt.yaml
baselines; diff_helpers.py -- not itself debt-tracked, but sitting
exactly at the architecture gate's general 800-line production ceiling --
required the identical "trim existing verbose comments to make room"
treatment checker_types.py needed, condensing TypeMap's and
type_map_key's own pre-existing docstrings rather than growing past it).
Next slice, in order: resolve_change_identity/_change_discriminator
consuming Change.entity_id (the true completion of (c2)), then (b), the
post-parse consumer migrations -- exhaustive Change.entity_id population
beyond the function-diff path is a separate, larger follow-on this
sequencing does not schedule.
Landed (eighth slice, 2026-08-31): exhaustive function-diff population,
plus resolve_change_identity consuming Change.entity_id -- the true
completion of (c2). Two parts, landed together since the first is what
made the second worth doing. First, the same old-else-new entity_id
fallback reached every remaining function-backed Change construction
site the seventh slice's own scoping investigation had found but not yet
wired: diff_hidden_friends.py (HIDDEN_FRIEND_ADDED/REMOVED, both the
matched-pair bool_transition call and the two single-sided make_change()
sites), diff_param_qualifiers.py (PARAM_RESTRICT_CHANGED,
PARAM_BECAME_VA_LIST/LOST_VA_LIST), diff_symbols.py's own
_diff_ctor_overload_ambiguity (CTOR_OVERLOAD_AMBIGUITY_RISK, a
single-sided new-side-only site), and the auxiliary
param_defaults/param_renames/pointer_levels/method_access/
func_deprecated/func_override_specifier detectors -- ten sites in
total, found and fixed across four review rounds (each fix drew a fresh
"still missing a sibling site" finding until a full audit of every
make_change()/Change() call in diff_symbols.py and its split-out
sibling modules, not just the named sites, actually closed the class).
Second, EntityId gained its own .key property (model/identity.py) --
a flat, collision-safe string (_packed, a local duplicate of
storage.entity_ids._packed's own audited length-prefixing scheme, since
model may not import storage), excluding every compare=False payload
field (Record.access) so two == segments produce the same key.
finding_identity.resolve_change_identity folds change.entity_id.key in
as a new entity:<key>\x1f<discriminator> alias, qualified with the
discriminator like every other alias there so two distinct findings on the
same entity never collide -- additive only: never promoted to
primary_id/tier, so every existing suppression rule and canonical finding
ID is bit-for-bit unchanged (tests/test_canonical_finding_id*.py pin
that). Deliberately conservative rather than replacing the CANONICAL tier's
mangled-name basis outright: EntityId.key's cross-release stability is
not yet established the way a persisted wire identity would need to be
(Anonymous/LocalToFunction's own ordinals are parse-order-only, not
cross-revision, by their own documented design), so promoting it to
primary_id now would risk silently reshaping every stored suppression
rule's matching behavior -- exactly the class of risk this slice's own
scoping investigation flagged as needing a separate, explicit design
decision rather than a rushed migration. Tests: tests/
test_model_identity.py::TestEntityIdKey (collision-safety across sibling
scope-segment variants, the extra-tuple boundary-forgery case in the same
spirit as storage.entity_ids's own audited collision test, Record.
access exclusion, LocalToFunction recursion) and tests/
test_change_entity_id_carrier.py::TestResolveChangeIdentityConsumesEntityId
(the alias's presence/absence and non-collision, through compare() +
resolve_change_identity end to end) plus one regression test per newly-
wired site in tests/test_change_entity_id_carrier.py, each confirmed to
fail against the pre-fix code. Full fast unit suite green; mypy
abicheck/ and ruff check/ruff format --check clean;
check_architecture.py 0 errors (finding_identity.py, diff_symbols.py,
and model/identity.py all land at or under their debt.yaml/general
800-line baselines, each requiring the same "condense existing verbose
docstrings to make room" treatment prior slices needed). Remaining Phase 2
items unchanged by this slice: exhaustive entity_id population beyond
the function-diff path, the post-parse consumer migrations, and promoting
the new entity: alias into a real alias-match reconciliation tier once
EntityId.key's stability question above is resolved.
Landed (ninth slice, 2026-09-01): entity_id reaches the non-diff_types.py
tail of exhaustive population, plus one hardening attempt explicitly not
taken. First part: every remaining make_change()/bool_transition()
call site in diff_layout.py, diff_vtable_layout.py, and
diff_symbols_variables.py now carries entity_id -- all ten sites
across those three files (diff_layout.py's six layout-descriptor findings,
diff_vtable_layout.py's two vtable-group findings,
diff_symbols_variables.py's VAR_ACCESS_CHANGED/WIDENED and
VAR_ALIGNMENT_CHANGED), each keyed off the two matched RecordType/
Variable objects already in scope at the call site
(old_rec.entity_id or new_rec.entity_id, following the seventh/eighth
slice's old-preferred/new-fallback convention). diff_symbols.py's own
remaining gap closed too: _check_variable's VAR_TYPE_CHANGED and its
bool_transition const-flip pair, _var_removed/_var_added
(VAR_REMOVED/VAR_ADDED, one-sided), _check_field_access_changes
(FIELD_ACCESS_CHANGED, attributed to the containing RecordType since a
TypeField carries no EntityId of its own), _check_anon_field_at_offset
(ANON_FIELD_CHANGED, both branches -- the containing-record identity is
now threaded through as an explicit parameter from
_check_anon_fields_for_type, since the field-level helper itself never
had a RecordType in scope), and _diff_var_deprecated
(VAR_DEPRECATED_ADDED/REMOVED). Nine sites in diff_symbols.py, all
regression-tested by the existing compare=False invariant (no equality
based test can regress) and confirmed by a direct code-level count
(make_change/bool_transition call-site keyword arguments vs. the file's
total make_change/bool_transition call count, distinguishing an actual
call-site entity_id= keyword from the one signature parameter declaration
and the one local entity_id = ... variable assignment this slice also
added). Three sites deliberately stay unwired, all in diff_symbols.py's
_diff_constants (CONSTANT_REMOVED/CHANGED/ADDED): AbiSnapshot.
constants is a plain dict[str, str], with no parsed declaration object
behind either side to carry an EntityId -- fabricating one was explicitly
out of scope. Second part: the type_reachability.py/
type_reachability_spelling.py hardening this slice was scoped to look at
(_record_identity(name, qualified_name)'s three call sites --
_partition_snapshot_types, and the two snapshot.enums-keyed
enum_identities sites in directly_referenced_stdlib_types/its sibling)
was investigated and not taken: every one of those three call sites'
output strings feed directly into _non_stdlib_signature_spellings, which
is consumed by _StdlibReferenceScan's own substring/spelling-matching
machinery -- the exact "core signature-spelling substring-matching
machinery" this slice's own scope explicitly excluded. There is no
_record_identity call site in either module that is an independent
RecordType/EnumType-keyed anchor separable from that string-spelling
domain; swapping any of the three to prefer .entity_id.key would mix
opaque EntityId keys into a domain whose whole contract is "a spellable
string that can appear inside a rendered Function.return_type/
Param.type/TypeField.type", breaking the matching algorithm for any
type that happens to carry a populated entity_id. Verification: full fast
unit suite green, mypy abicheck/ clean at the documented 0-error
baseline, ruff check/ruff format --check clean on every file this
slice touched (two pre-existing, unrelated formatting violations were
independently confirmed present on the base commit before this slice's own
changes, via git stash). Remaining Phase 2 items unchanged by this
slice: entity_id population for DWARF/PDB/ELF-only tiers (still always
None there, an out-of-scope gap noted since the background section
above), the post-parse consumer migrations, and promoting the entity:
alias into a real alias-match reconciliation tier.
Landed (tenth slice, 2026-09-01): entity_id reaches diff_types.py/
diff_types_field_facts.py, closing exhaustive population for the
type-diff detector family. Every make_change() call site in
diff_types.py (43 sites) and diff_types_field_facts.py (16 sites) now
carries entity_id, following the same old-preferred/new-fallback
convention (t_old.entity_id or t_new.entity_id for a matched
RecordType/EnumType/Function pair; the single available side's own
entity_id for a one-sided add/removal). Field-level findings (a struct/
union field or enum member gaining, losing, or changing a property)
attribute to the containing RecordType/EnumType's entity_id, since
TypeField carries no EntityId of its own (EntityKind.FIELD is
declared but unimplemented -- the same conclusion the ninth slice reached
for diff_symbols.py's field-level detectors). Four helper functions that
previously had no record object in scope at all at their own call sites
(_try_match_reserved_field, _diff_removed_field, and
_diff_type_field_pair in diff_types.py; _check_field_qualifier_pair
in diff_types_field_facts.py) gained an explicit entity_id
keyword-only parameter threaded down from their caller (_diff_type_fields/
_diff_field_qualifiers, both of which do have the matched RecordType
pair in scope), mirroring the existing qualified_name threading pattern
already established at those same call sites. 56 of 59 make_change()
sites wired (40/43 in diff_types.py, 16/16 in
diff_types_field_facts.py); three sites deliberately stay unwired --
diff_types.py::_diff_typedefs's TYPEDEF_VERSION_SENTINEL/
TYPEDEF_REMOVED/TYPEDEF_BASE_CHANGED -- since _typedef_diff_maps
produces plain str -> str alias maps with no parsed declaration object
behind either side to carry an EntityId (matching the ninth slice's
identical conclusion for diff_symbols.py::_diff_constants;
entity_id_for_typedef in model/identity.py still has zero confirmed
production callers). One OVERLOAD_ADDED site in
_diff_overload_additions is a genuine new shape not seen in earlier
slices: its "new" side is a group of overload Functions rather than a
single matched object, so it is stamped with only the still-present
old-side declaration's own entity_id. Verification: full fast unit suite
green (one pre-existing, unrelated failure --
test_platform_matrix.py::test_type_param_diff_implies_symbol_diff, a
dataclasses/sys.modules introspection error confirmed present on the
base commit via git stash -- and one pre-existing mypy error in
frontends/cli/commands/dump.py from concurrent, unrelated work in
progress on this branch; neither touched by this slice); mypy abicheck/
clean except that one pre-existing, unrelated error; ruff check/ruff
format --check clean on both files this slice touched. Remaining Phase 2
items unchanged by this slice: entity_id population for DWARF/PDB/
ELF-only tiers, the post-parse consumer migrations, and promoting the
entity: alias into a real alias-match reconciliation tier.
Landed (eleventh slice, 2026-09-01): diff_types.py split into
diff_types_bases_vtable.py — a mechanical AI-readiness follow-up, not
new ADR-063 semantic work. The tenth slice's entity_id= wiring pushed
diff_types.py from 1999 to 2066 lines, past the AI-readiness gate's
2000-line hard cap (file-size, no allowlist mechanism), and shifted the
line numbers thirteen fact-field-readers KNOWN_UNMIGRATED_READERS
baseline entries in diff_layout.py/diff_types.py/
diff_vtable_layout.py were keyed against. Fixed by extracting the
largest genuinely self-contained function group in diff_types.py --
base-class diffing (_diff_type_bases) and the vtable-diffing group
feeding _diff_type_vtable (_vtable_transition_is_evidenced,
_vtable_transition_rests_on_unresolved_evidence,
_layout_evidence_is_unverifiable, _owned_virtual_signatures,
_owned_virtual_signatures_for_record) -- 501 lines, into a new sibling
leaf module, abicheck/diff_types_bases_vtable.py, following the exact
precedent diff_types_field_facts.py/diff_types_abicc_parity.py/
diff_types_surface.py already established for this same file: diff_types.py
imports the moved names back with the same as-aliased re-export
convention (so from .diff_types import _vtable_transition_is_evidenced,
which tests/test_vtable_evidence_guard.py uses directly, keeps
resolving), and the new module imports nothing from diff_types.py
itself, so no import cycle is introduced. diff_types.py now sits at
1566 lines (well under the cap, restoring margin), and the new module is
553 lines. Registered in architecture/modules.yaml's compare layer
legacy_paths and frozen_root_families.diff_ allowlist alongside its
three siblings -- required since ADR-061's architecture gate
(scripts/check_architecture.py) independently forbids an unregistered
new flat diff_*.py root module; the new module is exempt from
compare/'s own may_import: [model] restriction the same way its
already-registered siblings are, since that restriction only applies to
files physically inside abicheck/compare/, not to a legacy_paths-listed
flat file (verified against the checker's own _source_layer_for/
migrated_source logic before registering, rather than assumed).
The thirteen fact-field-readers baseline entries were updated in
scripts/fact_field_readers.py's KNOWN_UNMIGRATED_READERS (a 37-key
swap covering every entry whose file, containing-expression text, or both
moved: the ten in diff_types.py for _diff_type_bases/
_vtable_transition_is_evidenced/
_vtable_transition_rests_on_unresolved_evidence/_diff_type_vtable now
key off diff_types_bases_vtable.py with the entity_id=-bearing
call-site text, plus the twenty-two sibling reads in that same moved
block the task's own error list didn't enumerate individually, plus the
three genuinely-stale entries in diff_layout.py/diff_vtable_layout.py
whose exact text shifted from the tenth slice's entity_id= insertion
without moving file) -- derived mechanically via
fact_field_readers.unmigrated_fact_reader_sites() run against the
post-split tree rather than hand-edited, so the new keys are exactly what
the scanner itself would produce. No reader's own logic changed; this is
purely baseline-key bookkeeping for code that already existed pre-split.
Verification: python scripts/check_ai_readiness.py back to the
documented 3 pre-existing adr-status-sync errors only (zero new
file-size/fact-field-readers errors); ruff check/ruff format
--check/mypy abicheck/ clean; python scripts/check_architecture.py
clean except one pre-existing, unrelated debt-no-growth finding on
diff_symbols.py from the tenth slice's own concurrent, uncommitted
entity_id= wiring (not touched by this slice); targeted tests
(test_diff_types_deep.py, test_vtable_evidence_guard.py,
test_vtable_severity.py, test_checker.py,
test_ai_readiness.py, test_fact_field_readers.py, plus a
type/layout/enum/vtable-keyed slice of the full suite, ~4370 tests) green
-- the same one pre-existing, unrelated test_platform_matrix.py
test-isolation failure the tenth slice's own entry already names
reproduces only under that large combined run and passes standalone,
confirming it predates this slice too.
Update (2026-09-01, merge with main): diff_types_bases_vtable.py no
longer exists — folded into main's independently-landed
diff_types_vtable.py. Merging this branch's PR against main surfaced
that a2cc048 ("feat(model): complete ADR-063 Phase 0 detector migration")
had, concurrently and independently, split the identical base-class/vtable
function group out of diff_types.py into its own new sibling module --
diff_types_vtable.py (same functions, same rationale, already Fact-migrated
per Phase 0) -- landing on main before this slice's own extraction merged.
Rather than keep two near-duplicate modules, the merge conflict was resolved
by keeping main's diff_types_vtable.py (it already carries the Phase 0
resolved_fact_value() migration this slice's own copy did not) and layering
only this slice's entity_id= addition onto its _diff_type_vtable; _diff_
type_bases itself stayed where main already had it, inline in
diff_types.py (also Fact-migrated), with entity_id= added to its three
make_change() calls. diff_types_bases_vtable.py was deleted along with
its architecture/modules.yaml registrations; the fact-field-readers
baseline resolved to main's side (now empty -- Phase 0 migrated every
known reader, so there is nothing left for this slice's raw-field version of
these functions to add back). No behavior change beyond what each side
already carried; tests/test_vtable_evidence_guard.py's existing from
abicheck.diff_types_vtable import _vtable_transition_is_evidenced import
was the confirming signal for which filename to keep.
Landed (twelfth slice, 2026-09-01): typedefs and constants — the last
Change-producing detector families with no entity_id at all — now carry
one. entity_id_for_typedef/entity_id_for_constant had existed in
model/identity.py since the first slice with zero production callers,
because AbiSnapshot.typedefs/typedefs_qualified/constants are plain
dict[str, str] with no parsed declaration object to hang a carrier field
on, the way RecordType/EnumType/Function/Variable have. The scope
was not missing, only discarded: both header-AST backends already walk it
(clang's _Decl.scope_path, castxml's _scope_path(el)) at the exact point
those three dicts are built. So this slice reads it one step earlier rather
than extending any AST walk.
Two additive AbiSnapshot sidecars carry it — typedef_entity_ids (keyed
like typedefs_qualified) and constant_entity_ids (keyed like
constants), both kw_only, both defaulting to {}. Producers:
dumper_clang.parse_typedef_entity_ids/parse_constant_entity_ids and
dumper_castxml's same-named pair. On both backends the constant sidecar
shares one filtering pass with parse_constants itself (clang gained a
_iter_public_constants() helper; castxml's existing one now yields the
element too), so the two maps cannot disagree about which constants qualify
— the property the join depends on. The castxml typedef sidecar calls the
resolver from dumper_castxml.py itself, iterating
dumper_castxml_typedefs.iter_typedef_entries rather than adding a resolver
call to that helper module, since tests/test_entity_id_carrier.py's
_ALLOWED_RESOLVER_CALLERS confines entity_id_for_* to the two backend
front doors.
Plumbing, all of it in the same direction as the dict each sidecar
annotates: TuFragment/MergedTuFragments/ElfHeaderAstResult carry them
so a manifest (and the legacy single-header ELF path, which also routes
through TuFragment) does not lose what a direct PE/Mach-O parse keeps;
tu_merge.merge_fragments unions them plainly rather than through
_merge_scalar_group (that helper diagnoses two TUs disagreeing about a
key's value, and an EntityId is derived entirely from the key's own
scope and leaf name, so there is no disagreement to report);
dumper_hybrid unions them castxml-wins, matching typedefs_qualified;
storage/entity_id_codec.py gained an encode_sidecar_entity_ids/
decode_sidecar_entity_ids pair (a dict-of-documents encode, not the
positional-list pairing the declaration carrier uses) and lost its
now-false "typedefs/constants get no carrier" comment;
serialization.py bumped SCHEMA_VERSION 30 → 31, with an absent sidecar
loading as {} — no migration adapter, the identical reasoning v25's
typedefs_qualified used; snapshot_cache.py bumped its own cache version
22 → 23, since the same TU inputs now produce a different snapshot.
One non-obvious asymmetry, worth not rediscovering: typedef_entity_ids
joins qualified_name_segments._LAMBDA_IDENTITY_FIELDS and
constant_entity_ids deliberately does not. A sidecar's keys must be
renumbered exactly when its partner's are — typedefs_qualified is
rewritten by that walk, so its sidecar must be; constants is excluded
from it (its values are payload literals a closure marker could legitimately
appear in), so renumbering only its sidecar is what would desynchronize
the pair. The reasoning lives on the two AbiSnapshot fields themselves,
since qualified_name_segments.py had one line of headroom against the
architecture gate's 800-line production maximum.
Consumers: diff_types._diff_typedefs (all three of TYPEDEF_REMOVED/
TYPEDEF_BASE_CHANGED/TYPEDEF_VERSION_SENTINEL) and
diff_symbols._diff_constants (CONSTANT_REMOVED/CONSTANT_CHANGED/
CONSTANT_ADDED), old-side-preferred with a new-side fallback, the same
convention every slice above uses, and .get-based so an absent sidecar
degrades to today's identity-less Change rather than fabricating one from
a flat key.
Tests: tests/test_entity_id_carrier.py gained TestSidecarFieldShape
(dataclass contract plus the partner-key-space property),
TestSidecarIsPersisted (round trip, a Record("ns")-vs-Namespace("ns")
counterexample the wire form must keep apart, and a pre-v31 snapshot loading
with both sidecars empty and its partner dicts untouched), a typedef
assertion inside the shared _assert_probe_identities both live backends
run, and two new live-backend constant tests
(test_live_{clang,castxml}_populates_the_constant_sidecar, split out
because parse_constants is provenance-gated and needs a parser configured
with a real public-header set). tests/test_change_entity_id_carrier.py
gained TestTypedefAndConstantSidecarsReachTheChange, covering all five
reachable finding kinds plus the absent-sidecar degradation.
Verification (all run, not assumed): ruff check abicheck/ tests/ clean;
mypy abicheck/ clean (0 errors, 562 files); python
scripts/check_ai_readiness.py 0 errors (doc-count-sync caught the schema
bump against docs/reference/snapshot-format.md, which was updated in the
same pass); python scripts/check_architecture.py back to the single
pre-existing cli_buildsource_helpers.py debt-no-growth finding;
pytest tests/test_entity_id_carrier.py 47 passed and its
-m integration lane 12 passed against real castxml 0.7.0 and clang,
including both new constant-sidecar tests; tests/test_change_entity_id_
carrier.py 32 passed; the full fast unit lane green apart from one
pre-existing, unrelated failure
(test_action_run_sh_baseline_set_fallback.py::TestBaselineSetFallback::
test_extraction_and_resolution_work_when_python3_is_absent, confirmed to
fail identically on the base commit under git stash). ruff format
--check fails broadly on this tree (580 files at the base commit), a
pre-existing condition; every file this slice touched is either
format-clean or was already unformatted before it.
Six architecture/debt.yaml no-growth baselines were raised, each by the
exact plumbing this slice adds and each with its own rationale line, in the
same style and for the same reason commit 09de29f raised
cli_scan.py/scan_engine.py: dumper.py 1985 → 1991 (two sidecar
keywords at each of three AbiSnapshot construction sites),
dumper_hybrid.py 979 → 989, serialization.py 1737 → 1747 (call-site
plumbing only — the codec itself lives in storage/), tu_merge.py
1552 → 1566, and on the test side tests/test_dumper_manifest.py
1765 → 1771 (its stub parser gained the two new methods) and
tests/test_tu_merge.py 1888 → 1890. No smaller diff was found that still
populates both fields on every producing path without hiding the field names
behind a spread helper. qualified_name_segments.py was the one file held
under its cap rather than raised — it had a single line of headroom against
the 800-line production maximum, which is why the reasoning for its one-line
addition lives on the AbiSnapshot fields instead of beside the tuple entry.
Three test stubs and one generated-fixture set needed updating, all
mechanically: tests/test_dumper_manifest.py/tests/test_dumper_phase1.py's
fake header-AST parsers gained the two new methods (they already implemented
parse_typedefs_qualified/parse_constants for the same reason),
tests/test_tu_merge.py's empty-merge expected value gained the two
keywords, and python scripts/gen_g20_fixtures.py re-emitted the eleven
examples/case*/…abi.json snapshots whose schema_version and field set
this bump changes (both new keys serialize as {} there — none of those
cases has header-resolved typedef/constant identity).
Still genuinely deferred after this slice — exactly two items, unchanged
by it. (1) Every DWARF/PE/Mach-O/ELF-symbol-table-only detector
(diff_platform.py's DWARF-tier functions, diff_elf_layout.py,
diff_platform_elf_dynamic.py, diff_platform_elf_symbols.py,
diff_versioning.py, diff_sycl.py) still has no EntityId-carrying
object behind it, because only the header-AST (L2) backends resolve one;
closing that needs EntityId extended to the DWARF backend (a ScopePath
built from a DIE's parent chain), a separate extraction-side project, not a
diff-site wiring pass. This slice deliberately did not touch
dwarf_snapshot.py/dwarf_metadata.py/dwarf_unified.py: a DWARF-only
snapshot leaves both new sidecars empty, exactly as it already leaves
typedefs_qualified. (2) Promoting the entity: alias in
finding_identity.resolve_change_identity into a real alias-match
reconciliation tier stays blocked on EntityId.key's cross-release
stability, which is not established — Anonymous/LocalToFunction
ordinals are stable only within one process's lifetime, and two prior
attempts at ordinal stability were each designed and reverted (see
model/identity.py's own docstring). Neither was attempted here.
buildsource/*.py's L5 source-graph sites remain out of ADR-063's scope by
design, as before.
Landed (fourteenth slice, 2026-09-02): item (1)'s extraction-side half,
for DWARF and the ELF-symbol-only fallback — the object each of them
carries, not the detector wiring on top of it. dwarf_snapshot.py's DIE
walk threads a typed ScopePath alongside the pre-existing flat scope
string, built at the exact points a DW_TAG_namespace/DW_TAG_structure_
type/DW_TAG_class_type/DW_TAG_union_type DIE is entered — the same
"widen the walker's own scope representation, don't reconstruct it from a
flattened spelling" fix this phase's own Design section required for the
two header-AST backends. RecordType/EnumType/Function/Variable
gain a populated entity_id, and a new typedef_entity_ids sidecar on the
builder (wired into AbiSnapshot.typedef_entity_ids, keyed identically to
typedefs) mirrors the two header-AST backends' own — AbiSnapshot.
constant_entity_ids stays empty for a DWARF-only snapshot, since a
constexpr initializer is a header-AST-only fact DWARF does not carry.
dumper_elf_fallback.py's header-less, export-table-only Function/
Variable construction gets the same treatment with an empty ScopePath
(a raw exported symbol carries no scope information at all) — the case
model/identity.py's own docstring already named as needing this
mangled_name/leaf_name interaction (entity_id_for_variable's
mangled=sym reused for both fields, so the mangled branch drops
leaf_name rather than let a header/DWARF observation of the identical
symbol fail to merge with it). The DWARF-specific construction (the
mangled-name-vs-extern-C gate, the two scope-segment builders) lives in a
new leaf module, abicheck/extract/dwarf_scope.py, under extract/ per
ADR-061 D9's task routing even though its one caller, dwarf_snapshot.py,
is itself still a legacy root module; tests/test_entity_id_carrier.py's
_ALLOWED_RESOLVER_CALLERS was extended to name it (and
dumper_elf_fallback.py) as legitimate entity_id_for_* front doors,
alongside the two header-AST backends. abicheck/extract/dwarf_records.py
(four small, dependency-free DWARF record/access helpers, previously
inline in dwarf_snapshot.py) was split out in the same slice purely to
keep that file under its architecture/debt.yaml no-growth baseline —
unrelated to identity, a byproduct of "move responsibility instead of
raising the baseline."
Anonymous DWARF scopes get no Anonymous segment in this slice: an
anonymous namespace (namespace { ... }, no DW_AT_name) contributes no
scope segment at all, matching this walker's own pre-existing flat-scope
behavior (its members were already treated as declared directly in the
parent scope); an anonymous struct/union/enum DIE is skipped by
dwarf_snapshot.py's own pre-existing logic unless reached through a
typedef (typedef struct { ... } Point;), which borrows the typedef's own
name — so no bare-unnamed record/enum ever reaches this slice's
entity_id_for_type/entity_id_for_enum call sites at all. This is
narrower than the two header-AST backends (which do construct Anonymous
segments, ordinal-tracked per parent), a documented gap rather than a
silent one: DWARF exposes no per-block merge signal for a reopened
anonymous namespace the way clang's originalNamespace/previousDecl
does, so extending this slice to attempt one was not pursued. A method's
own cv-qualification, ref-qualifier, and variadic-ness are similarly not
read from DWARF here (unlike the two header-AST backends' own AST-node
reads) — inert in practice, since they only feed entity_id_for_function's
"sig" fallback branch, reached only by a non-extern "C" function with no
real DW_AT_linkage_name at all; every ordinarily-mangled C++ overload
takes the mangled branch instead and never touches them.
Update (2026-09-02, landed): PE/Mach-O extraction-side entity_id.
dumper.py's own header-less _dump_macho/_dump_pe export-table-only
branches now populate entity_id too, via a new shared leaf module,
abicheck/extract/export_symbol_identity.py, that also absorbs
dumper_elf_fallback.py's own construction (one shared builder for all
three export-table-only producers instead of three drifting copies). The
with-headers path for both formats already ran through the same
header-AST parser ELF uses and so already carried entity_id before this
slice — undocumented until now, not new. export_symbol_identity.py's PE
helper recognizes both the MSVC ? and Itanium _Z mangling prefixes (a
MinGW/GCC-built PE DLL's C++ exports are Itanium-mangled, a real, distinct
supported PE lane from MSVC's own), so a MinGW DLL's headerless and
header-backed dumps agree on the mangled branch instead of the headerless
path silently falling back to extern-"C" for every MinGW C++ export. This
closes item (1)'s remaining extraction-side gap in full: every
DWARF/PE/Mach-O/ELF-symbol-table-only producer now populates entity_id.
Still open after this slice, updated by a follow-up pass: the diff-site
wiring pass was carried out module by module, with a deliberately narrow
scope — populate Change.entity_id only where a producer already resolves
a real matched Function/Variable/RecordType object, never invent a
lookup where the detector only ever sees a raw string or a DWARF-only
layout struct. That landed three real sites: diff_platform.py's
ELF-fallback deleted-function finding, and
diff_platform_elf_symbols.py's exported-object size/alignment findings
(both via AbiSnapshot.var_by_mangled). Every other call site across
diff_platform.py, diff_elf_layout.py, diff_platform_elf_dynamic.py,
diff_platform_elf_symbols.py, diff_versioning.py, and diff_sycl.py
was individually reviewed and left unwired for one of three structural
reasons, not because the wiring pass stopped short of them: (a) the
detector only has raw ELF/PE/Mach-O container facts in scope
(ElfMetadata/ElfSymbol/PE export strings), which carry no entity_id
field at all — the large majority of sites, including every one of
diff_platform_elf_dynamic.py's 40 and diff_versioning.py's 7 and
diff_sycl.py's 9 (the latter two operating on SyclMetadata/
SyclPluginInfo, equally plain); (b) the detector matches against
DWARF-only StructLayout/FieldInfo/EnumInfo (model/dwarf_facts.py),
which don't carry entity_id — only the header-layer RecordType/
EnumType do, a genuinely separate, deeper extraction-side gap from
"detector forgot to read entity_id"; this is diff_platform.py's
DWARF-tier layout functions and all of diff_elf_layout.py (pure L0/
binary-only, matching _ZTV<mangled>/_ZTI<mangled> vtable/RTTI symbol
strings with no RecordType ever in scope — its functions take only
old: AbiSnapshot, new: AbiSnapshot); or (c) the finding is
batch-shaped — aggregating multiple matched entities into one Change
(e.g. diff_platform_elf_dynamic.py's VISIBILITY_LEAK, which samples up
to five leaked Function names into one summary finding) — where this
codebase's own established convention (see finding_identity.py's
resolve_change_identity docstring) is that no producer sets entity_id
on a batch-shaped Change, since a single id cannot represent several
matched entities. diff_filtering.py/type_reachability.py's bespoke
string-suffix ambiguity trackers remain unmigrated, unattempted this slice
given that module's own extensive, multiply-reviewed fragility (see its own
module docstring) — a real rewrite risk this slice judged not worth taking
on without the same adversarial review rigor that code's prior fixes
received. Item (2), the entity: alias promotion, was investigated for a
third time this slice. The suppression-facing hash side still has no
accepted design and stays exactly as blocked as before: the same
Anonymous/LocalToFunction within-one-parse-only ordinal limitation
this phase's Design section already names blocks it identically for
DWARF as for the two header-AST backends (DWARF's own DIE walk assigns no
ordinals at all in this slice, so it inherits the identical gap rather
than closing it), and changing what report_canonical_finding_id hashes
for any already-shipped finding shape would silently invalidate a user's
existing stored finding_id: suppression rules — a backward-compatibility
call this phase's own discipline treats as needing explicit maintainer
sign-off, the same bar IMPORT_CYCLE_ALLOWLIST extensions are held to,
and deliberately not made unilaterally here.
What did land, as a genuinely new (not a third attempt at either
previously-reverted) design: model/identity_stability.py's
entity_id_is_cross_snapshot_stable(entity_id) — a gating predicate, not
a stabilizer. It makes no claim about the Anonymous/LocalToFunction
ordinal instability at all; it only lets a caller ask "does this
EntityId avoid the unstable construct in the first place" and refuse to
treat a match as authoritative when it doesn't (False whenever any
scope segment is Anonymous/LocalToFunction, or extra is the
anonymous-self marker). Split into its own leaf module purely to stay
under identity.py's 800-line production cap — see that module's own
docstring for the full design writeup, and
tests/test_identity_stability.py for the Hypothesis-driven
primitive-level property tests (AGENTS.md's own doctrine for a new
reusable predicate) stating its contract: true for every all-stable
scope regardless of shape/position, false for any single unstable
segment regardless of where it sits in the chain, false for the
anonymous-self extra marker independent of scope, and a pure
function of its input. This predicate is real, tested, safe-to-use
infrastructure — but it is deliberately not wired into
diff_filtering.py's _deduplicate_cross_detector (or anywhere else)
this slice: that file carries the identical "extensive, multiply-
reviewed fragility" this section's item (1) paragraph already declined
to touch for the same reason, and a promotion consumer belongs behind
the same adversarial review rigor that code's prior fixes received, not
bundled into the slice that first makes the primitive available. A
True result from this predicate is a necessary, not sufficient,
precondition for any future promotion — see the module's own docstring
for what a real consumer still has to establish beyond it.
Landed (fourteenth slice, 2026-09-03): the StableEntityId/
SnapshotLocalIdentity split, and the first two post-parse consumer
migrations (item (b)) — not all of item (b), and not the collision
narrowing. This is the slice the twelfth slice's own "Still genuinely
deferred" note and the thirteenth's identity_stability.py paragraph both
pointed at, taken in the shape the plan asks for rather than as another
attempt at a globally-stable ordinal (the two reverted designs stay
reverted; nothing here re-proposes a third).
The primitive. abicheck/model/identity_tiers.py (new leaf, depends only
on model.identity/model.identity_stability) defines two frozen types and
three constructors. StableEntityId wraps an EntityId that
entity_id_is_cross_snapshot_stable admits, and is only ever constructible
through stable_entity_id(), which answers None rather than wrapping one
that fails — so the type itself carries the guarantee and a holder needs no
second check. SnapshotLocalIdentity is keyed on a caller-chosen spelling
with the (possibly unstable) EntityId carried alongside as a
field(compare=False) payload: excluding it from equality is load-bearing,
since an entity whose ordinal shifted between two parses must still match
itself by spelling, which is the entire fallback this tier provides. The two
are distinct dataclasses and therefore never compare equal, so a set of
one tier can never be satisfied by a lookup in the other — a consumer wanting
both must state a precedence order explicitly, which is the decision this
module deliberately refuses to make on anyone's behalf.
tests/test_identity_tiers.py states that contract as Hypothesis properties
over the whole input space (the wrapper agrees with the predicate exactly for
every input; no unstable ordinal at any depth can enter the stable tier; the
anonymous-self extra marker is refused independent of scope; keys stay
injective and never collide across tiers; the payload never affects equality
or hash).
Consumer 1 — diff_filtering.py's opaque-type suppression. The bare
set[str] of RecordType.name this phase's Design section names by name is
gone, replaced by compare/opaque_types.OpaqueTypeIndex (relocated to its
ADR-061 owner rather than grown in place — diff_filtering.py sits on a
zero-slack debt.yaml no-growth pin, and the index is a matching concern,
not a filtering one). Every opaque declaration contributes to the index's
local tier keyed on the same RecordType.name spelling as before; one with
a stable EntityId additionally contributes to the stable tier.
intersect() is per-tier, and contains() consults stable first, then
spelling. That closes a real false negative: Change.symbol is rendered
bare on some paths and qualified on others while RecordType.name is bare on
the header backends and namespace-baked on DWARF, so a string compare could
miss a declaration both sides agree on — where two matching stable
EntityIds prove it.
What this consumer deliberately does NOT do, stated rather than implied: it
does not narrow. The bare-name collision (two unrelated types sharing a leaf
spelling in different scopes, one opaque, both suppressed) is still reachable
through the spelling tier. Making the stable tier authoritative — matching
only on it whenever the change carries one — would close that collision but
silently drop a real suppression whenever the two sides' producers disagree
about whether an identity was resolved at all (a mixed header-AST/DWARF
comparison), which is a live false-positive risk against the FP-rate gate.
Closing it needs stable-tier completeness on both sides first, which is its
own separately-reviewable step. The gap is pinned as an executable test
(tests/test_opaque_identity_tiers.py::TestKnownGapStaysDocumented) rather
than left as prose, so a later slice that closes it has to change an
assertion, not merely delete a paragraph.
Consumer 2 — type_reachability.py's closure walk. The six pieces of
record-tracking state inside _StdlibReferenceScan (_reached_records,
_worklist, _record_pending, _record_direct, _record_typedef_origins,
_record_walked) plus reach_record/mark_record_walked/
record_provenance/next_reached_record/_walk_reached_records's own
non_stdlib_records keys are now typed SnapshotLocalIdentity rather than
bare str. Behavior is bit-for-bit unchanged (equality on the wrapped
spelling is string equality); what changes is that the walk's node-key domain
is now named as valid-within-one-snapshot, and separated at the type level
from the spelling domain it sits next to. The conversion happens at one
boundary in _run_stdlib_reference_scan rather than by widening
_partition_snapshot_types's shared contract, since two of that helper's
three return values feed _spelling_index's substring machinery where an
opaque identity would be meaningless.
What was investigated and deliberately not migrated. The rest of
type_reachability.py/type_reachability_spelling.py — _record_identity's
three call sites, _spelling_index, _typedef_spelling_targets,
_namespace_suffix_spellings, _stripped_signature_spelling — stays on raw
strings, which is the identical conclusion the ninth slice reached and
recorded, re-verified here rather than taken on trust. Those strings are not
opaque identities: they must appear inside a rendered
Function.return_type/Param.type/TypeField.type, and mixing EntityId
keys into that domain breaks the matching algorithm for any type that happens
to carry a populated entity_id. Wrapping them in an identity type and
unwrapping at every use would add ceremony without closing a bug class. This
phase's Acceptance criteria ("their string-based helpers are deleted") is
therefore satisfied for the identity helpers and explicitly not for the
spelling helpers, which were never identity helpers in the first place.
Also still open, unchanged by this slice. Promoting the entity: alias in
finding_identity.resolve_change_identity into a real alias-match
reconciliation tier: StableEntityId makes the stability gate
un-forgettable, but it does not by itself establish that changing what
report_canonical_finding_id hashes is safe for a user's already-stored
finding_id: suppression rules — the same backward-compatibility call the
thirteenth slice declined to make unilaterally, and this slice does not make
either.
Verification. Full fast unit lane green; ruff check/mypy abicheck/
clean; python scripts/check_architecture.py 0 errors (both new pieces given
a real compare/ owner rather than raising a debt.yaml baseline — the
"move responsibility out" answer AGENTS.md prescribes);
python scripts/check_ai_readiness.py 0 errors;
python scripts/check_fp_rate.py 0 FP / 0 FN, both deltas 0;
python scripts/check_tier_accuracy.py OK (top-tier correct, under-call
monotonic).
Landed (2026-09-03): Phase 6B's first real checker cutover — the typedef
family, the legacy adapter, and the closing architecture gate. "PR 2"'s
first slice landed SemanticIRIndex deliberately with no live caller; this
slice gives it one, which is what turns the whole SemanticIR line of work
from "producers exist and are round-trip-tested" into "something a user's
verdict depends on actually reads it".
Why typedefs. Phase 6's own non-goal is not to keep widening producer
coverage until one full vertical slice is proven, so the first cohort had to
be one the IR already covers completely rather than one whose migration
would really be extraction work. Typedefs are the only family where the IR
carries exactly what the detector needs and nothing it does not: identity
(EntityId, resolved by both header-AST backends since the twelfth slice)
plus one payload fact (CanonicalEntity.canonical_spelling — for a typedef,
its resolved underlying type, precisely the value the detector compares).
Records carry layout facts the IR does not model; functions need a canonical
signature spelling whose cross-backend agreement is its own open question;
constants' payload is a value literal, not a type spelling.
The adapter, and why it was the actual blocker. A detector reading only
through the index would see nothing at all on a snapshot carrying no
SemanticIR (DWARF-only, PE-only, or any pre-v38 reload) — silently losing a
whole detector family. abicheck/model/semantic_ir_legacy_adapter.py removes
that by projecting the legacy typedef collections into a real SemanticIR,
so the same SemanticIRIndex reads both and a migrated detector cannot tell
which it was handed. Producing a real IR rather than a parallel "index-like"
type is the point: a second read shape would be a second thing to keep in
agreement with the first.
Fidelity gating rather than optimism. render_display_name() is the one
projection between the IR's EntityId keys and the legacy collections' flat
qualified keys, and it answers None — never a best-effort string — for an
identity whose scope contains an Anonymous/LocalToFunction segment, since
any string for those would be an invention and two distinct such declarations
would render alike. typedef_index_pair() then hands back the IR-backed
index only when its own rendered display-name key set exactly equals the
alias maps the comparison already resolved, on both sides; anything else
— an unrenderable anonymous scope, a producer that resolved identity for only
some typedefs, a DWARF-only side, a pre-v38 reload — falls back to the
adapter for both. Both-or-neither is not tidiness: pairing an IR-backed old
side with an adapted new side compares two differently-derived key spaces,
fabricating a removal or addition out of a projection difference. Set
equality is checked, never a count.
Synthetic identity is marked, not hidden. A legacy declaration whose
producer resolved no EntityId still needs one to key an occurrence by, so
the adapter derives one from the display spelling and tags it
SYNTHETIC_IDENTITY_EXTRA; producer_entity_id() answers None for it. The
detector stamps Change.entity_id only from that predicate, because
resolve_change_identity folds that field into an entity: alias real
stored suppression rules match against — passing the index's own bookkeeping
off as backend evidence would change which suppressions fire.
The detector. diff_types._diff_typedefs keeps only the comparison-level
half (stdlib-namespace exclusion, RD2-5's unconfirmed-removals flag, and
which alias map the pair trusts) and delegates to
abicheck/compare/typedefs.py, which reads only through the index. One
behavior detail worth not rediscovering: extract/semantic_normalizer.py
records an unfollowable typedef chain as Fact.failed(...) where the legacy
path carried the literal "?", so the migrated detector maps a non-present
spelling back onto that same placeholder — otherwise an
unresolved-vs-unresolved pair would stop reading as "unchanged".
_is_version_stamped_typedef moved with the family (its regex copied
verbatim, not re-derived) and is re-exported from diff_types under its
original private name so checker.py and
tests/test_typedef_version_sentinel.py are unaffected.
The closing gate. scripts/semantic_ir_cutover.py (semantic-ir-cutover,
wired into check_ai_readiness.py) is a real AST scan forbidding a migrated
cohort's modules from reading the legacy collections they were migrated off —
direct attribute access, getattr(obj, "name"), and a resolved getattr
alias alike, while never flagging a same-named local or an inbound
typedefs= keyword. Deliberately not an allowlist-and-shrink baseline
like KNOWN_UNMIGRATED_READERS: a cohort is only added at the moment it is
migrated, so a grandfathered reader cannot exist, and there is no per-site
exemption. tests/test_typedef_cutover.py exercises the gate against real
source for every forbidden spelling and for the two shapes it must not
flag, rather than asserting its allowlist is empty — a gate that cannot be
shown to fail proves nothing.
What remains open, explicitly. Every other detector family still reads
the legacy collections directly, by design — each further cohort needs its
own MIGRATED_COHORTS entry and its own equivalence evidence.
SemanticIRIndex.references() is still unimplemented (it names a graph
traversal belonging with the public-surface reference index, ADR-063 D5 /
Phase 3), so this cohort does not use it, and the index's own accessor set is
otherwise unchanged by this slice. CanonicalEntity.template_arguments
remains unpopulated for function templates, and clang still produces no
occurrence for a template specialization — both unchanged here.
Verification. tests/test_typedef_cutover.py states the cutover's central
property as a Hypothesis equivalence over generated typedef map pairs (add/
remove/change/unchanged across bare, singly and doubly qualified aliases and
five underlying spellings including the unresolved placeholder): the same
comparison run through a real SemanticIR and through the adapter must
produce identical findings, with the adapter path — the pre-migration
behavior — as the oracle. Plus tests/test_semantic_ir_legacy_adapter.py's
property tests for render_display_name's round-trip and refusal contract
and the selector's both-or-neither rule. Full fast unit lane green;
ruff check/mypy abicheck/ clean; check_architecture.py 0 errors;
check_ai_readiness.py 0 errors; FP-rate gate 0 FP / 0 FN; tier-accuracy
gate OK.
Landed (2026-09-03): Phase 6B's second checker cutover — the constant family, reusing the typedef cohort's adapter and gate shape unchanged.
Why constants. The typedef cohort's own reasoning already ruled out
records (layout facts the IR does not yet model) and functions (a canonical
signature spelling whose cross-backend agreement is its own open question)
as the next cohort. Constants are what's left of the "IR already covers
this completely" set: extract/semantic_normalizer.py's fourth slice
already gave every public constant a real EntityId
(parse_constant_entity_ids(), Phase 2) plus exactly one payload fact —
its raw, deliberately-uncanonicalized value text
(CanonicalEntity.canonical_spelling, matching diff_symbols.
_diff_constants's own long-standing raw-string != comparison) — so, like
typedefs, migrating this family is a real read-path change, not extraction
work in disguise.
The shape, reused rather than re-derived. abicheck/compare/constants.py
is compare/typedefs.py's design with the constant collections substituted:
index-only reads (diff_constants), a fidelity-gated selector
(constant_index_pair) requiring the IR's rendered names/values/identities
to exactly reproduce AbiSnapshot.constants on both sides before trusting
it, both-or-neither on any divergence. abicheck/model/
semantic_ir_legacy_adapter.py gained one sibling function,
legacy_constant_ir, reusing render_display_name/producer_entity_id/
SYNTHETIC_IDENTITY_EXTRA unchanged — a constant's qualified name has the
same flat-spelling shape a typedef's alias does, so nothing about the
rendering or synthetic-identity story differs between the two families.
The one real difference: an injected reliability predicate, not a
snapshot-blind detector reading a new fact directly. Constants carry one
comparison-level suppression typedefs don't:
diff_default_value_reliability.
constant_value_fingerprint_comparison_unreliable declines a CONSTANT_
CHANGED verdict when either side's value is a pre-stabilization
direct-clang fingerprint that can't be trusted against a fresh one. That
function reads AbiSnapshot.ast_producer/clang_field_initializer_facts_
reliable — snapshot fields the migrated cohort is not forbidden from
reading — but keeping compare/constants.py snapshot-blind (matching
compare/typedefs.py's own discipline of taking only indexes and plain
values) meant injecting it as a predicate closed over both snapshots by the
caller (diff_symbols._diff_constants), the same reasoning that already
motivated diff_typedefs's own is_non_abi_surface_type injection.
A defensive floor with no reachable legacy sentinel. A typedef's
unresolved-chain placeholder ("?") is a real string both backends agree
on, so compare/typedefs._underlying can fall back to it defensively. A
constant's Fact.unsupported() occurrence (a clang compound-initializer
fingerprint or Python-bool-derived literal spelling) carries no such
placeholder — the raw fingerprint text isn't retained on the fact at all —
so compare/constants._value returns None instead. This is never reached
through the real constant_index_pair gate in practice: a None
projection can never equal the legacy raw string the fidelity gate compares
against, so any entity in this state already forces a fallback to the
adapter for both sides before a detector would iterate it. Exercised
directly in tests/test_constant_cutover.py as the defensive floor, not
the mechanism.
The closing gate. One more MIGRATED_COHORTS entry in
scripts/semantic_ir_cutover.py (constants, forbidding
AbiSnapshot.constants/constant_entity_ids reads from
abicheck/compare/constants.py) — no changes to the check itself, since the
same real AST scan already generalizes across cohorts by construction.
What remains open, explicitly. Same as the typedef cohort: bare-name-
collision narrowing and promoting the entity: alias into a real match
tier are both still out of scope, needing their own sign-off/completeness
work first (see the identity concept's own removal_gate in the status
ledger). Every family beyond typedefs and constants remains unmigrated.
Verification. tests/test_constant_cutover.py states the identical
Hypothesis equivalence property test_typedef_cutover.py does, substituted
for constants (add/remove/change/unchanged across five qualified-name
shapes and five value spellings) — the same comparison run through a real
SemanticIR and through the adapter must produce identical findings, with
the adapter path as the oracle — plus the gate-exercise tests proving
semantic_ir_cutover.py's scan actually fires on every forbidden read
shape for the new cohort. Plus TestLegacyConstantIr in
tests/test_semantic_ir_legacy_adapter.py, mirroring the typedef adapter's
own round-trip/fallback/sidecar-mismatch coverage. Full fast unit lane
green; ruff check/mypy abicheck/ clean; check_architecture.py 0
errors; check_ai_readiness.py 0 errors; check_docs_contract.py 0
errors.
The next two candidate cohorts (records, functions): investigated, declined for now (2026-09-03). The original cohort-1 landing text named both as blocked -- records on "the IR does not yet model layout facts", functions on "cross-backend signature-spelling agreement" being an open question. Checked both claims against the codebase as it stands today, rather than carried them forward unexamined, and found neither still holds as a blocker; what's missing instead is a bug to justify the migration cost, the same bar typedefs/constants/opaque-narrowing above each cleared and this pair does not.
Records. RecordType/TypeMap matching (diff_helpers.py, ADR-045,
predating ADR-063 entirely) is already qualified-name-keyed with an
ambiguity-safe bare-name alias for schema-evolution compatibility --
structurally the same shape render_display_name-based matching would
give a migrated reader, not the flat, unqualified alias map typedefs
carried before their own migration. And the simplest candidate layout
facts, size_bits/alignment_bits, are a direct, single-source
pass-through of the identical RecordType field a normalizer would read
to populate a SemanticIR fact for them -- there is no second value
source for those two projections to ever disagree on, unlike a rendered
type spelling (where two backends really can disagree). So "records
carry layout facts the IR does not yet model" was true but was not
itself the blocker: modeling them is straightforward (verified by design
during this investigation, not implemented, since doing so would close
nothing).
Functions. CanonicalEntity.canonical_spelling already resolves the
cross-backend signature-spelling agreement the original framing named as
open -- Phase 2's third slice landed it (the canonical
"<return>(<param>, ...)" spelling, built from the same
canonicalize_function_signature_param_type/canonicalize_type_name
primitives entity_id_for_function itself uses) after that framing was
written, and this investigation is what caught the plan text not having
caught up. But checked one level deeper, past the resolved-doc claim, for
whether the underlying spelling problem was ever actually reachable by
a real detector: it is not. diff_symbols.py's own return-type/
parameter-type comparisons already canonicalize via canonicalize_type_name
directly, with no dependency on SemanticIR at all, and function/variable
matching keys on the mangled name (resolve_function_identity's
CANONICAL tier) -- an unambiguous, already-qualified identity with no
bare-name collision analogous to the opaque-type one to close.
Conclusion. Declined on the identical basis 2B's entity: alias
promotion was (see that section's own note, above): not blocked on
missing infrastructure, but lacking a currently-identifiable finding to
justify the migration cost. Revisit either family the moment a concrete
cross-backend matching or spelling divergence surfaces that today's
TypeMap/mangled-name/canonicalize_type_name mechanisms cannot already
close -- at which point the infrastructure work this investigation
scoped (a size_bits/alignment_bits-shaped CanonicalEntity addition
for records; nothing further needed for functions, whose spelling is
already modeled) is the concrete next step, not a redesign.
Investigated (2026-09-11): a third checker cutover -- the function
family. Groundwork landed; the cohort itself is NOT registered as closed.
This revisits, but does not contradict, the "declined for now" note above:
that finding was about the SIGNATURE/matching-key problem specifically
(canonical_spelling's cross-backend agreement, bare-name-collision
matching), which correctly found no bug there to close. This investigation
targeted something else: the architectural invariant every other landed
cohort already satisfies (a migrated module never reads the legacy flat
collection directly), applied to the one part of _diff_functions that
might migrate without inventing new normalizer output -- the old/new
matching index itself (SymbolIdentityIndex.for_functions's
exact-mangled-name join plus its ambiguity-checked extern "C" name-alias
fallback).
Why it does not close. A function's identity has no legacy-vs-IR duality
to adjudicate the way a typedef's or constant's does: Function.entity_id
is resolved exactly once, at parse time, by every producer (DWARF, both
header-AST backends, and the ELF-fallback exporter alike), and
extract/semantic_normalizer.py's own third-slice docstring is explicit
that it "computes nothing about identity, only reads the entity_id each
backend already resolved" when building a real SemanticIR occurrence for
a function -- there is no second, independently-derived representation for
finding_identity.resolve_function_identity to prefer over the flat
Function object it already reads. A first draft of
abicheck/compare/functions.py's function_identity_index built a
SemanticIRIndex per comparison (real IR when the snapshot carried one
covering FUNCTION, else a new legacy_function_ir adapter projection)
and looked up each function's entity_id in it -- but discarded the
lookup's result, since the identity itself was still computed from the flat
object regardless. Review (Codex, PR #1224) correctly flagged this as not a
migration: a lookup whose result influences nothing cannot be
distinguished, by any test or mutation run, from not existing at all, and
registering it as a closed MIGRATED_COHORTS entry would have made the
semantic-ir-cutover gate pass while SemanticIR content still cannot
affect function matching -- false assurance for future work, exactly what
that gate exists to prevent.
What actually landed. function_identity_index is now an intentionally
thin wrapper -- behaviorally identical to SymbolIdentityIndex.for_functions
-- with no SemanticIR/adapter consumption to gate on, so functions is
not added to scripts/semantic_ir_cutover.py's MIGRATED_COHORTS.
What does land and stays, as tested groundwork with no live caller yet (the
same precedent SemanticIRIndex itself was accepted under -- "landed and
proven correct in isolation first"): abicheck/model/
semantic_ir_legacy_adapter.py's legacy_function_ir, a real SemanticIR
projection of the flat function map shaped like legacy_typedef_ir/
legacy_constant_ir, and the compare/functions.py module boundary itself
as the one place a future consumer belongs.
What a real cohort 3 would still need. CanonicalEntity growing
per-position payload facts a signature-level comparison could actually read
-- a separately-addressable return-type spelling, ref_qualifier, variadic
status (canonical_spelling today combines a whole signature into one
opaque "<return>(<param>, ...)" string, and the third slice's own
normalizer does not carry ref_qualifier/variadic status at all -- see
extract/semantic_normalizer.py's own "Deliberately excluded from this
slice" list). Only once such a fact exists does a consumer have something
real to read through the IR instead of the flat Function object, at which
point registering the cohort reflects an actual migration rather than an
architectural gesture with nothing behind it. Not attempted speculatively
here, per this phase's own non-goal against inventing normalizer output
with no detector ready to consume it.
Verification. tests/test_function_cutover.py proves
function_identity_index resolves identically to
SymbolIdentityIndex.for_functions regardless of what the snapshot's
SemanticIR looks like (absent, present-but-not-covering-FUNCTION,
fully/partially covering, or the function itself having no entity_id),
both directly and end-to-end through diff_symbols._diff_functions (via
checker.compare) for a removed function, an added function, the extern
"C" name-alias fallback, and an unchanged pair. legacy_function_ir is
tested directly (TestLegacyFunctionIr). Full fast unit lane green; ruff
check/ruff format --check/mypy abicheck/ clean;
check_architecture.py/check_ai_readiness.py/semantic_ir_cutover.py/
check_fp_rate.py/check_tier_accuracy.py all pass with zero regressions.
Phase 3 — public surface as a graph query over one evidence graph (D5)¶
Landed (thirteen slices, 2026-08-31): the plumbing, not the traversal
migration. Every piece of new infrastructure this section's design calls
for exists and is wired end to end: model/occurrence.py
(OccurrenceId/canonical_key); SurfaceGraphLike (model/graph_facts.py,
a structural Protocol with nodes/edges as read-only properties, not
plain attributes — required to satisfy mypy's covariance check against
SourceGraphSummary's mutable list fields); AbiSnapshot.surface_graph
(unconditional, schema v29) and its storage/surface_graph_codec.py
encode/decode pair; compare/surface_graph.py (the new compare/-layer
builder, build_public_surface_facts); policy/public_surface.py
(PublicSurfaceQuery, .resolve()/.resolve_public_domain()/
.resolve_export_domain()); surface_graph.py's (root module)
public_entity_ids threading through build_surface_graph()/
compute_surface_metrics(); the old_public_entity_ids/
new_public_entity_ids pair through pattern_verdicts.py/
diff_surface_metrics.py/checker.compare(); service.compare_snapshots()
resolving and forwarding the pair (which service_compare_pipeline.
classify_compare_pair inherits for free — see below); and
service_header_graph_attach._attach_header_graph() sharing one
SourceGraphSummary instance between AbiSnapshot.surface_graph and
AbiSnapshot.build_source.source_graph -- deliberately without also
populating compare/surface_graph.py's own facts onto it there, since a
first version that did regressed the header-graph attach-cost perf gate by
47-96% at realistic sizes (this builder runs unconditionally on
essentially every real dump, and nothing in this phase's own wiring reads
those facts back yet); build_public_surface_facts() stays available for
a caller that does need them to populate the same shared instance
explicitly.
Traversal migration landed (2026-09-01), for the public domain — the
"deliberately not landed" scope directly below is now stale for that half
and corrected here rather than silently rewritten (this section's own
established convention). surface.py's closure-walk implementation
(_index_surface_types/_seed_public_roots/_walk_type_closure/
_walk_exact_type_closure/_record_exact_identities/
_record_nested_in_known_record/_record_is_confirmed_public_seed, plus
the PublicSurface result type itself) moved to a leaf module pair,
policy/public_surface.py (the dataclass + _index_surface_types'
indexing/bookkeeping) and policy/public_surface_closure.py (the actual
walk, plus the real resolve_public_surface() entry point) — split purely
to stay under the 800-line new-file production cap, not a design split.
surface.py's own copy of every one of those functions is deleted, not
kept alongside; compute_public_surface() is now a thin wrapper delegating
to policy/public_surface_closure.py (re-exporting PublicSurface for
existing callers). export_surface.py's own root-seeding (the
contract=exports domain's export-table matching) is unchanged, but its
final type-closure step calls the same, now-migrated _walk_type_closure
it always reused "verbatim" from the header-domain implementation, so that
domain's closure became graph-native for free, without its own separate
migration. PublicSurfaceQuery itself moved to a third module,
policy/public_surface_query.py — required because it is the one place
that depends on both the closure module and export_surface.py at once,
and export_surface.py now depends on the closure module too, so
PublicSurfaceQuery.resolve_export_domain living inside that same module
would have closed a real, check_architecture.py-detected import cycle
(a static-analysis gate: a deferred, function-body-local import does not
escape it, only a genuine restructuring does).
The actual "traverse the graph" substitution is narrower than "operate on
the graph's node/edge set directly" would suggest, for a reason found only
once a first version of this migration shipped and a regression test
caught it: compare/surface_graph.py's own node ids can legitimately
collapse two distinct declarations onto one id (two overloads sharing a
demangled name with no mangled name and no resolved entity_id to tell
them apart is the concrete, tested case) — and naively trusting a
graph-node-keyed cache of "what does this id reference" for such a
collision let a public overload appear to reference a hidden sibling's
own private parameter type, which a contract_replay.py test designed to
prove the opposite (that the private type stays PROVEN_OUT_OF_CONTRACT)
correctly failed against. Unioning the colliding contributors — the
over-keep direction that is safe for an ambiguous type name — is not safe
for a declaration identity collision, because it attributes one
declaration's reach to an unrelated one rather than merely widening a
single declaration's own reach. Fixed by having compare/surface_graph.py
flag such a node (identifiers_collision) and having the query fall back
to recomputing that one declaration's own identifiers directly whenever the
flag is set (_referenced_identifiers_for_function/_for_variable/
_for_record in policy/public_surface_closure.py) — the pre-migration
behavior, preserved exactly for the one case the graph's shared-node model
cannot represent precisely. The declaration/type indexing itself
(_index_surface_types, ambiguous_type_names, origin_by_key) was
not re-pointed at the graph's node set for the identical structural
reason — see policy/public_surface.py's own module docstring.
A second correctness hazard was found by post-merge review (Codex, PR #979)
rather than caught before landing — and a more consequential one than it
first looked, since it hits the default dump path rather than an aged
on-disk format. service_header_graph_attach._attach_header_graph installs
an L5 surface_graph on essentially every real dump (G31 Phase A), but
deliberately never calls build_public_surface_facts itself — an earlier
measurement found paying that per-declaration walk on every dump
regressed the header-graph-attach-cost perf gate 47-96% at realistic sizes,
so that attach site's own comment always said populating it "is deferred to
whichever later phase actually queries the graph." resolve_surface_graph_
nodes()'s graph is None check alone never triggered a rebuild for such an
already-attached graph, so it was trusted as-is and every lookup against its
un-stamped nodes silently read as "references nothing," collapsing the
transitive closure on the ordinary, default --scope-public-headers path —
not a stale-schema corner case, the common one. (A first fix attempt reached
for the file's own established schema-version-gated reliability-flag
pattern, e.g. header_cv_facts_reliable — that closes only the narrower
"persisted pre-migration snapshot" shape of this same defect and does
nothing for a fresh, never-serialized snapshot straight out of
_attach_header_graph, since such an object's flag would default True
with no schema version to gate on; reverted before landing once the
broader defect was understood.) Fixed at the actual deferral point instead:
resolve_surface_graph_nodes() now always calls build_public_surface_facts
on the resolved graph, not only when it was None. That call is idempotent
and evidence-preserving (SourceGraphSummary.add_node/add_edge merge a
second registration's facts rather than replacing them), so it enriches an
already-attached graph's existing nodes in place — never discarding
_attach_header_graph's own L5 edges/facts — and this is the "later
phase" that attach site's docstring always deferred the cost to, so no new
per-dump cost is introduced; the cost only lands when a caller actually
resolves a public/export-domain surface, exactly as before this traversal
existed at all. TestUnpopulatedAttachedGraphIsBackfilled and
TestStrippedGraphAttrsAreReconstructedNotTrusted in
tests/test_policy_public_surface.py cover, respectively, the real
_attach_header_graph shape (a graph attached but never run through
build_public_surface_facts) and a stripped-attrs shape, each confirming a
type only transitively reachable through such a graph survives.
Superseded (2026-09-01) by a third round that removed the graph from
this computation entirely, rather than trying to make trusting it safe —
the paragraph above is preserved for its own history, but its
resolve_surface_graph_nodes()-enrichment design is no longer what ships.
A second Codex security review found that even the enrichment fix above
was not sufficient: GraphNode.attrs are derived through
model.graph_facts' cross-producer evidence-merge machinery, which
resolves a same-key disagreement between two registrations by
confidence/producer/content precedence — correct for genuinely independent
producer facts, wrong for referenced_identifiers/identifiers_collision
specifically, since they have exactly one legitimate source (the
snapshot's own current declarations) and no legitimate second producer to
reconcile evidence with. A schema-v29 or otherwise untrusted/adversarial
snapshot could carry a stale or crafted referenced_identifiers fact at a
confidence this module's own freshly-registered fact (always
CONF_UNKNOWN, the lowest rank) cannot outrank, so the enrichment fix's
own fresh, correct recomputation could still silently lose to a poisoned
persisted value — the identical collapsed-closure failure mode as the
paragraph above, reached through its own fix instead of around it. This
also explains the real, separately-measured performance regression against
scripts/benchmark_scaling.py's "Baseline regression (PR vs base)" gate
that resolve_surface_graph_nodes()'s unconditional enrichment introduced:
building real GraphNode/GraphFact objects for every declaration, twice
per compare (once per side, since checker.compare() defaults
scope_to_public_surface=True), carries meaningfully more overhead than
the deleted regex-based re-parse it replaced. An identity-keyed cache was
tried to close that regression specifically and reverted: it broke
tests/test_export_surface.py::TestUnresolvedTypeEdges::
test_a_scope_lost_alias_key_is_followed_to_its_target, which mutates a
snapshot's typedefs/types in place between two calls and correctly
expects the second to see the new content — an identity-keyed cache
silently served the stale first result instead.
The actual fix needed no merge-precedence override and no cache:
compare/surface_graph.py's referenced_identifiers_by_node() (renamed
public, alongside its ReferencedIdentifiers return type) was already a
pure function of the snapshot's own declarations, computed before any
GraphNode is built. policy/public_surface_closure.py and
export_surface.py's closure-walk entry points now call it directly and
thread the result through (_referenced_identifiers/
_node_identifiers_or_collision/_seed_public_roots/_walk_type_closure/
_walk_exact_type_closure all take a ReferencedIdentifiers now, not a
dict[str, GraphNode]), never touching snap.surface_graph or
GraphNode.attrs at all. resolve_surface_graph_nodes() had no remaining
caller once both sites switched and was deleted rather than kept as unused
surface — along with its own regression tests, replaced by
TestClosureIgnoresSurfaceGraphEntirely (three cases: no graph at all, an
empty attached graph, and a deliberately adversarial high-confidence
poisoned fact) and TestResolvePublicSurfaceIsNotIdentityCached in
tests/test_policy_public_surface.py. Removing the graph from the
computation also removed essentially all of the GraphNode/GraphFact
construction cost from the hot path as a direct consequence — an ad hoc
local re-run of scripts/benchmark_scaling.py after this change showed
the previously-regressed scenarios back in line with the pre-migration
baseline, confirmed by CI itself: the Performance workflow's own
"Baseline regression (PR vs base)" job (PR #979, commit 5544540)
completed with conclusion: success, the gate's own noise-controlled
PR-vs-base measurement.
Deliberately still not migrated: export_surface.py's own root-seeding
(the export-table-matching logic itself, as opposed to the type-closure
step it shares with the header domain) and
type_reachability.directly_referenced_stdlib_types() (unchanged reason:
reclassifying type_reachability.py into policy would introduce a
genuine new policy -> extract violation), and compare/surface_graph.py's
node-id-namespace unification with buildsource/header_graph.py's L5 ids
remains open, exactly as the paragraph below already described. FP-rate
gate and per-tier accuracy gate both show zero regression against this
migration.
Deliberately not landed (original text below, now correct only for
the residuals named above — the "not deleted, delegates to both unchanged"
framing no longer describes the public domain; see the corrected paragraph
just above). surface.py's closure-walk traversal and
export_surface.py's independent one are not deleted —
PublicSurfaceQuery delegates to both unchanged rather than reimplementing
either as a literal graph traversal. This was a scoped, documented risk
decision: both algorithms are exactly the kind of intricate, multi-round-
corrected logic this plan's own "Primitive-level property tests" AGENTS.md
section warns cost six review rounds to get right once already (the
_paired_stable_indices incident) — reimplementing either from scratch as
a graph BFS, inside the same phase that also had to build every piece of
graph infrastructure underneath it, was judged a materially higher-risk
combination than landing the infrastructure now and migrating the
algorithm itself as a separate, later, narrowly-scoped phase. Consequently:
there is no lazy, graph-reading legacy-snapshot backfill path (compute_
public_surface()'s signature was never changed to accept a structured
resolution parameter, since nothing inside it reads the graph yet);
type_reachability.directly_referenced_stdlib_types() was not migrated
into policy/public_surface.py (doing so would require reclassifying
type_reachability.py into the policy layer, which would introduce a
genuine new policy -> extract architecture violation — that module
imports two already-extract-classified siblings); and compare/
surface_graph.py's own node ids (canonical_key(occurrence_id)/
approx::/typedef:: fallbacks) do not unify with buildsource/
header_graph.py's pre-existing L5 node ids (decl://<identity>/
type://<identity>) — the two id namespaces coexist in one shared graph
instance (real, tested) without deduping onto a common node for a
declaration both builders see. surface.py's own traversal deletion and
the node-id unification are both real, separate follow-up phases, not
silently-abandoned scope — see compare/surface_graph.py's and
policy/public_surface.py's own module docstrings for the exact reasoning
each carries.
Two corrections to this section's own design text below, found while
implementing it rather than assumed from the prose: the "real, working
identity checked against architecture/modules.yaml, not assumed" bullet
originally analyzed surface.py/export_surface.py/pattern_verdicts.py
as "still flat, top-level modules" for the policy -> compare layering
question — checked against the real architecture/modules.yaml during
implementation, all three (and several siblings) are already classified
into the policy layer via legacy_paths, an unrelated prior
classification pass this section's text predates, which simplified the
actual implementation (no circular-import risk to route around) rather
than complicating it. And the "service.compare_snapshots() ... the
second, documented Tier-2 production verb that also calls checker.
compare() directly" framing below assumed service_compare_pipeline.
classify_compare_pair was a third independent call site needing its own
resolution wiring; reading the real code, classify_compare_pair already
calls service.compare_snapshots() (never checker.compare() directly),
so wiring compare_snapshots() alone was sufficient to cover both
production Tier-2 paths.
Goal. compute_public_surface() answers "is this declaration public"
by traversing one authoritative evidence graph, not by independently
reconstructing include/reference/export relationships from the flat
snapshot a second time.
Design. Two things were wrong with this phase's first draft, both
caught by review, and both point to the same corrected design. First,
"is this declaration public" is a relevance decision — AGENTS.md's own
task-routing table assigns exactly that class of question ("decide
relevance, suppression, classification, severity, or gating") to
policy/, not to compare/ ("match old/new entities or identify a raw
change"); putting the decision itself in compare/ would make that
package own policy behavior. Second, and more fundamentally: this
repository already has a general-purpose, producer-agnostic node/edge
graph primitive with an evidence-preserving merge —
buildsource.graph_facts.GraphNode/GraphEdge/GraphFact/
merge_graph_facts (ADR-031 D2, ADR-046 D1/D2), currently used to build
the optional L5 source/build-evidence graph (buildsource.source_graph.
SourceGraphSummary, NODE_KINDS/EDGE_KINDS). A first draft of this
phase defined a second, parallel node/edge dataclass hierarchy in a new
compare/surface_graph.py for the public-surface graph — which is exactly
the "one concept, two representations" defect the Governing Invariant
forbids, and it would have left public-surface relevance and L5 impact
analysis (ADR-057's consumer graph, ADR-053's TU→link-unit→DSO
attribution) looking at the same declaration through two graphs that can
still disagree, the opposite of this phase's own stated goal.
The corrected design reuses the existing primitive rather than adding a sibling:
- Relocate the generic node/edge/merge primitive (
GraphNode,GraphEdge,GraphFact,FactConflict,merge_graph_facts) frombuildsource/graph_facts.pyintoabicheck/model/graph.py. This is exactly what ADR-061's own task-routing table says belongs inmodel/("add an ABI entity/value shared across stages") — the primitive itself is already producer-agnostic and was never actually specific to L5 evidence; only its vocabulary (NODE_KINDS/EDGE_KINDS) and its construction from source/build evidence are.buildsource/ graph_facts.py/source_graph.pyimport and re-export from the new location (mirroring the re-export shimsource_graph.pyalready uses for its own split-out pieces), so every existing L5 caller is unaffected.
Already landed independently, under a different name, before this
phase's own implementation began — confirmed by reading
abicheck/buildsource/CLAUDE.md's current module table and
abicheck/model/graph_facts.py directly, not assumed from this plan's
own stale text. ADR-061 Phase 5 item 2's own follow-up split
buildsource/graph_facts.py into three sibling model/ modules rather
than the single model/graph.py this bullet names:
model.graph_facts (GraphNode/GraphEdge/GraphFact/FactConflict/
merge_graph_facts/ensure_facts_and_resolve/register_fact/
edge_relation_key/edge_occurrence_id), model.graph_identity (node-id
construction/normalization), and model.graph_vocabulary (the
NODE_KINDS/EDGE_KINDS-family constant blocks) — buildsource/
graph_facts.py is now itself the back-compat re-export shim this
bullet asked for, already exporting every original name (X as X)
rather than only what a repo-local usage scan could prove was called.
GraphNode.id: str/label: str = "" match this section's own later
design exactly (confirmed by reading the dataclass directly). This
bullet's own remaining, still-open work is therefore not the relocation
itself but picking model.graph_facts (the real current location) over
the stale model.graph name wherever a later slice writes
from abicheck.model.graph import ....
Applies to every other model/graph.py reference below in this same
Phase 3 section, not only this bullet's own text (Codex review on
PR #958, catching that the correction above fixed only the one place it
was written and left the phase's own "Files" checklist and Phase 10's
cleanup-list row still instructing a future implementer to create a new
model/graph.py and migrate callers to it — actionable checklist items
a reader follows literally, unlike this bullet's own prose). Read every
model/graph.py elsewhere in this Phase 3 section, its own "Files" list,
and Phase 10's cleanup list below as model.graph_facts (for
GraphNode/GraphEdge/GraphFact/FactConflict/merge_graph_facts/
ensure_facts_and_resolve/register_fact/edge_relation_key/
edge_occurrence_id), model.graph_identity (node-id construction/
normalization, including _normalize_if_decl_or_type), or
model.graph_vocabulary (NODE_KINDS/EDGE_KINDS-family constants) as
the symbol requires — three already-existing modules, not one new file
to create. The "Files" section's own abicheck/model/graph.py (new...)
entry and its full-dependency-closure reasoning (which symbols must move
together) still correctly states what a from-scratch relocation would
need; only "new" and the single-file target are stale — every symbol it
names already lives in one of the three modules above, so nothing there
needs moving again. Phase 10's cleanup-list row is even further stale in
a second way: it describes buildsource/graph_facts.py as still holding
"the original, in-place copies," but that file is already the trimmed
re-export shim (confirmed above) — its own residual is narrower than
written, just "every real importer of the shim migrates to
model.graph_facts/graph_identity/graph_vocabulary directly." Not
the "five named readers" the very next, separate row lists (Codex
review on PR #958, catching a conflation in an earlier revision of this
same correction) — those five (internal_leak.py, buildsource/
crosscheck.py, buildsource/evidence_report.py, evidence_depth.py,
cli_graph.py) are that other row's own BuildSourcePack.source_graph
→AbiSnapshot.surface_graph migration, a different attribute path on a
different object; this row's own real importer set was never enumerated
in the original text. Confirmed here by resolving every relative
ImportFrom node's actual target module via a real AST walk, not a
text grep for the string graph_facts — a second review round on
this same commit correctly caught that the grep-based first attempt
counted a bare from .graph_facts import ... in model/source_graph.py/
model/entity_resolver.py/model/entity_identity.py as the shim
(Python resolves that relative import within model/ itself, to the
already-canonical model.graph_facts — no migration needed), likewise
buildsource/source_graph_findings.py's from ..model.graph_facts
import GraphEdge (already canonical), and a bare code-comment mention
in checker_types.py with no import at all. The real, resolved set is
ten files: buildsource/archive_graph.py, buildsource/callback_graph.py,
buildsource/graph_impact.py, buildsource/macro_graph.py,
buildsource/template_graph.py, buildsource/type_graph.py,
buildsource/virtual_dispatch_graph.py, impact/consumer_graph.py,
impact/use_cases.py, and internal_leak.py (also one of the other
row's five, since one module can need both migrations independently).
- A third, pre-existing graph-shaped module already answers a
public-surface question independently, and a first draft of this phase
missed it entirely — abicheck/surface_graph.py's SurfaceGraph/
build_surface_graph(), with real production consumers in idioms.py,
pattern_verdicts.py, and diff_surface_metrics.py (ADR-025's A1-A4
surface-intelligence features). Naming the new module
compare/surface_graph.py below, without addressing this one, would
leave two same-named-in-spirit graph modules in the codebase — exactly
the outcome the Governing Invariant forbids. Worse than the name
collision: SurfaceGraph.public_roots() computes "what's public" by
filtering Visibility.PUBLIC directly off the flat snapshot, which is
not the same answer surface.py's reachability closure (and this
phase's PublicSurfaceQuery.resolve()) computes — a declaration tagged
Visibility.PUBLIC but unreachable through real header inclusion, or
vice versa, is exactly the disagreement this phase exists to close for
compute_public_surface(), and SurfaceGraph has had its own,
independent version of that same risk the whole time.
Two things were wrong with the first fix for this, both caught by
review, and both point at the same corrected shape. First,
SurfaceGraph.public_roots() calling PublicSurfaceQuery.resolve()
directly would make surface_graph.py — a comparison/index-layer
module (ADR-025's A1-A4 surface-intelligence substrate, the same role
idioms.py/pattern_verdicts.py/diff_surface_metrics.py already play)
— import policy/public_surface.py, reversing ADR-061's required
policy -> compare direction (policy/ is allowed to depend on
compare/, never the reverse). Second, PublicSurfaceQuery.resolve()
returns frozenset[EntityId] (per this phase's own primitive below),
while SurfaceGraph.public_roots() is documented and consumed today as
a frozenset[str] of symbol/mangled names — pattern_verdicts.py's
_recognise_create_destroy() passes each root straight into
re.Pattern[str].match(), which a bare delegation would hand an
EntityId object instead of a string and fail outright, not just
disagree in content.
The corrected shape fixes both at once by moving the decision, not the
call: the workflow/compare orchestration code that already calls
PublicSurfaceQuery.resolve() for compute_public_surface() (the same
assembly step this phase's earlier bullets already describe) resolves
the public EntityId set once and threads it down to
build_surface_graph/compute_surface_metrics — surface_graph.py
itself never imports policy/public_surface.py, it only receives an
already-resolved answer, same direction as every other policy ->
compare edge in this plan.
Threading it down means widening real call chains, not declaring the
three consumers unaffected — a first draft of this phase claimed the
latter and a reviewer checked the actual call sites and found it false.
pattern_verdicts.py:196-197 calls build_surface_graph(old)/
build_surface_graph(new) directly inside apply_pattern_verdicts(),
and surface_graph.py:371's own compute_surface_metrics() does the
same — neither receives anything from a policy-layer caller today, and
both are themselves reached from checker.py's own explicit call sites
(_apply_pattern_verdicts_step/_apply_surface_metrics, not the generic
detector registry, so widening their signatures doesn't touch dispatch
machinery).
A single call's worth of public_entity_ids is still the wrong
shape for the two-snapshot callers, and a first draft of this fix
threaded exactly one shared set to both — a review round correctly
traced the real call sites and found apply_pattern_verdicts()/
compute_surface_metrics() each build two
separate graphs, one per side (build_surface_graph(old) and
build_surface_graph(new)), not one. Old and new can genuinely have
different public reachability — a declaration added to, or removed from,
the public-header set between versions is exactly the kind of change
compare() exists to detect — so resolving one shared id set and handing
it to both sides' graph builds classifies one side using the other
side's surface, corrupting pattern modulation and surface-metric findings
for precisely the changes that matter most (a declaration crossing the
public/private line). The old/new pair belongs on the two-snapshot
callers, not on build_surface_graph/compute_surface_metrics
themselves — a further review round correctly found this paragraph's
own fix put the pair on the wrong functions: each of those two helpers
operates on exactly one snapshot per call, so giving either of them
both old_public_entity_ids/new_public_entity_ids leaves no
unambiguous way to route the pair for a single-snapshot invocation.
build_surface_graph/compute_surface_metrics each gain exactly
one optional parameter instead — public_entity_ids:
frozenset[EntityId] | None = None — and the old/new pair lives only on
their two-snapshot callers, apply_pattern_verdicts()/
compute_surface_metrics()'s own caller, each passing its own side's set
to its own matching call (build_surface_graph(old, public_entity_ids=
old_ids)/build_surface_graph(new, public_entity_ids=new_ids)), never
the same set to both. checker.py's
_apply_pattern_verdicts_step/_apply_surface_metrics both gain the
two-snapshot pair — received as an already-resolved compare()
parameter from its caller, not computed by compare() itself; see the
correction a few paragraphs down for why checker.compare() may not
call PublicSurfaceQuery.resolve() directly (the same compare ->
policy direction violation surface_graph.py's own fix above already
closed once, reappearing here at a second call site if compare()
resolved this itself) — and passed straight through to
apply_pattern_verdicts()/
diff_surface_metrics(), which pass the matching half of the pair
through as the single public_entity_ids argument to build_surface_
graph()/compute_surface_metrics()
in turn. When both are None (the only case possible outside
compare()'s own pipeline), SurfaceGraph.public_roots() falls
back to its pre-existing Visibility.PUBLIC filter — an explicit,
narrow, named residual for a caller this phase cannot reach, not a
second silent implementation competing with the real one.
That residual turned out to be reachable from production after all —
a review round found a second, documented entry point that calls
checker.compare() directly and is never resolved, the exact gap the
paragraph above claimed didn't exist. service.compare_snapshots()
is a real, documented Tier-2 production verb
(its own docstring: "Thin wrapper over the Tier-1 core... so that
front-ends never call the core directly") that forwards
pattern_verdicts/surface_metrics straight into compare() with no
resolved-ids parameter at all — confirmed by reading service.py's
compare_snapshots() directly, not assumed. Any caller reaching
compare() through this path (not only service_compare_pipeline.
classify_compare_pair's typed-pipeline path) with pattern_verdicts=
True/surface_metrics=True hits the Visibility.PUBLIC fallback named
above, producing findings that can genuinely differ from the equivalent
CLI/typed-pipeline comparison of the same two snapshots — not a
theoretical caller this phase cannot reach, but the second of exactly
two production routes into compare(). Fixed by giving
compare_snapshots() the identical per-side resolution
classify_compare_pair performs — calling the same resolve_public_
surface() wrapper for old/new independently before invoking
compare(), and passing the two resulting frozenset[EntityId] | None
through as the same two parameters compare() itself gains — rather
than inventing a second resolution path. service.py is not itself
gated by the ADR-061 compare -> policy direction restriction (it is a
flat, unmigrated module today, same as the other residuals this plan
already tracks under its architecture-boundary notes), so this call is
not a new violation; it is a second call site doing exactly what the
workflow layer's own resolution call already does. Every documented
production route into compare() — the typed pipeline and this direct
Tier-2 verb — now resolves and passes real, side-specific ids; the
Visibility.PUBLIC fallback is reachable only from a caller that
imports checker.compare() directly, bypassing both documented
entry points, which is exactly the ADR-037 D1/D10.1 violation the CLI-
contract gate already exists to catch.
PublicSurfaceQuery.
resolve()'s result is not already a function/variable-only set, and a
first draft of this phase's mapping step assumed it was — resolve()
traverses declares and type-reference edges, so it genuinely returns
record/enum/typedef EntityIds too (a public function's return type, a
public struct reachable from a public signature), which is correct for
compute_public_surface()'s own purpose (deciding what's public at all,
types included) but has no symbol/mangled-name spelling to map to for
SurfaceGraph.public_roots()'s specific contract — that function's own
docstring already states its root set is "Visibility.PUBLIC functions
and variables," never types. SurfaceGraph.
public_roots() therefore filters the received EntityId set to
kind in (FUNCTION, VARIABLE) before mapping — a type-kind id in the
resolved set is simply not part of this particular root set and is
dropped, not mapped-and-failed.
Mapping the resolved id back to its mangled spelling is the wrong
target, and a review round correctly traced why: public_roots()'s
existing contract is keyed on the plain declaration name, not the
mangled one, and every one of its own consumers already depends on
that. Reading surface_graph.py directly: public_roots() is
frozenset(self._root_seed_types), and _root_seed_types is built by
_build_root_seed_types() keyed on fn.name/var.name — the plain
Function/Variable display name, deliberately (its own docstring:
"C++ overloads share a demangled name... their seed sets are unioned"),
never the mangled spelling. reachable_types(root) looks root up in
that same dict directly. idioms.py's _recognise_create_destroy()
applies human-readable create_*/destroy_* regexes against these same
keys, which only make sense against a plain name — a mangled Itanium
symbol never matches those patterns at all. Returning _Z...-mangled
spellings from the resolved-id mapping would therefore either silently
disconnect every returned root from its own seed types (a root string
reachable_types/_root_seed_types has never heard of) or suppress
create/destroy pattern recognition outright — changing pattern-verdict
and surface-metric findings, not merely an internal representation
detail. Fixed by mapping each remaining EntityId back to the existing
Function.name/Variable.name spelling instead — the exact key
_root_seed_types/reachable_types/idioms.py's regexes already use
today, unchanged by this phase — not the mangled name; EntityId's
function variant carrying the mangled name in extra remains useful
for identity/matching purposes elsewhere in this plan, just not as
public_roots()'s own return value. This preserves its existing
frozenset[str] return type
and its existing consumers' string-based contract exactly. SurfaceGraph
itself
is not deleted or folded into model/graph.py — it answers a genuinely
different question from the evidence graph (a snapshot-local
declaration-reference index for surface-intelligence metrics, not a
multi-evidence-source relevance graph), and conflating the two into one
module would be the opposite error: forcing a real distinction into one
representation. What must be one representation is what counts as
public, not the index structure built for a different purpose on top
of it, and not the direction that decision travels in.
- Build the public-surface graph as instances of that same primitive,
not a new dataclass hierarchy — abicheck/compare/surface_graph.py
(new) registers its own node/edge kind vocabulary (header,
translation_unit, declaration, type, symbol, target; edge
kinds includes, declares, references, instantiates, exports,
owned_by_target — several of which already have an L5 analogue worth
reusing directly rather than renaming for its own sake: declares
is SOURCE_DECLARES, exports is BINARY_EXPORTS_SYMBOL,
owned_by_target is TARGET_HAS_SOURCE/TARGET_HAS_PUBLIC_HEADER),
built from facts the core L0-L2 extraction layer already produces (the
header origin/scoping data dumper_scoping.py reads, the export-table
data export_surface.py already computes for contract=exports, the
declaration/reference data type_reachability.py/surface.py each
independently reconstruct today) — available unconditionally, unlike
source_graph.py's graph, which only exists when L3-L5 evidence was
collected. Only declaration/type nodes are keyed by the EntityId
Phase 2 established — not every node kind. A first draft of this
phase said "nodes are keyed by EntityId" without qualification, which
overclaims: Phase 2's EntityId is specifically an ABI-declaration
identity (record/enum/typedef/function/variable/constant); header,
translation_unit, symbol, and target nodes are not ABI
declarations and have no natural EntityId form — GraphNode.id in
the real, existing buildsource/graph_facts.py/source_graph.py is
already a plain string with its own per-kind URI scheme for exactly
these non-declaration kinds (header://, source://, target://,
symbol://, ...), which this phase reuses unchanged rather than
replacing. A declaration/type node's id must be an injective
encoding of its EntityId, not the lossy flattened qualified_name
string — a first draft of this phase used the flattened string
directly, which is exactly the collision this plan's own Phase 2
section warns against two sections above ("Two domain EntityIds
whose ScopePaths differ only in segment kind... can render to the
identical qualified_name string"): SourceGraphSummary.add_node()
merges any two registrations sharing one id, so a record nested in a
record and the same names nested in a namespace would coalesce into one
graph node, mixing their GraphFacts and corrupting public-surface
reachability for both. This key must be built from model.identity's
own identity-only encoding, not storage/entity_ids.py's to_dto() —
a first draft of this phase pointed the graph key at the same {"kind":
..., "name": ..., ...} segment records the storage DTO encodes, and
review correctly caught that those two encodings answer different
questions and must not share one function. to_dto() is deliberately
a lossless, full-structure round trip — it preserves Record.access
as real payload, because storage wants to recover everything a
ScopePath carries, access included. But Record.__eq__/__hash__
(this phase's own ScopePath-identity section, above) deliberately
excludes access from identity — two EntityIds differing only in a
member's access level are the same identity by design, so that a real
access-level change reads as "this declaration changed," not "removed,
then added." A graph key built from the full-structure DTO encoding
would give those two equal EntityIds two different node ids, which
re-introduces this section's own target bug from the opposite direction:
instead of two different scopes colliding into one node, one same scope
would now silently split into two. The fix is a second, narrower
function, model.identity.canonical_key(entity_id) -> str, built only
from the fields each segment's own __eq__/__hash__ already uses (so
it is injective on identity, never on full structure) — used by both
model/graph.py's GraphNode.id for a declaration/type node and any
other consumer that needs a collision-free, equality-consistent key
(kind/leaf_name/extra plus each segment's identity-only fields).
Already landed, ahead of this phase, under a different name —
Phase 2's own resolve_change_identity-consumer slice added exactly
this contract as EntityId.key (a property, not a free function; a
local _packed/_segment_key length-prefixing scheme, independently
arriving at the identical "exclude every compare=False payload field"
rule this paragraph states), confirmed by reading model/identity.py
directly: it already excludes Record.access and is built from
scope/kind.value/leaf_name/extra only. Superseded by Phase 3's
actual landing, corrected here rather than left to mislead a future
reader (Codex review, PR #958): the internal-linkage static
collision this section raises next is real, so GraphNode.id for a
declaration/type node does not call entity_id.key directly —
model/occurrence.py's landed canonical_key(EntityId | OccurrenceId)
overload is what compare/surface_graph.py's node-id construction
actually calls, with an OccurrenceId, exactly as the "still collides
for a real, ordinary case" paragraph below this one specifies and
requires. EntityId.key alone stays the everyday case (an empty
disambiguator reduces canonical_key(occurrence_id) to it), but the
wrapper this paragraph once called "redundant" is the landed, correct
design, not a mistake to avoid. storage/entity_ids.py's to_dto()
stays the separate,
intentionally fuller encoding for its own persistence purpose —
GraphNode's own pre-existing label: str field (already documented as
"human-readable name/path") is where the flattened, lossy qualified_name
spelling belongs instead, exactly the role that field already plays for
the existing URI-scheme node kinds below. Every other kind keeps the
existing URI-scheme id (header://, source://, ...), which was never
the display spelling and was never at risk of this collision.
canonical_key(EntityId) alone still collides for a real, ordinary
case EntityId's own precision was never built to resolve, and a
reviewer correctly traced why: two internal-linkage (static)
functions in different translation units sharing the same scope, leaf
name, and signature — e.g. two files each defining a file-local
static void helper() — mangle to the identical Itanium symbol (a
mangled name carries no file/TU component), so EntityId's own
function-kind extra (mangled name, or the normalized-signature
fallback) cannot tell them apart either; both collapse to one
canonical_key, and SourceGraphSummary.add_node() merges their facts
and edges into a single node. This is not a new ambiguity this phase
introduces — it is the identical one ADR-046/048's existing L5 source-
graph identity (buildsource/entity_identity.py) was already built to
resolve, by preferring a compiler-provided USR (which does encode
enough context to disambiguate two same-named internal-linkage
declarations) over a bare mangled name. Losing that resolution the
moment declaration/type nodes key on canonical_key(EntityId) directly
would be a real regression for exactly the nodes the L5 builder
populates, not merely an unlikely edge case. The fix reuses a
mechanism this plan already designed for the adjacent "same identity,
genuinely different declaration" shape, rather than inventing a fourth
one: model/graph.py's GraphNode.id for a declaration/type node
is canonical_key(occurrence_id) — OccurrenceId, not bare EntityId
— where the disambiguator Phase 2 already defined for the ODR-duplicate/
incomplete-declaration case is populated, for this case, from the same
USR/TU-context signal entity_identity.py already prefers when the
underlying evidence carries one (L5 source evidence, which is exactly
when two internal-linkage declarations can coexist as distinct graph
nodes in the first place — a pure L0-L2 binary/header-only snapshot has
no TU-level view to distinguish them from either, the identical
structural limit the flat EntityId layer already accepts). A
declaration with a globally-unique identity at the EntityId level
(the overwhelming common case — anything with external linkage, or an
internal-linkage entity that merely happens not to collide) gets an
empty disambiguator, so canonical_key(occurrence_id) reduces to
exactly canonical_key(entity_id) for every node this finding doesn't
apply to, with no behavior change for them.
This phase is ordered after Phase 2 because the declaration/type half
needs it — the same dependency Phase 6 (SemanticIR) has on Phase 2 —
not because every node kind does.
Defining canonical_key(occurrence_id) does not, by itself, make the
new public-surface builder's node ids agree with the existing L5
builder's — and a review round correctly found this phase's text never
actually closes that gap. canonical_key() is specified above as
compare/surface_graph.py's own new encoding; meanwhile
buildsource/source_graph.py:1498,1533 and every sibling L5 module
(header_graph.py, call_graph.py, type_graph.py,
override_graph.py, macro_graph.py, template_graph.py,
callback_graph.py, graph_backends.py — twelve call sites across
eight files, by grep) still construct declaration/type node ids via
graph_facts._decl_node_id(identity)/_type_node_id(identity), which
predate this phase and have their own independent f"decl://
{_normalize_graph_identity(identity)}"/f"type://{...}" format. Two
independently-written formats cannot be relied on to agree string-for-
string for the same declaration merely because both are "collision-
free" in isolation, and nothing in this phase's text migrates those
twelve call sites — so handing both builders one shared
SourceGraphSummary instance (the fix two bullets below) reconciles
nothing: the public-surface builder's node for a given declaration and
the L5 builder's node for the identical declaration land under two
different ids, add_node()'s id-collision merge never triggers, and the
two representations sit side by side in one container without ever
reconciling.
A "move the function, have canonical_key delegate to it" fix was
tried here and is itself wrong, for a reason a further review round
caught precisely: relocating _decl_node_id/_type_node_id does not
make their inputs equal, and the inputs are where the real
incompatibility lives. canonical_key(entity_id) exists specifically
because a flattened qualified-name string is not injective on
EntityId identity — two domain EntityIds whose ScopePaths differ
only in segment kind (a record nested in a record vs. the same names
nested in a namespace) can render to the identical flattened string,
which is exactly the collision this phase's own surface_graph.py fix
(two sections up) was built to avoid by keying on the segments'
__eq__/__hash__ fields directly, not on a flattened rendering.
_decl_node_id/_type_node_id's own normalization
(_normalize_graph_identity) only ever strips a checkout-dependent
absolute path out of an anonymous/lambda marker — confirmed by reading
it — it carries no segment-kind information at all, because its input,
ent.identity(), never had any: every one of the twelve L5 call sites
computes a bare, already-flattened string with no ScopePath/kind
breakdown behind it. So relocating the two functions and defining
canonical_key() to call them does not produce one shared, injective
encoding — it produces exactly one of two bad outcomes: either
canonical_key()'s rendering stays flattened (matching the L5
callers' ids, but reopening the segment-kind collision this same phase
already closed for the public-surface builder's own nodes), or it stays
segment-kind-aware (closing that collision, but then no longer matching
what the unchanged L5 callers compute, so add_node()'s id-collision
merge never triggers and nothing reconciles — the original finding's
own failure mode, unsolved by the relocation).
Left as an explicit, scoped-out residual rather than attempted a
third time under review pressure, matching this plan's own established
discipline for a gap of this shape (the dump/scan typed-API convergence
in AGENTS.md's "PR C" note is the same class of problem: real,
cross-cutting, not a same-phase afterthought). A correct fix needs
one of: (a) migrating the twelve L5 call sites to construct a real
EntityId/OccurrenceId from whatever scope/kind information their own
producers (SourceEntity, clang AST nodes, USR strings) actually carry
before flattening it away — a genuine, separate data-flow change to
eight already-complex modules, not a drive-by edit; or (b) a lossless
mapping recovering the lost segment-kind information from each
producer's own provenance, which would need its own audit of what each
of the twelve call sites' inputs actually preserve today. Until one of
those lands, the two builders sharing one SourceGraphSummary instance
(the assembly-step fix below) reconciles nodes only where the two
encodings happen to coincide — unparameterized, unambiguous declarations
with no real segment-kind collision, the common case — and a
declaration that does hit the segment-kind collision keeps two separate
nodes across the two builders, an accepted limitation for this phase
rather than a silently-assumed-closed gap.
- Sharing node ids alone does not merge two graphs — this phase adds the
actual assembly step, not only a shared identity. merge_graph_facts
only folds the GraphFact list already attached to one node; it is
not itself what combines two independently-built graph objects, and an
earlier draft of this phase described the disagreement as closed on the
strength of shared node ids alone, which a reviewer correctly rejected —
two builders each producing their own, separate graph object can share
every node id and still never actually merge, because nothing calls the
merge. What actually merges two registrations today is
SourceGraphSummary.add_node()/add_edge(): within one
SourceGraphSummary instance, registering a second GraphNode under
an id already present calls merge_entity_facts, which is what invokes
merge_graph_facts underneath. So the real fix is an assembly step, not
an identity claim: both builders write into the same
SourceGraphSummary instance for a given snapshot side.
compare/surface_graph.py's public-surface builder and source_graph.py's
L5 builder (when L3-L5 evidence is present) are both given the same
SourceGraphSummary object and both call its real add_node/add_edge
— exactly the pattern buildsource/header_graph.py's existing
build_header_only_graph() already uses internally (graph =
SourceGraphSummary(); ...; graph.add_node(...)), generalized here to
two independent builders sharing one instance instead of one builder
filling it alone. Who constructs and threads that one instance matters
for the same import-direction reason D5 already corrected once in this
phase: compare/surface_graph.py may not import SourceGraphSummary
from buildsource/ directly (compare -> model is the allowed edge;
buildsource/ is extract-layer, and compare -> extract is not), so
the instance is constructed and handed to both builders by the
orchestrating workflow code (workflows/, which is allowed to import
model, extract, and compare alike) — each builder function receives
the shared SourceGraphSummary as a parameter and only ever calls
.add_node/.add_edge on it, never constructs or imports it itself.
AbiSnapshot.build_source.source_graph cannot be where this
unconditional graph lives — build_source: BuildSourcePack | None is
itself None for an ordinary L0-L2 snapshot with no --sources/
--build-info, which is the common case this phase's "available
unconditionally" claim is specifically about; attaching the graph only
under an optional evidence pack would mean fabricating a pack just to
hold it, silently widening what build_source is not None means
elsewhere in the codebase (it currently means "build/source evidence
was collected," which several existing checks rely on). Fixed by a new,
always-present field directly on AbiSnapshot — surface_graph:
SurfaceGraphLike | None = field(default=None, kw_only=True) (None
only for a snapshot this phase hasn't touched yet, e.g. an old loaded
snapshot predating this field; always populated for a freshly-extracted
one, regardless of whether build_source is set). This is the one
shared instance both builders write into — when build_source evidence
also exists, the L5 builder writes into the same AbiSnapshot.
surface_graph instance rather than a separate graph attached under
build_source, so there is exactly one graph-shaped field per snapshot
after this phase, not a conditional one nested under an unrelated
optional pack.
A snapshot persisted before this field existed has surface_graph is
None, and a query over the public surface must not treat that the same
as "nothing is public" -- a first draft of this phase left the backfill
unaddressed, which would have broken (or silently emptied) every existing
baseline's public/export-surface queries the moment compute_public_surface
stopped falling back to its own flat-snapshot traversal.
The fix is a lazy backfill, but it cannot live inside
PublicSurfaceQuery.resolve() itself — a first draft of this paragraph
placed it there without checking the resolver's own declared signature,
and review correctly caught the contradiction: resolve(graph,
explicit_roots) takes a pre-built graph, never the AbiSnapshot the
backfill would need to build one from when that graph is None. A
resolver that only ever receives graph=None for an old snapshot has
nothing to backfill from — there is no snapshot reference in scope to
read header origin/declaration/export-table data out of. The backfill
therefore runs one layer up, in a single shared helper every caller of
PublicSurfaceQuery.resolve() routes through rather than each
reimplementing its own None check — policy.public_surface.
resolve_public_surface(snapshot, explicit_roots) (a thin wrapper, not a
second query implementation): it reads snapshot.surface_graph,
lazily builds one on the fly, in memory, using the flat AbiSnapshot
fields that are actually available on an old snapshot (header origin,
declarations, export-table data) when that field is None — and then
calls PublicSurfaceQuery.resolve() with that graph, which stays exactly
the graph-only traversal its signature already states.
That graph is a lossy approximation, not the real thing — a first
draft of this paragraph claimed the backfill reuses "the exact same
compare/surface_graph.py builder a fresh extraction already uses," and
that claim contradicts Phase 2's own finding directly above. Phase 2
establishes that EntityId/ScopePath construction needs the typed
scope-segment list the parsers track internally during the AST walk —
which node kind each scope-stack entry actually is (namespace vs. record
vs. inline namespace vs. anonymous scope), plus kind-specific data like a
record's access specifier — and that qualified_name's flattened
"::".join(...) string is structurally, not merely
implementation-incompletely, insufficient to reconstruct that list: the
segment-kind tag was never captured in the string in the first place, so
no amount of re-parsing qualified_name recovers it. An old snapshot
predating this phase was written by a parser that only ever produced the
flattened string — it has no typed scope list anywhere to read, on disk
or in memory, because Phase 2's widening of entry.scope from
list[str] to typed segment records is exactly the parser-side change
that old snapshot's own extraction run never had. The "fresh extraction"
builder this sentence originally pointed to is building EntityId-keyed
nodes from that typed list during parsing; the backfill has no
parsing step to draw it from, only the flat fields the old snapshot
actually persisted. So the backfill cannot build a true EntityId-keyed
graph for an old snapshot, full stop — it is not a gap in this
wrapper's implementation to close later, it is the direct consequence of
the fact the backfill's only inputs are exactly what Phase 2 already
proved is insufficient.
The honest fix is to build an approximate graph instead, keyed on the
qualified-name string itself (optionally paired with kind to at least
separate a record from a function sharing one bare name) rather than on
a real EntityId, and to carry that distinction in the type system
rather than leave it implicit: the backfill returns a graph over
EntityId-shaped keys synthesized with an empty/best-effort ScopePath
(every segment collapsed to a single untyped Namespace-kind entry, the
closest-fitting existing segment type, rather than inventing a sixth
segment kind solely to mean "unknown") — which is exactly the same
collision class compute_public_surface()'s/export_surface.py's own
pre-migration string-keyed traversal already has today (two
same-named declarations in different namespaces, or a record and a
function sharing a bare name, collapsing onto one key) — so this backfill
is a lateral move to the new query shape with the same known, already-
accepted fidelity loss, not a regression and not a new capability. A
fresh extraction under this phase never takes this path at all (its
surface_graph is never None), so the approximation is reached only
for a snapshot already using today's qualified-name-string semantics —
it degrades to what that snapshot already had, nothing worse.
The approximation's effect on resolve_public_domain()'s own
structured result — resolvable/ambiguous_type_names/
exact_type_identities — is not automatic, and needs stating
explicitly rather than left to be inferred from "it's the same
collision class." The backfilled graph's collapsed ScopePath
segments mean two genuinely distinct declarations (same leaf name,
different enclosing scope kind — a record nested in a record vs. the
same names nested in a namespace, the exact distinction Phase 2's own
widened entry.scope exists to preserve and this backfill has no way
to recover) can merge onto one synthesized EntityId that a fresh
extraction would have kept separate.
The first version of this rule could not actually detect the
collision it names, and a review round caught why: "two or more
distinct qualified-name+kind pairs onto one synthesized key" is not a
condition that can ever hold — the synthesized key is a function of
the qualified-name+kind pair, so two genuinely distinct pairs can never
map onto the same key in the first place; by the time the backfill
runs, a real collision (different original ScopePath, identical
flattened spelling) has already reduced to one, not two, observable
pairs. The corrected, observable signal is different: not "distinct
pairs merged," but "the same pair was produced by more than one
separate flat declaration" — i.e. two or more entries in
snapshot.types/snapshot.functions/etc. that already share an
identical qualified-name+kind spelling before the backfill ever
touches them, the same producer-side namespace-dropping collision
class AGENTS.md's own type_reachability.py/opaque-type entries
already document for this codebase's bare/partially-qualified-name
matching. Any such duplicate lands the shared key in
ambiguous_type_names. But an unduplicated key is not, on that
basis alone, promoted to exact_type_identities either — a second
correction past the first fix's remaining gap. A single observed flat
entry for a given spelling is not proof that no collision occurred:
this codebase's own upstream producers already first-wins-dedup by
identity in several places (model.py's function_map/variable_map/
type_by_name), so two genuinely distinct declarations sharing a
flattened spelling could already have been silently reduced to one
surviving flat entry before this backfill ever sees the snapshot —
leaving no duplicate for it to observe. The backfill therefore never
promotes any of its own synthesized keys to exact_type_identities at
all, duplicate-observed or not; a key is in ambiguous_type_names when
a collision is actually observed, and in neither set otherwise (simply
absent from the anti-hiding mechanism, not asserted safe) — strictly
more conservative than the first draft's rule, and the only rule this
backfill's own inputs can actually support. resolvable itself is
unaffected by the approximation: it answers "does this snapshot have
header-derived visibility at all" (a question the flat fields already
answer on their own, independent of EntityId fidelity), not "is this
graph's identity resolution trustworthy" — conflating the two would
incorrectly downgrade a genuinely resolvable old snapshot's surface to
the unscoped-everything fallback merely because it predates typed
ScopePath data, which is a strictly worse outcome than the accepted
ambiguity-tracking loss this paragraph already owns.
The regression test this fix needs is also corrected: a first draft
paired a record and a function sharing one spelling, which EntityId's
own kind discriminator already keeps apart (they could never
synthesize onto the same key to begin with), so that fixture exercised
no real collision at all. The actual regression test and the
parity-test requirement below (Phase 3's own) instead fixture two
same-kind declarations sharing one flattened spelling: two separate
RecordType entries in snapshot.types, both with qualified name
Outer::Foo (the realistic trigger being the producer-side
namespace-dropping collision this codebase already documents
elsewhere, not a hand-contrived input) — resolving through the
backfill with the shared key landing in ambiguous_type_names and
absent from exact_type_identities, confirmed to fail against a
version of the backfill that either treats the collapsed key as
unambiguous or promotes an unduplicated key to exact_type_identities
on the strength of its single observed occurrence alone.
The backfilled
graph is not written back onto the loaded snapshot object (no silent,
surprising mutation of a caller's loaded snapshot) -- a query against
the same old snapshot pays the build cost each time, which is the
correct tradeoff for what should be a rare path once fresh snapshots
carry the field. compute_public_surface(snapshot) and any other direct
caller call resolve_public_surface(snapshot, ...), never
PublicSurfaceQuery.resolve() directly, so the None-backfill and the
graph-only resolver stay two separably-testable pieces rather than one
function quietly doing both — and a workflow-layer caller that already
holds a resolved, non-None graph (the common, fresh-extraction case)
pays no extra indirection beyond the one wrapper call, with no fabricated
pack to thread through either way.
ADR-057/053's consumers still
read the L3-L5-gated graph only when it exists, and migrating them onto
querying through PublicSurfaceQuery's shared instance directly is
still explicitly not part of this phase (each stays its own later,
separately-justified phase, per this plan's "don't attempt a change with
no real caller" discipline) — but what changes this time is structural,
not aspirational: there is one graph object per snapshot side after this
phase, not two that merely happen to agree on node spelling.
The first of two items this relocation owes a real design is now
resolved, not deferred a fourth time — a review round correctly found
that AbiSnapshot.surface_graph's own declared type above is not
actionable while this stayed open, which is a different problem than
"this would benefit from being decided eventually." SourceGraphSummary
itself — the container class with add_node/add_edge/
resolve_entities, as opposed to the GraphNode/GraphEdge primitives
Phase 3's own model/graph.py relocation already covers — still lives in
buildsource/source_graph.py today, and does not relocate alongside
them: its own imports (buildsource.build_evidence.BuildEvidence,
buildsource.entity_resolver.EntityResolver) are genuine L3-L5
build/source-evidence types, not model-layer primitives, so moving the
whole class to model/ would drag those two modules (and whatever they
themselves depend on) into model/ too — the same kind of inversion the
GraphNode/GraphEdge relocation was careful to avoid by checking its
own dependency closure first, just failed here by not checking
SourceGraphSummary's. The resolution is the protocol option this
paragraph named but didn't choose: model/graph.py gains a narrow,
structural typing.Protocol (e.g. SurfaceGraphLike) — covering both
the write side and the read side, not only add_node/add_edge as a
first draft of this fix had it. That first draft checked only what
the Design section's builders call on the shared instance, and missed
the one caller who actually needs to read the graph back:
PublicSurfaceQuery.resolve()/resolve_public_domain() must traverse
whatever AbiSnapshot.surface_graph already holds — closing reachability
through includes/declares/references/instantiates edges is this
phase's whole Goal — and a protocol exposing only two write-only methods
gives that traversal nothing to read, forcing exactly the
buildsource.SourceGraphSummary-importing cast this protocol exists to
avoid. SurfaceGraphLike therefore also declares the two plain,
already-existing attributes a traversal actually needs —
nodes: Sequence[GraphNode]/edges: Sequence[GraphEdge] (Sequence,
not list, since the protocol only ever needs read access, and widening
to a broader container type is exactly what a Protocol is for) — plus
has_node(self, node_id: str) -> bool, SourceGraphSummary's own
existing O(1) membership check a naive node in self.nodes linear scan
would otherwise have to reimplement. All three already exist on
SourceGraphSummary exactly as declared, so this widening needs no
change to that class, only to the protocol's own declared surface.
AbiSnapshot.surface_graph: SurfaceGraphLike | None in
model/snapshot.py needs no import from buildsource at all —
SourceGraphSummary already structurally satisfies the protocol (Python
Protocols check structurally, not by inheritance, so the existing class
needs no base-class change either) — and every caller that actually needs
resolve_entities/other SourceGraphSummary-specific methods narrows
back from the protocol to the concrete type at its own call site.
That narrowing is an ordinary isinstance(graph, SourceGraphSummary)
check against the concrete class, and a first draft of this paragraph
mis-attributed why it works — @runtime_checkable has nothing to do
with it. isinstance against a concrete, imported class needs no
decorator at all; that check works for any class, protocol-adjacent or
not, with or without SurfaceGraphLike existing. What @runtime_
checkable on SurfaceGraphLike actually enables is the other
direction — isinstance(x, SurfaceGraphLike), a structural check
against the protocol itself, useful to a caller that wants to confirm
something conforms to the read/write surface this protocol declares
without needing (or having) the concrete buildsource import in scope
at all. Narrowing to reach resolve_entities specifically is the
concrete-class check, which needs the buildsource.SourceGraphSummary
import regardless of the protocol's own @runtime_checkable status —
a real, ordinary, localized import at that one call site, not a
model-layer concern, and not something the protocol's decorator
changes either way. SurfaceGraphLike stays @runtime_checkable
anyway, for the structural-conformance case that decorator genuinely
does enable, just not for the reason the first draft gave. The
second item, below, is the one still left to the implementation PR, for
the reason already stated — it depends on auditing and migrating real
existing readers, not on a type-contract decision a planning document can
make in the abstract. Second: moving the L5 graph's attachment point off
BuildSourcePack.source_graph has real existing readers —
internal_leak.py, buildsource/cross_source_checks.py, buildsource/
evidence_report.py, evidence_depth.py, and cli_graph.py among them —
each would observe
no graph at all the moment the L5 builder stops writing to the old
location, silently regressing impact/cross-check/assurance behavior
that works today.
A review round correctly rejected leaving this as a pure "known gap
for the implementation PR" — there is a concrete, low-risk fix available
now, not just a later migration obligation, and not adopting it would
leave five real readers observing None the moment this phase ships.
Rather than migrating every reader to AbiSnapshot.surface_graph
directly in this same phase (a real, separate audit this phase does not
have the implementation in front of it to safely perform, per the
existing reasoning below), BuildSourcePack.source_graph is kept as a
live alias to the same object, not left unpopulated: whenever
build_source exists, the L5 builder's assignment snapshot.
surface_graph = built is immediately followed by build_source.
source_graph = built — the identical object, not a copy — so every
existing reader keeps observing the real, current graph through its own
already-working access path with zero code changes on their side, while
AbiSnapshot.surface_graph is simultaneously the one new, unconditional
field every new consumer (the public-surface graph builder, this
phase's own query layer) reads from. This is not a second representation
competing with the first — it is the single SourceGraphSummary
instance reachable through two attribute paths, exactly preserving the
"one object, not two that happen to agree" guarantee this phase's own
assembly-step design already states, just exposed at both of its
pre-existing and newly-added access points rather than only the new one.
Migrating each of the five readers to stop going through the alias and
read AbiSnapshot.surface_graph directly remains real, scoped,
follow-up work — genuinely Phase 3's own implementation PR's to
schedule and verify against each reader's existing tests, since that
part is not itself safety-critical once the alias prevents the silent
None regression — but it is no longer a precondition for this phase
to ship without breaking existing behavior.
The in-memory alias does not, by itself, survive a save/load round
trip — a review round correctly traced what actually happens on
serialization and found the "one object, two attribute paths" guarantee
breaks exactly there. serialization.snapshot_to_dict() already
encodes BuildSourcePack.to_embedded_dict() (which includes
source_graph) for any snapshot carrying build-source evidence, and
this phase's own AbiSnapshot.surface_graph field is additionally
serialized at the top level (the schema-version-bump field named below)
— so a snapshot with both populated writes the identical graph twice,
as two independently-encoded blobs. On load, decoding each field
separately reconstructs two distinct (if currently equal) SourceGraph
Summary objects rather than rebinding one to the other — a real
mutable-object alias that held in memory is gone the moment a snapshot
is saved and reloaded, silently doubling on-disk size for a real,
potentially large L5 graph and letting legacy and new readers diverge
after deserialization if either is mutated afterward. Fixed by treating
the write side the same way the in-memory assembly step already does:
snapshot_to_dict() encodes AbiSnapshot.surface_graph once, and
BuildSourcePack.to_embedded_dict() omits source_graph whenever the
owning snapshot already has one (the ordinary case for every snapshot
this phase's assembly step touches) rather than re-encoding the same
object a second time. snapshot_from_dict() decodes the top-level
surface_graph once and rebinds build_source.source_graph to that
same decoded instance — restoring the alias on load, not just on
construction.
Aliasing a legacy document's nested graph forward into
AbiSnapshot.surface_graph — the direction this paragraph originally
also specified, for a document written before this phase — is itself
wrong, and a review round correctly traced the consequence: it
silently defeats the approximate-backfill design two sections below.
resolve_public_surface()'s whole reason for existing is that a
snapshot with surface_graph is None gets the lossy-but-designed-for-
this-case approximate graph built from its flat fields; a snapshot with
surface_graph already non-None skips that backfill and queries the
graph directly. A pre-Phase-3 document's nested build_source.
source_graph is an L3-L5 evidence graph that predates the public-
surface builder entirely — it was never populated with the includes/
declares/references/exports edges PublicSurfaceQuery.resolve()
actually traverses, so aliasing it forward makes surface_graph
non-None while still lacking exactly the edges the query needs,
silently skipping the intentional approximate-backfill path in favor of
querying a graph that resolves to a smaller or empty public surface
than either the backfill or the pre-migration flat-snapshot traversal
would have produced — worse than leaving it None, not equivalent to
it. Fixed by not aliasing in this direction at all: for a legacy document
(no top-level surface_graph key), AbiSnapshot.surface_graph stays
None exactly as it would for any other snapshot predating this field,
triggering resolve_public_surface()'s own designed fallback correctly;
build_source.source_graph is decoded from its own nested key exactly as
it always was, unaffected, so the five pre-existing L5 readers
(internal_leak.py/cross_source_checks.py/evidence_report.py/
evidence_depth.py/cli_graph.py) see the identical graph they always
did. The "one object, two attribute paths" guarantee is therefore scoped
to what this phase's own assembly step actually produces — a freshly
extracted or freshly re-saved snapshot, whose surface_graph has
genuinely been through the public-surface builder — not retroactively
forced onto a document this phase never touched.
- The relevance query — abicheck/policy/public_surface.py (new):
PublicSurfaceQuery.resolve(graph, explicit_roots) -> frozenset[EntityId],
a traversal from explicit public roots through includes/declares
edges (closing the reachable-header surface) and references/
instantiates edges (closing the reachable-type surface). policy -> compare
is an already-allowed import edge under ADR-061, so policy/ can consume
the compare/-built graph directly; this is where compute_public_
surface()'s actual decision logic — which declarations count as part of
the public contract — lives after migration.
resolve()'s bare frozenset[EntityId] is not a complete replacement
for today's PublicSurface, and a first draft of this bullet implied it
was by never saying otherwise — review correctly read that silence as a
real gap, not a simplification. surface.py's PublicSurface carries
far more than membership: resolvable/has_typed_roots/has_provenance
(three independently-meaningful "can this surface be trusted at all"
signals — no header-derived visibility at all, an export-table-only
surface with no typed roots to close a type closure from, no provenance
because the snapshot wasn't dumped with a public-header set),
ambiguous_type_names/exact_type_identities (which bare-name
resolutions are trustworthy vs. collision-prone), and two origin indices
(origin_by_key/origin_by_qualified_key) that _hidden_friend_owner_
effective_origin and other callers read directly. FilterNonPublicSurface
(post_processing.py) checks surf_old.resolvable or surf_new.resolvable
before it will scope anything at all — collapsing that into "is this id
in resolve()'s frozenset" erases the same distinction the exports
domain fix two paragraphs above already had to preserve for ExportSurface:
"not reached" vs. "reachability could not be established." An empty
frozenset is indistinguishable from either, and reading the latter as the
former would scope out every finding on a snapshot with no resolvable
surface at all, instead of correctly falling back to "keep everything
unscoped" the way FilterNonPublicSurface does today.
Fixed the same way the exports domain already is: resolve() stays the
bare-membership convenience method for a caller that genuinely only needs
set membership (the new type_reachability-replacement query below, which
never needed anything but membership), but the actual replacement for
compute_public_surface()'s public-domain result is a second, structured
method — PublicSurfaceQuery.resolve_public_domain(graph, explicit_roots)
-> PublicSurfaceResolution — returning a result that carries the same
resolvable/has_typed_roots/has_provenance/ambiguous_type_names/
exact_type_identities/origin-index shape PublicSurface already does,
computed from graph traversal state instead of surface.py's own
independent walk. compute_public_surface(snapshot) (via resolve_
public_surface(), the backfill wrapper above) migrates to build this
structured result from the query instead of running its own closure walk;
its callers (FilterNonPublicSurface, classify_change_surface,
contract evaluation's confirmed-type-match logic) keep reading the exact
same field names they read today, unaffected by where the computation now
happens.
The contract=exports domain does not collapse into this same
bare-frozenset[EntityId]-returning resolve(), and a first draft of
this phase claimed it did — review correctly found that claim
incompatible with what export_surface.py's real consumers actually
need. ExportSurface is not a membership set — it is a structured
result carrying resolvable: bool and the exclusion_is_provable
property, computed from several independent completeness conditions (no
observed export table, no resolved root, an untyped root, an unaccounted
export, an unresolved type edge — export_surface.py's own documented
fail-closed gate), and contract_evaluation.py/contract_evidence_
collect.py consume exactly that structured state to decide whether a
PROVEN_OUT_OF_CONTRACT classification is actually safe to make, not
only whether a given EntityId is in some resolved set. Collapsing this
into "is this id in the frozenset resolve() returns" erases the
distinction between "not reached" and "reachability could not be
established" — exactly the distinction exclusion_is_provable exists to
keep, and losing it could let an incompletely-evidenced exclusion read as
proven, or silently drop a real contract-coverage failure. The fix:
exports queries the same shared graph this phase builds (the
evidence stays unified, per the Governing Invariant), but through a
second, differently-typed method — PublicSurfaceQuery.
resolve_export_domain(graph, ...) -> ExportSurface (or an equivalent
structured result preserving resolvable/exclusion_is_provable), not
the bare-set resolve() — since this domain's consumers need the
completeness state resolve()'s own return type has nowhere to carry.
export_surface.py's own closure-walk algorithm migrates to build this
structured result from graph edges instead of its own independent scan;
its result shape does not migrate into resolve()'s, and Phase 10's
later deletion of export_surface.py's independent closure walk (named
below) means the walk, not the structured ExportSurface type or its
consumers' own contract-evaluation logic, both of which stay exactly as
they are, fed by the new query instead of the old scan.
type_reachability.py's directly_referenced_stdlib_types() — itself a
relevance decision (it un-filters a record for suppression purposes) —
becomes a second, narrower query in policy/public_surface.py over the
same graph (a one-hop references filter) rather than its own independent
scan with its own ambiguity-tracking machinery — the machinery this phase
removes is exactly what Phase 2 already started removing for the identity
half of the same problem; this phase removes the reachability half.
Files. abicheck/model/graph.py (new — the full dependency closure
the relocated types actually need, not only the five originally named
symbols: GraphNode/GraphEdge/GraphFact/FactConflict/
merge_graph_facts plus _normalize_if_decl_or_type/edge_relation_key/
ensure_facts_and_resolve — a first draft of this phase named only the
first five, but GraphNode.from_dict/GraphEdge.relation_key call the
other three directly, so leaving them behind in buildsource/graph_facts.py
would either break those methods once that module is trimmed to a
re-export shim, or force model/graph.py to import back out to
buildsource/ to reach them — the identical import-direction mistake
this phase already corrected once for SourceGraphSummary itself.
ensure_facts_and_resolve's own identity normalization imports
abicheck.name_classification, which is safe to bring along — that
module has zero internal imports of its own (pure re-based string
utilities), so it is already a leaf and importing it from model/
introduces no cycle); buildsource/
graph_facts.py/buildsource/source_graph.py (trimmed to re-export from
the new location, NODE_KINDS/EDGE_KINDS and L5-specific construction
logic unchanged in place); abicheck/compare/surface_graph.py (new —
public-surface node/edge kind vocabulary and builder, using model/
graph.py's primitive, not a new one); abicheck/surface_graph.py —
each of build_surface_graph()/compute_surface_metrics() operates on
exactly one snapshot, and a review round correctly found an earlier draft
of this bullet gave each of them the two-snapshot old_public_entity_ids/
new_public_entity_ids pair, which neither function has any unambiguous
way to route for a single-snapshot call. Each gains exactly one
optional parameter instead — public_entity_ids: frozenset[EntityId] |
None = None — and the pair lives only at the two-snapshot callers below,
each of which passes its own side's set to its own matching call:
build_surface_graph(old, public_entity_ids=old_ids)/
build_surface_graph(new, public_entity_ids=new_ids). SurfaceGraph.
public_roots() maps a given set of ids back to the existing
Function.name/Variable.name declaration spelling — not a mangled/
symbol spelling, per the correction above (_root_seed_types/
reachable_types/idioms.py's create/destroy regexes are all keyed on
this plain name already) — preserving its existing frozenset[str]
return type exactly — surface_graph.py itself still never imports
policy/public_surface.py, per the note above).
Threading public_entity_ids into compute_surface_metrics()'s own
signature does not, by itself, make its metrics reflect the resolved
surface, and a review round correctly traced why: every one of its
public-only counts is computed straight from Visibility.PUBLIC, with
no reference to the parameter at all. Reading the real function:
public_functions/public_variables/exported_symbols/
undocumented_export_ratio/the per-header exported_counts tallies
each sum fn.visibility == Visibility.PUBLIC/var.visibility ==
Visibility.PUBLIC directly, and
public_types/public_enums come from _public_type_counts(), which
calls its own, entirely independent compute_public_surface(snap) —
never the public_entity_ids its caller already resolved. Adding the
parameter without touching any of these means diff_surface_metrics()'s
PUBLIC_SURFACE_GREW/SHRANK findings (via _public_decl_count(),
which sums exactly these fields) keep reflecting the legacy
Visibility.PUBLIC definition regardless of what public_entity_ids
says — a regression test proving the parameter was threaded through the
call signature would pass while the emitted findings stayed unchanged,
the exact gap this finding names. Fixed by having compute_surface_
metrics() use public_entity_ids, when non-None, for every one of
these public-only tallies: a function/variable counts as public when
its own
resolved EntityId is a member of the set (matching build_
surface_graph's own root-seeding rule, not a second definition), and
_public_type_counts() takes the same public_entity_ids argument and
counts record/enum membership directly from it instead of running its
own independent compute_public_surface() resolution — the caller
already resolved this once; a second, separate resolution inside
_public_type_counts() is exactly the redundant-recomputation this
phase's own design elsewhere avoids.
The per-header declared_counts tally is not one of these — a
further review round correctly caught this finding's own first draft
grouping it in with the visibility-filtered tallies. HeaderCoverage.
declared is explicitly the count of declarations physically defined in
the header, incremented unconditionally for every function/variable/
type/enum regardless of visibility — the denominator HeaderCoverage.
exported (from exported_counts) is measured against. Filtering
declared_counts to the resolved public set too would change that
denominator and misreport header coverage, not fix it; only
exported_counts (a public-only tally, alongside the other fields named
above) is affected by public_entity_ids. When public_entity_ids is None
(every call site outside compare()'s own pipeline), every one of these
tallies keeps its exact current Visibility.PUBLIC-based behavior,
_public_type_counts() included — the same explicit, narrow residual
public_roots()'s own Visibility.PUBLIC fallback already states,
extended to the metrics this function computes directly rather than
through the graph — and declared_counts stays unfiltered in every
case, None or not. pattern_verdicts.py
(apply_pattern_verdicts() gains the two-snapshot old_public_entity_ids/
new_public_entity_ids pair, each threaded through as the single
public_entity_ids argument to its matching build_surface_graph()
call); diff_surface_
metrics.py (diff_surface_metrics() gains the identical pair, each
threaded through the same way to its matching compute_surface_metrics()
call); checker.py (_apply_pattern_verdicts_step/_apply_surface_metrics
both gain the identical pair — received as an already-resolved parameter
from compare()'s own caller (classify_compare_pair()/service.
compare_snapshots()), never resolved by compare() itself; see
immediately below for why). Neither
checker.py nor compare()
itself may call PublicSurfaceQuery.resolve() directly to populate it —
a first draft of this phase's text said exactly that ("the same
PublicSurfaceQuery.resolve() result compare() already computes"),
which is the identical policy -> compare direction violation this
phase's own surface_graph.py fix (above) already corrected once, just
reappearing at a second call site compare() itself. checker.compare()
stays compare/-layer code with no import of policy/public_surface.py
anywhere in it; the actual
resolve_public_surface() call (the snapshot-aware wrapper around
PublicSurfaceQuery.resolve(), per the backfill fix below) moves to the
workflow layer that
already orchestrates checker.compare() for the typed pipeline —
service_compare_pipeline.py's classify_compare_pair (or wherever the
Phase 4 AnalysisPlan resolution already runs, since both need the same
graph) resolves the ids once, which is the workflows -> model, storage,
extract, compare, policy edge ADR-061 already permits. service.
compare_snapshots() — the second, documented Tier-2 production verb that
also calls checker.compare() directly, per the residual fix above —
performs the identical per-side resolve_public_surface() call before
forwarding into compare(), rather than being left to fall back to the
Visibility.PUBLIC default classify_compare_pair's callers never hit.
Whether compare()'s own new parameter is optional (with a fallback)
or required was left as this phase's second open design question, and
a later review round resolved it by proving "required, no fallback" is
the wrong answer outright — it breaks real, existing in-pipeline
callers, not just compare()'s own external callers. A repo-wide
check found compute_public_surface() called directly, with a bare
snapshot and no resolution in hand, from inside the detection
pipeline itself: diff_stdlib_impl.py/surface_graph.py's own
_public_type_counts() call it with a single snapshot argument and no
PipelineContext/workflow caller anywhere in their own call chain to
have pre-resolved anything; post_processing.py's FilterNonPublic
Surface and contract_pipeline.py's evidence-collection stage already
call it too, but — tellingly — contract_pipeline.py's own existing
code already shows the right shape for this: it reads pp_ctx.surf_old/
.surf_new (the cache FilterNonPublicSurface populates) first, and
falls back to an independent compute_public_surface() call only when
that cache is empty (a POST-manifest-only run, or scope_to_public_
surface=False, which never populates it). A "required, no fallback"
signature cannot serve any of these calls — none of them have a
pre-resolved PublicSurfaceResolution to pass, and diff_stdlib_impl.py/
surface_graph.py have no PipelineContext to read a cached one from
at all. Resolved instead the way build_surface_graph()/compute_
surface_metrics() already resolve the identical tension one section
up: compute_public_surface() keeps an optional resolution:
PublicSurfaceResolution | None = None parameter, not a required one —
when None, it calls the same lazy, snapshot-aware resolve_public_
surface() wrapper internally (the identical call a workflow-layer
caller would make, just made on the callee's behalf rather than the
caller's) instead of requiring every call site to pre-resolve. A caller
that already has a cached resolution in hand (contract_pipeline.py's
pp_ctx.surf_old/.surf_new reuse, or compare()'s own pipeline once
it resolves once per snapshot up front) passes it explicitly purely as
an optimization avoiding redundant recomputation — never because the
function would otherwise fail to run.
That internal fallback call is itself surface.py importing from
policy/public_surface.py, and a further review round correctly asked
whether this reintroduces the exact compare -> policy direction this
phase's own design elsewhere forbids. Checked against the actual
enforcement, not assumed either way: scripts/check_architecture.py's
package-boundary gate only evaluates a file once _source_layer_for()
resolves it to an already-migrated ADR-061 package — its own loop reads
if source_layer is None: continue before checking a single import.
surface.py, diff_stdlib_impl.py, surface_graph.py, post_
processing.py, and contract_pipeline.py are all still flat, top-level
modules today — none of them has been migrated into abicheck/compare/
(or any other ADR-061 package) by any phase in this plan — so
source_layer resolves to None for every one of them, and the gate
checks nothing about what they import, policy/public_surface.py
included. There is therefore no currently-enforced violation this design
introduces, and the policy -> compare directionality this phase
enforces elsewhere is specifically about code that has migrated into a
package — policy/public_surface.py itself, which the gate does check,
imports only compare/ and model/, never backward. This is a real
residual nonetheless, not a closed question: the moment a future phase
migrates surface.py (or any of its callers) into abicheck/compare/,
this exact import becomes a real, gate-enforced violation, and that
future migration would need to either move surface.py's compute_
public_surface() into policy/ alongside the query it already calls,
or relocate the lazy-resolve fallback itself to whichever workflow-layer
caller triggers that migration — named here so that phase's own
implementation PR inherits the constraint explicitly rather than
discovering it as a fresh gate failure. abicheck/policy/public_surface.py
(new — PublicSurfaceQuery, migrated from surface.py's existing
traversal logic); surface.py (compute_public_surface(snapshot,
resolution: PublicSurfaceResolution | None = None) — not
public_entity_ids: frozenset[EntityId], which a first draft of this
Files entry still said, predating (and left un-synced with) the
structured-result fix a few paragraphs above. A bare frozenset is
exactly the membership-only collapse that fix exists to prevent:
compute_public_surface()'s own PublicSurface result carries
resolvable/has_typed_roots/has_provenance/ambiguous_type_names/
exact_type_identities/both origin indices, none of which a set of ids
can express, and FilterNonPublicSurface's surf_old.resolvable or
surf_new.resolvable check (among others) runs before
compute_public_surface() has any membership set to offer at all — there
is no point reconstructing that state a second time from a bare id set
once a caller already computed it via PublicSurfaceQuery.resolve_
public_domain(). compute_public_surface() takes that structured
PublicSurfaceResolution directly (when given one) and projects it into
the existing PublicSurface field shape, so every existing reader of
PublicSurface's own fields is unaffected by where the computation
happened. Every one of the four in-pipeline callers named above keeps
its existing call shape completely unchanged (compute_public_surface
(snap)/compute_public_surface(ctx.old), no new argument required at
any of them) — this phase's acceptance bar for them is exactly that
they compile and behave identically with zero edits, not that they
thread a new parameter through. surface.py's own pre-existing
traversal logic (the actual algorithm PublicSurfaceQuery migrates) is
still deleted per this phase's Acceptance criteria below — it is the
internal implementation that changes, not the public call shape every
existing caller already depends on. dumper_scoping.py/
export_surface.py/type_reachability.py (each becomes a graph builder
contributing nodes/edges in compare/, or a relevance query in
policy/, not an independent reachability algorithm); abicheck/model/
snapshot.py (new AbiSnapshot.surface_graph: SurfaceGraphLike | None
field, unconditional — not nested under build_source); the
workflows/-layer dump/compare orchestration code that already calls
buildsource.header_graph.build_header_only_graph()/attaches
build_source.source_graph (service_header_graph_attach.py and
siblings) gains the one line constructing a single SourceGraphSummary,
assigning it to snapshot.surface_graph, and threading that same
instance into both the public-surface builder and the L5 builder, per the
assembly-step design above. abicheck/
workflows/consumer_graph.py (ADR-057's consumer graph) and ADR-053's
TU→link-unit→DSO attribution are explicitly not migrated to query the
graph in this phase — each stays a candidate for a later, separate phase,
per this plan's "don't attempt a change with no real caller" discipline
(see AGENTS.md's "shape first, wiring later" gap and ADR-063 D7's
capability-lifecycle states) — but the moment this phase ships, the graph
they would eventually query is already the single, merged
SourceGraphSummary instance per the assembly step above, not a second
object they'd need their own migration to reconcile with.
Tests. Every existing surface.py/type_reachability.py regression
test (including the namespace-collision property suite Phase 2 already
restated for identity) is kept and re-targeted at
PublicSurfaceQuery.resolve's output — this phase's acceptance bar is
that none of those tests need a behavior change, only a different call
path, and any test that does need a behavior change is a sign this phase
introduced a real regression, not a refactor. The existing L5 source-graph
test suite (tests/test_source_graph*.py/tests/test_graph_facts.py or
their current equivalents) is re-run unchanged against the relocated
model/graph.py primitive via buildsource/graph_facts.py's re-export
shim, proving the relocation is behavior-preserving rather than asserted.
One new end-to-end regression is added specifically for the shared-assembly
claim above: a project with both a public header (no L3-L5 evidence
needed) and real --sources/--build-info evidence produces exactly one
SourceGraphSummary instance containing exactly one graph node for a
declaration both builders see, not two separate summary objects that
happen to agree on a node id — asserted by identity (is) on the summary
object each builder was handed, not only by comparing their outputs after
the fact, so a future regression that quietly goes back to constructing
two independent SourceGraphSummary() instances fails this test
immediately rather than only failing once two disagreeing facts happen to
surface. A second, separate regression locks down persistence: today's
writer never round-trips a graph through plain asdict() at all — the
existing BuildSourcePack.to_embedded_dict()/SourceGraphSummary.
from_dict() pair is a deliberate special case precisely because the
graph's canonical encoding isn't the dataclass default, reached today only
through build_source's old attachment path. Moving the graph onto
AbiSnapshot.surface_graph directly needs the identical special-casing
added to serialization.py's snapshot_to_dict()/snapshot_from_dict()
for the new field — without it, a saved-then-reloaded snapshot's
surface_graph comes back as None or a bare dict, not a
SourceGraphSummary, silently breaking resolve_public_surface()'s
snapshot-reading backfill on every persisted (as opposed to freshly-dumped)
snapshot. A populated-graph
save/load round-trip test (construct a snapshot with a real, non-empty
surface_graph, write it, read it back, assert the reloaded object is a
SourceGraphSummary with the same nodes/edges) is required by this phase,
not deferred to Phase 10's cleanup. A third regression covers the legacy
backfill: load an old-schema snapshot (surface_graph=None, constructed
the way a pre-this-phase snapshot would be) alongside a fresh one with a
real surface_graph, run compute_public_surface/compare() against
both, and assert the old snapshot's query result matches what
resolve_public_surface()'s lazy,
in-memory backfill produces rather
than crashing or returning an empty surface — and assert the loaded
snapshot object's own surface_graph attribute is still None afterward,
proving the backfill is genuinely not persisted back onto it. A fourth
regression covers abicheck/surface_graph.py's own migration: every
existing idioms.py/pattern_verdicts.py/diff_surface_metrics.py unit
test calling build_surface_graph()/compute_surface_metrics() directly
is re-run unchanged (behavior-preserving, same as the surface.py
migration bar above) — these keep their existing call shape and the
None-triggered legacy fallback, since they have no policy-layer caller
in their chain. A fifth, new regression covers the threaded path itself:
a checker.compare() run over a fixture where Visibility.PUBLIC and
real reachability disagree, asserting the pattern-verdict/surface-metrics
findings compare() actually produces reflect the resolved EntityId
answer, not the legacy Visibility.PUBLIC-only one — confirmed by
patching two different call boundaries, not one, since the pair and the
singular value live on different functions after the single-snapshot-
helper correction (a review round correctly caught a version of this test
description that patched build_surface_graph/compute_surface_metrics
for the pair, which those functions no longer accept at all): asserting
checker.py's _apply_pattern_verdicts_step/_apply_surface_metrics
call apply_pattern_verdicts()/diff_surface_metrics() with non-None
old_public_entity_ids/new_public_entity_ids when reached through
compare(), and separately asserting each of build_surface_graph/
compute_surface_metrics receives its own side's value as the singular
public_entity_ids argument — not only by comparing output, so a
future regression that silently stops threading either the pair or the
per-call singular value through checker.py fails this test even if it
happens not to change the specific fixture's output. A sixth regression pins the two-sided correction
itself, directly: a fixture where a declaration is public in old but
removed from the public-header set in new (or the reverse) — the one
shape a single shared id set would misclassify — asserting
build_surface_graph(old)/build_surface_graph(new) each receive their
own side's resolved ids, not the other side's, confirmed to fail against
a version of the threading that resolves one shared set and passes it to
both calls.
Separately, asserting
SurfaceGraph.public_roots() — still returning frozenset[str], still
consumable by re.Pattern.match() with no caller change — agrees with
PublicSurfaceQuery.resolve()'s answer rather than the old
Visibility.PUBLIC-only one, confirmed to fail against the
pre-migration SurfaceGraph for this exact input; and a second case
asserting surface_graph.py imports nothing from policy/, enforced by
the same architecture-gate mechanism this plan already uses elsewhere for
a leaf module's import direction. A seventh regression pins the kind-filter
fix directly: a fixture where PublicSurfaceQuery.resolve()'s resolved
set genuinely includes a record/enum/typedef EntityId (a public
function's return type, reachable via a declares/type-reference edge)
alongside function/variable ids, asserting SurfaceGraph.public_roots()
still returns a clean frozenset[str] of only the function/variable
spellings — the type-kind id silently excluded from the root set, not
attempted-and-failed — confirmed to fail against a version of
public_roots() that maps every received id unconditionally.
Acceptance criteria. surface.py's own traversal implementation is
deleted, not kept alongside the graph query — landed 2026-09-01, see
this phase's own "Traversal migration landed" paragraph above. export_
surface.py's independent closure walk is not a second implementation to
separately delete: it always called surface.py's _walk_type_closure
verbatim rather than keeping its own copy, so once that one function
migrated, both domains' closures did. What's still open there is narrower
than "delete a closure walk" — export_surface.py's own root-seeding
(the export-table-matching logic, genuinely independent of the header
domain) remains unmigrated, named explicitly as a residual rather than
silently left. No second node/edge dataclass hierarchy exists anywhere in
the repository after this phase — compare/surface_graph.py constructs
model.graph_facts.GraphNode/GraphEdge instances with its own kind
vocabulary, the same way buildsource/source_graph.py already does, never
a parallel type. The new AbiSnapshot.surface_graph field bumps
serialization.SCHEMA_VERSION the same way Phase 0's Fact[...] fields do
(a third, independent bump by this plan, on top of Phase 0's and Phase 7's
— all three are additive and bump the same pre-existing AbiSnapshot/
report-schema counters, not ADR-062's ProjectSnapshot schema). FP-rate
gate and per-tier accuracy gate both show no regression.
Phase 4 — AnalysisPlan: pre-flight resolution, not mid-run discovery¶
Goal. An unsatisfiable request (an evidence requirement no resolved collector/backend combination can produce) is rejected before extraction, with a named reason, not discovered as a silent no-op mid-run.
Design. abicheck/workflows/plan.py: AnalysisPlan as a frozen
dataclass (operation, per-side SidePlan, requested depth, required
facts, requested toolchain/compile-context inputs) built by a new
AnalysisPlanner.resolve(request) -> AnalysisPlan, raising
PlanningError on failure, not returning it — a first draft of this
signature wrote -> AnalysisPlan | PlanningError, a union return type
that directly contradicts the very next sentence's own "raise
PlanningError," and a review round correctly caught a contract this
plan states two different ways. Chosen for the raise-not-return
direction to match this codebase's own existing idiom for exactly this
shape of failure (a request a resolver cannot satisfy at all, as opposed
to a resolved value with its own partial-failure fields) —
DumpDepthNotSatisfiedError/ValidationError and siblings are raised,
never returned alongside the success type in a union a caller must
narrow before using. PlanningError carries one entry per failed
requirement (requested, why_unsupported), modeled directly on the
--build-target + pre-captured aquery gap and the -H + unsupported-
collect-mode gap AGENTS.md already documents as silent failures — this
phase's acceptance test is exactly "these two scenarios now raise
PlanningError instead of silently dropping the request," and every
caller of AnalysisPlanner.resolve() can therefore treat a returned value
as always a usable AnalysisPlan, with no isinstance/union-narrowing
step of its own.
"Resolved toolchain/compile context" is not a field this phase can
actually put in AnalysisPlan, and a review round correctly found this
plan already states why, one phase over, without connecting the two.
service_dump_pipeline.py's own ResolvedDumpRequest docstring (Phase 1,
already landed in this codebase) is explicit that the P0.3 L3→L2
compile-context fold cannot be determined without invoking it, and the
fold can raise HeaderCompileContextAmbiguousError on genuinely ambiguous
build evidence — which is exactly why that object deliberately excludes
the fold's result and the fold itself stays inside execute_dump_request,
never resolve_dump_request: running it during a side-effect-free resolve
step would be a real behavior change to --dry-run's existing contract
(never raising on anything but a usage error), not an additive one. An
AnalysisPlan built during resolve_compare_request/resolve_dump_request
is bound by the identical constraint — it cannot carry the fold's actual
resolved compile context without either running the fold during resolution
(the same contract change ResolvedDumpRequest's own design already
rejected) or leaving the field permanently unresolved, which is worse than
not stating it. Fixed by narrowing the field to what AnalysisPlan can
honestly carry: the requested toolchain/compile-context inputs (explicit
--gcc-path/--ast-frontend/language, and whatever --build-info/
--sources path was given) — the same inputs ResolvedDumpRequest itself
carries rather than the fold's output — not a resolved compile context.
This phase's own two named acceptance scenarios (--build-target +
pre-captured aquery; -H + unsupported collect mode) don't need the
fold's result either: both are about build-info/depth/collect-mode
compatibility, resolvable from the request's own inputs before any
compile-unit matching runs, so narrowing this field costs this phase
nothing it actually needed. HeaderCompileContextAmbiguousError itself
stays exactly where it already lives — raised from execute_dump_request,
not surfaced as a PlanningError — since catching it pre-flight would
require running the fold at resolve time, the one thing this phase cannot
do without reopening the behavior change ResolvedDumpRequest's own
design already closed.
AnalysisPlan deliberately does not carry resolved policy or the
surface contract, and a first draft of this phase's field list included
both — a reviewer correctly traced why neither belongs here. This
phase's own Goal is extraction-feasibility pre-flight: rejecting a request
no resolved collector/backend combination can satisfy, before extraction
runs. Policy/pack overrides, contract mode, and severity configuration
answer a different question — how an already-extracted comparison's
findings are classified and scored — and for the native compare/scan
CLIs specifically, that question isn't even fully answered at the point
resolve_compare_request/resolve_dump_request return: cli_compare_
receipt.resolve_and_apply() (ADR-049 Phase 5) is a separate, Click-
dependent step that runs strictly after snapshot resolution
(cli_compare_helpers.py's own compare_cmd calls _resolve_compare_
snapshots() first, _resolve_evaluation_config()/resolve_and_apply()
only afterward — confirmed by reading the real call order, not assumed),
since it depends on CLI-specific inputs (--policy/--pack/--exit-
code-scheme/a discovered .abicheck.yml) AnalysisPlanner.resolve's own
request shape has no seam for. An AnalysisPlan that tried to carry a
"resolved policy" field populated at the earlier point would therefore be
stale or incomplete for exactly the front end D1 names first — recording a
policy the run does not actually score under, which is a worse defect than
not recording one at all. AnalysisPlan stays scoped to what its Goal
actually needs (evidence/extraction satisfiability, fully knowable before
any front-end-specific configuration seam runs); policy/pack/contract
resolution keeps its own existing timing, wherever a front end's own seam
for it already sits, and is not something this phase moves earlier or
threads through planning.
ADR-063 D1's own scope is wider than compare/dump's resolution
path alone, and a first draft of this phase didn't reach the rest of
it — D1 names the Action, cli_project.py, and bundle/release fan-out
explicitly as adapters that must stop orchestrating independently.
Checking each against the real code narrows what's actually missing,
rather than treating all three as equally unconverged: cli_compare_
release.py's _run_compare_pair already routes through service.
run_compare — ADR-037 D1's existing single Tier-2 chokepoint, confirmed
by that function's own docstring — so the release fan-out's main path
is not a second implementation of compare orchestration.
That claim held only for _run_compare_pair, and a review round found
two more branches in the same file that it doesn't cover — cli_compare_
release.py is not uniformly converged, just its main path.
_collect_matrix_result() (the --probe-matrix-* release-global
build-configuration feature) calls service.compare_snapshots() directly
over a pair of empty snapshots with extra_changes — the sanctioned Tier-2
chokepoint, not the disallowed Tier-1 checker.compare() core, so this
doesn't itself trip the cli-contract gate — but it's still not through
service.run_compare/resolve_compare_request, so it never constructs an
AnalysisPlan either and has no pre-flight check for its own inputs.
_resolve_stranded_library() (the --bundle-facts-out path's own
fallback for a library missing from the normal per-pair comparison) used
to call cli_resolve._resolve_input() directly — the same Tier-2
resolution resolve_compare_request itself calls, but reached
independently, bypassing the AnalysisPlan-producing wrapper around it,
with its own bespoke ELF fallback (except Exception: ... AbiSnapshot(...))
on top. Neither was this phase's own Goal to migrate (an AnalysisPlan
pre-flight check for a probe-matrix build-config diff or a
deliberately-degrading stranded-library fallback is a real, separate
design question, not a drive-by widening of this phase's Files list) —
named here explicitly instead, as a residual this phase does not
close.
_resolve_stranded_library() was later migrated (ADR-063 Phase 8
follow-up, not this phase), narrowing the residual to one branch. Unlike
the matrix branch, a stranded library genuinely is one dump-shaped input
(a path, headers, includes, version, language, an optional depth) once
looked at correctly, so it now builds a real DumpRequest and runs it
through resolve_dump_request/execute_dump_request — gaining a real
AnalysisPlanner.resolve() pre-flight check and dropping its hand-rolled
depth=binary header-clearing special-case, while keeping its
degrade-to-ELF-only-on-failure fallback exactly as before. _collect_
matrix_result() remains unmigrated and is expected to stay that way: it
has no requested-vs-resolved evidence input for an AnalysisPlan to check
feasibility of at all (two already-empty synthetic snapshots, an
already-computed extra_changes list), so cli_compare_release.py's
release fan-out is now converged everywhere except that one branch, a gap
for a future, separately-scoped pass to close rather than a silent
omission from this plan's own accounting. bundle.py's compare_bundle()
takes already-computed per_library_results as an input rather than
calling checker.compare/service.run_compare itself, so it isn't an
orchestrator at all, just an aggregator over results the release fan-out
already produced; and the Action (action/run.sh) invokes the CLI as a
subprocess rather than importing checker.compare/dumper.dump in
Python, so once cli.py itself is the one pipeline (this plan's own
point), the Action inherits that for free as a CLI consumer. What
is still missing, confirmed by reading the same code: none of these
call sites construct an AnalysisPlan — _run_compare_pair builds and
resolves its own CompareRequest-shaped inputs without the pre-flight
PlanningError check this phase adds to resolve_compare_request, so a
release/bundle comparison can still hit the same silent-failure shape
(--build-target + pre-captured aquery, -H + unsupported collect
mode) this phase exists to close for a single-pair compare — the
existing Tier-2 chokepoint narrows the gap to "no second implementation,"
not "the same pre-flight guarantees." cli_project.py's project_plan_cmd
is a narrower case again: it only generates run-plan.json (a document
aggregate --run-plan consumes later) and neither calls compare/dump
nor constructs an AnalysisPlan itself — closing its own gap (the
--toolchain-bindings identity-probe mismatch check it already performs
is a different, narrower pre-flight than AnalysisPlan's) is not part of
this phase's scope, since it resolves a different question (build-output
coverage, not evidence-requirement satisfiability) and has no compare/
dump request to build a plan from at generation time.
The claim two paragraphs up — that _run_compare_pair "builds and
resolves its own CompareRequest-shaped inputs without the pre-flight
PlanningError check" — was stale the moment a later review round (see
the Files section below) traced the real call chain and found the
opposite: _run_compare_pair already calls service.run_compare, which
calls run_compare_request, which calls resolve_compare_request — the
exact function this phase wires to construct an AnalysisPlan. A
release/bundle comparison therefore does get this phase's pre-flight
guarantee, for free, through the shared chokepoint, the same way every
other service.run_compare() caller does; it cannot "still hit the same
silent-failure shape" this phase exists to close, because it runs through
the identical resolver a single-pair compare does. Left uncorrected here,
this paragraph and the Files section's own correction below instruct an
implementer in opposite directions — build a second, independent plan for
_run_compare_pair versus don't, it already gets one — so this paragraph
is corrected rather than left standing: _run_compare_pair needs no
change and constructs no AnalysisPlan of its own; see the Files section
for the full reasoning.
Files. abicheck/workflows/plan.py (new); service_compare_pipeline.
resolve_compare_request/service_dump_pipeline.resolve_dump_request
(construct AnalysisPlan as part of resolution — the extraction-
feasibility check only, per the Design section's own correction above; no
policy/pack resolution is constructed or reused here, since AnalysisPlan
carries none). cli_compare_release.py's _run_compare_pair
itself is not a Files entry, and a first draft of this phase's Files
list had it independently constructing and checking a second
AnalysisPlan before calling service.run_compare — a real mistake a
later review round caught. _run_compare_pair already calls service.
run_compare, which itself calls run_compare_request, which calls
resolve_compare_request — the one function this phase just wired to
construct an AnalysisPlan as part of its own resolution. Every
service.run_compare() caller, _run_compare_pair included, already
reaches that check through the shared resolver; having the frontend build
a second, independent plan performs the same preflight twice and leaves
two copies of workflow orchestration that can silently drift apart the
moment planning gains a new probe or normalization step only one of them
gets. The release/bundle fan-out gets this phase's pre-flight guarantee
for free, through the exact chokepoint it already shares with single-pair
compare — no change to cli_compare_release.py itself is needed or
correct here. (A genuinely new capability — say, inspecting the resolved
plan before the comparison runs, which _run_compare_pair cannot do
through service.run_compare's current signature — would need the
service API explicitly widened to return or accept a plan; that is a
real, separate change this phase does not need and does not attempt.)
buildsource/adapters/bazel.py (the --build-
target scoping gap gets its first real pre-flight check site here, per
its own AGENTS.md entry's recommended option 2 — reject, don't silently
scope-miss); scripts/check_architecture.py's cli-contract/
engine-cli-boundary gates (widened to confirm every service.
run_compare/run_compare_request caller — not _run_compare_pair
independently — resolves an AnalysisPlan through the one shared path,
per ADR-063 D1's own statement that these gates are "widened to check
this directly").
Tests. Two direct regression tests reproducing the exact named gaps
from AGENTS.md (--build-target with pre-captured --build-info; -H
with an incompatible collect mode) — each asserting PlanningError, not
a warning or silent continuation. A third test reproduces the identical
--build-target gap through compare-release/bundle's own fan-out (not
only single-pair compare), confirming _run_compare_pair now raises the
same PlanningError for one library in a release — through the shared
resolve_compare_request path, not a second, frontend-local plan — rather
than silently scope-missing that one library while the rest of the
release proceeds.
Acceptance criteria. Both named silent-failure gaps in AGENTS.md close
as a side effect of this phase, not as separate fixes — if either needs a
bespoke patch instead of falling out of the planner, the planner's design
is incomplete and should not be landed yet. The release/bundle fan-out
gets the identical pre-flight guarantee a single-pair compare does, not
only the pre-existing single-chokepoint property. cli_project.py's own
adapters and the Action remain out of this phase's direct scope for the
reasons stated above — named explicitly rather than left for a future
reader to rediscover by re-checking D1's adapter list against the Files
section.
Landed (first slice) — one of the two named gaps closed, the other named
out of scope, not silently dropped. abicheck/workflows/plan.py
(AnalysisPlan/SidePlan/PlanningFailure/AnalysisPlanner) and a new
PlanningError in abicheck/errors.py. AnalysisPlanner.resolve() is
wired into service_compare_pipeline.resolve_compare_request and
service_dump_pipeline.resolve_dump_request immediately after
request.validate(), before either function invokes a header-AST backend
or a build-info adapter — the pre-extraction point this phase's own Design
section requires. The --build-target + pre-captured Bazel
aquery/cquery gap is closed as "option 2" from its own known-gap entry
(reject, don't silently scope-miss): _check_bazel_target_scoping fires
when a side's build_targets is non-empty and its build_info sniffs as a
Bazel jsonproto (buildsource.inline.sniff_build_info_format), and does
not fire for the documented safe workaround (a live bazel query, no
pre-captured file) or for an ordinary, non-Bazel build_info. Every
service.run_compare()/run_compare_request() caller — the release/bundle
fan-out's _run_compare_pair included, per this phase's own Files-section
correction above — gets this guarantee for free through the shared
resolver; tests/test_analysis_plan.py's
test_reaches_through_the_shared_resolve_compare_request_chokepoint proves
it at that one chokepoint rather than needing a separate release-fan-out
fixture, since service.run_compare's own keyword surface has no
parameter to even express build_targets/a Bazel build_info in the
first place.
The known-gap entry names both dump and scan; both are now closed,
not only the resolve_compare_request/resolve_dump_request half.
scan --against's own candidate resolution (scan_engine._build_new_snapshot)
builds a raw InputSpec directly rather than a CompareRequest/
DumpRequest, so it has no AnalysisPlan of its own to resolve through —
checked against the real code rather than assumed closed by the
AnalysisPlanner wiring alone. The check itself is factored into a free
function, workflows.plan.bazel_target_scoping_failure(label, build_info,
build_targets), which _check_bazel_target_scoping wraps for the
AnalysisPlanner path and which _build_new_snapshot now calls directly.
Corrected after a first pass and two further Codex review rounds found
problems with each fix in turn: the first version raised PlanningError
inside the existing try/except AbicheckError block, which maps every
AbicheckError to click.ClickException — exit 1, not the exit-64 usage
error this combination actually is (AGENTS.md: "64 = usage error ... applies
across commands"). Fixed by raising click.UsageError directly at the point
of detection instead: since click.UsageError is not an AbicheckError, it
propagates straight past that except clause unmodified — no PlanningError
intermediate, no second except clause, and (debt-no-growth is no_growth
for this file) no added lines either. The second round found this check
never even ran on scan --dry-run/scan --artifact-set --dry-run, both of
which return their preview before _build_new_snapshot is ever called — a
dry-run could claim success (single-binary: an "UNSCOPED"-but-informational
estimate) for a request the real run then rejected. Fixed by running the
identical bazel_target_scoping_failure check in cli_scan.py itself,
before both dry-run renderers, raising the same click.UsageError.
tests/test_bazel_root_targets.py::test_scan_candidate_build_target_with_precaptured_aquery_raises_planning_error
pins the real-execution path; tests/test_bazel_root_targets_scan.py's
three new cases pin both dry-run shapes and the real-run parity case. The
baseline side of scan --against was checked and found not to reach the
Bazel adapter with build_targets at all (--against compares against an
already-produced baseline, not a second live build), so it needed no
equivalent call. The third round found the "no PlanningError
intermediate" shortcut from the first fix was itself wrong: _build_new_
snapshot also backs the typed run_scan(ScanRequest(...)) API (service_
scan.run_scan/run_scan_set), which has no Click context to catch a
click.UsageError for — a library caller with no CLI in the picture would
see a Click-framework exception leak out of a pure Python API, the same
class of layering violation PlanningError exists to prevent for dump/
compare. Fixed by raising PlanningError at the point of detection after
all (moved to just before the try/except AbicheckError block, not
inside it, so that pre-existing catch still can't recatch and remap it) and
adding the missing translation at the one real CLI boundary: cli_scan.py's
scan_cmd now catches PlanningError around its run_scan_core call and
raises click.UsageError there, mirroring the cli_resolve.py/
cli_buildsource.py pattern dump/compare already use. (The
--artifact-set path's own pre-flight bazel_target_scoping_failure call in
cli_scan.py — a second, CLI-layer-only check that runs before run_scan_set
— needed no change: it already raised click.UsageError directly from CLI
code, never from the engine, and run_scan_set's own except (ArtifactSetError,
ValueError) around run_scan_set(req) already catches a PlanningError
that reaches it too, since PlanningError is also a plain ValueError
subclass.) This closed the debt-no-growth line-count budget in the same
zero-net-growth way as the first fix: the check moved out of the try block
line-for-line, with only the raised exception's name changing.
tests/test_bazel_root_targets.py's same test now asserts PlanningError
instead of click.UsageError for the direct engine-level call, and a new
tests/test_bazel_root_targets_scan.py::
test_run_scan_typed_api_raises_planning_error_not_click_usage_error pins the
typed-API guarantee end to end (run_scan(ScanRequest(...)) raises
PlanningError, never click.UsageError), alongside the pre-existing
test_scan_cli_real_run_rejects_the_identical_combination pinning that the
CLI path still exits 64.
A fourth round found the third fix's own placement inside
_build_new_snapshot was still too late. run_scan_core runs its S3
pattern scan (find_pattern_facts) and points-of-interest build
(_build_scan_poi, which reads both sides' L0 export tables) before ever
calling _build_new_snapshot — real, if cheap, work a typed run_scan()/
run_scan_subprocess() caller (no cli_scan.py pre-flight ahead of it) paid
for on every rejected request. Fixed by moving the check to the very top of
run_scan_core, before that work, guarded on collection_for_ci_mode(
collect_mode)[1] being non-empty — the same condition, restated against the
already-resolved collect_mode rather than a raw depth string, that
workflows.plan._check_bazel_target_scoping uses for its own
depth="binary" exemption. The check was then removed from
_build_new_snapshot rather than left as a second copy: run_scan_core is
its only production caller, so the check had become pure duplication once
both were correctly guarded — and removing it caught a real, independent bug
in the process. The old _build_new_snapshot-only check had no
depth=binary exemption at all: the third round's own reasoning ("scan's
own path was already immune") verified only that cli_scan.py's
_normalize_depth_inputs prunes build_info to None at that depth for the
CLI, but never checked service_scan.run_scan, which does not prune it —
so a typed ScanRequest(depth="binary", build_info=<precaptured jsonproto>,
build_targets=(...)) was wrongly rejected by the unguarded check, the exact
false-positive class the depth=binary exemption exists to prevent.
tests/test_bazel_root_targets_scan.py gained two more cases:
test_run_scan_rejects_before_wasted_pattern_scan_and_poi_work (monkeypatches
find_pattern_facts to raise if called, proving the check now runs first) and
test_run_scan_depth_binary_exempts_the_early_bazel_scoping_check
(monkeypatches bazel_target_scoping_failure itself to raise if called,
proving the moved check's exemption actually fires rather than merely
succeeding for some unrelated reason). tests/test_bazel_root_targets.py's
own scan-side test was retired to a comment pointing at these three, since
_build_new_snapshot no longer performs the check that test exercised.
scan_engine.py's no_growth debt baseline moved 1483 → 1495 (architecture/
debt.yaml's own entry has the accounting) — the first fix's "no PlanningError
intermediate" zero-cost trick doesn't apply here, since removing the old
check's two lines is smaller than the ~14 lines the new guarded check plus
its two module-level imports needed net; a smaller diff that still states
both the check and why the old copy was removed was not found.
Not landed in this slice: the second named scenario, and the
cli-contract/engine-cli-boundary gate widening. Re-checked against
the real code (not assumed from this section's own prose), "a -H flag
accepted by a collect mode that cannot use it" does not correspond to any
isolated, currently-open known-gap entry — the one real combination
matching that description (--depth binary with an explicit header list,
which silently clears the headers to []) is intentional, already-shipped,
reviewed behavior with its own dedicated regression tests
(tests/test_cli_scan.py::test_depth_binary_clears_headers_in_scan,
tests/test_service_unit.py::test_depth_binary_clears_headers,
tests/test_typed_dump_request.py, tests/test_depth_vocabulary.py).
Turning it into a hard PlanningError would be an unreviewed behavior
change to already-tested surface, not a same-phase fix for a documented
silent failure — named here explicitly, per this file's own governing
"acknowledged gap over risky reactive patch" convention, rather than
forced to fit or silently left for a future reader to rediscover. A
genuinely new, currently-silent "input accepted by a collect mode that
cannot use it" case would be AnalysisPlanner's second check, whenever one
is found. The scripts/check_ai_readiness.py cli-contract/
engine-cli-boundary gate widening this phase's own Files section names is
also not attempted in this slice: both functions this phase changed are
already the sole callers AnalysisPlanner.resolve() needs to reach through,
so there is no second, independent call site for either gate to newly
police yet — a future phase adding one should widen them then, not this
one preemptively.
Second slice (later session): the dump --dry-run/compare --dry-run/
scan --dry-run parity gap named above is now closed, via the narrower of
the two designs that paragraph named rather than either literal option.
Neither "(a) a build_config field on the typed request objects" nor "(b)
a duplicated discovery-then-merge slice in three renderers" turned out to
be necessary once the actual seam was checked again: SidePlan (dump/
compare) already carries sources — exactly what auto-discovery needs
(discover_build_config(sources), mirroring embed_build_source's own
cfg_path = build_config or discover_build_config(raw_sources)) — and
scan's ScanRequest already carries both sources and build_config
(the explicit --config override, a seam dump/compare don't have at
the request level at all). So the fix is a widening of the check itself,
not a new field or a second discovery pass: bazel_target_scoping_failure
gained two new, defaulted keyword parameters (sources, build_config);
when the request's own build_targets is empty, it now falls back to
_discovered_config_build_targets(sources, build_config) — an explicit
build_config wins outright (mirrors cfg_path = build_config or ...),
otherwise a .abicheck.yml is auto-discovered at sources (skipped for a
pack directory, mirroring embed_build_source's own raw_sources = None
for one) — reproducing embed_build_source's own targets=list(
build_targets) if build_targets else cfg.targets precedence exactly. A
malformed config is deliberately swallowed to "no config found" here
(except ValueError: return ()) rather than raised as a second,
independently-worded error — embed_build_source already raises a
correctly-typed ValidationError for it at real-execution time, so
duplicating that diagnosis pre-flight would itself be the "second,
independently-maintained copy" drift this phase exists to avoid, for a
case that already fails loudly downstream. scan_bazel_scoping_failure
gained the identical two parameters and forwards them unchanged; its own
depth/collect-mode/header exemption logic (eleven Codex rounds' worth,
docs/contribute/known-gaps.md) is untouched, since that exemption
answers "is build_info ever consulted at all," a question independent of
where the requested target scope came from.
Every existing caller of either function keeps passing neither parameter
(both default None), so this is additive: _check_bazel_target_scoping
(the dump/compare path, via AnalysisPlanner) now passes
sources=side.sources — dump/compare have no build_config field to
pass, so only the auto-discovery half applies to them, closing their
dry-run parity for free (they resolve --dry-run through the identical
resolve_dump_request/resolve_compare_request chokepoint the real run
does, so no renderer needed touching). Three of scan's four pre-flight
call sites pass their own already-in-scope sources/build_config locals:
scan_engine.run_scan_core (both real-run and, since it runs before any
dry-run-vs-real-run branch, the dry-run path too) and cli_scan.py's two
direct bazel_target_scoping_failure call sites (scan_cmd's single-binary
pre-flight, which already ran ahead of both its real-run and --dry-run
branches, and _run_artifact_set's own pre-flight ahead of --artifact-set
discovery) — closing both the explicit---config and auto-discovered
halves for the CLI-reachable scan shapes. The fourth,
service_scan.run_scan_set, was left unwidened for one session:
service_scan.py sat exactly at the AI-readiness 2000-line hard cap, and
the widened call (5 positional args + 2 new keyword args) didn't fit
ruff format's column budget on one line — the resulting explosion would
have pushed the file 8+ lines over. Trimming unrelated content elsewhere
in that file to buy back the budget was rejected as its own kind of risk
(that file's existing content is all load-bearing review-history
documentation, not slack); adding it to LARGE_FILE_ALLOWLIST was
rejected too, since that allowlist's own comment reserves it for
pre-existing scripts//tests/ debt discovered when scanning was
widened to those trees, not a fresh production-file exemption for an
unrelated fix. So the change was reverted at that one call site and named
here instead: run_scan_set's own .abicheck.yml-only gap stayed open
for a direct typed-API call with no CLI in front of it (from
abicheck.service_scan import run_scan_set; run_scan_set(ScanRequest(...)))
— scan --artifact-set's own CLI path was unaffected, since
cli_scan._run_artifact_set's pre-flight (already widened) already ran
ahead of run_scan_set and caught the mismatch first.
Closed in a later session, by splitting service_scan.py rather than
raising its baseline. _descendant_pgids/_kill_process_tree — the
process-group kill machinery behind run_scan_subprocess/
run_scan_set_subprocess, with zero dependency on anything scan-specific
— moved to the new abicheck/workflows/scan_subprocess.py, re-exported
from service_scan.py for backward compatibility (mirroring
cxx20_pair_dialect.py's own precedent: a genuine one-directional edge,
since the worker/harness functions that actually call back into
run_scan/run_scan_set stayed in service_scan.py, avoiding the
mutual-dependency shape a full move of the subprocess harness would have
created). It lives under workflows/ rather than as a new flat
service_-prefixed root sibling, since ADR-061's frozen_root_families
closes that family to new members and service_scan.py is itself already
classified into the workflows layer via that layer's own legacy_paths.
That freed enough room for run_scan_set's own scan_bazel_scoping_failure
call to forward sources=/build_config= too, landing at a new, lowered
architecture/debt.yaml baseline (2000 → 1933) with real room left below
the hard cap. tests/test_bazel_root_targets_scan.py::
test_run_scan_set_config_sourced_target_scope_raises_planning_error pins
it, the run_scan_set sibling of test_run_scan_depth_headers_config_
sourced_target_scope_raises_planning_error. Phase 4 is now complete: all
four pre-flight call sites forward sources=/build_config=.
tests/test_analysis_plan.py::TestBazelBuildTargetScoping gained six new
cases (config-sourced scope raises for dump/compare; an explicit
build_targets still wins over a present config, with the failure message
correctly omitting the "auto-discovered" qualifier; the depth=binary
exemption still holds with a config-sourced scope; no .abicheck.yml
present is unaffected; a malformed one degrades to "no config found"
rather than raising a second error).
tests/test_bazel_root_targets.py::test_dot_abicheck_yml_build_targets_dry_run_parity
pins the closed dump --dry-run gap end to end (CLI invocation, exit 64,
same message as the pre-existing real-run sibling test immediately above
it); tests/test_bazel_root_targets_scan.py's
test_run_scan_depth_headers_config_sourced_target_scope_raises_planning_error
(renamed and re-asserted from the pre-existing ..._still_rejects_... test,
which had pinned the pre-fix click.ClickException leak this same gap
caused for the typed run_scan() API — now a clean PlanningError,
raised earlier too, from run_scan_core's own pre-flight check rather
than leaking out of _build_new_snapshot's pre-existing except
AbicheckError wart) confirms the fix at the typed-API layer as well as
the CLI.
Third slice: _run_artifact_set's own pre-flight had a narrower version
of the same false-positive/false-negative pair, specific to an unset
--depth. The initial fix for this slice added a bespoke
workflows.plan.artifact_set_bazel_scoping_failure, since
_run_artifact_set (unlike scan_cmd's single-binary path) has no
per-member resolved collect_mode at this point — each discovered
member resolves its own tier/level independently, later, inside
run_scan_set. That function's first version treated an unset --depth
as always non-"off", matching every other caller's shape, but this
false-positive-rejected a genuine no-op artifact-set request whose real
per-member risk scoring would resolve to "off". A revision treating it
as always exempt instead false-negative-accepted a seeded, high-risk
request (e.g. a public-header edit) that run_scan_core's own later,
correctly-resolved check would still reject with exit 64 — reproducing
this same phase's own dry-run/execution parity defect one level narrower
(real-run vs. real-run, not dry-run vs. real-run). Both were symptoms of
approximating a value AnalysisPlan's own design deliberately excludes
from a pre-flight check: an unset --depth only resolves to a real
collect_mode via risk scoring over the request's own seeded change.
That resolution already exists as a shared primitive —
service_scan._resolve_member_scan_level, the one estimate_artifact_set
itself calls for its own --dry-run cost totals — so the fix drops the
approximation and calls it directly: _run_artifact_set builds the same
probe ScanRequest estimate_artifact_set would (request-level fields
only, no binaries, no discovery needed), resolves the real eff_depth/
collect_mode, and hands those to the existing, already-shared
scan_bazel_scoping_failure — no bespoke artifact-set-shaped guard
needed. The now-dead artifact_set_bazel_scoping_failure was deleted
from workflows/plan.py rather than left as an unused second copy.
tests/test_bazel_root_targets_scan.py::
test_scan_cli_artifact_set_unset_depth_low_risk_seed_config_scope_is_unaffected/
test_scan_cli_artifact_set_high_risk_seed_config_scope_still_rejects pin
the no-op and risky cases respectively, keyed on a real low-risk vs.
high-risk --changed-path seed rather than an unset-vs-set --depth
alone. See docs/contribute/known-gaps.md's matching "thirteenth review
round" entry for the full account.
Adjacent, additive infrastructure landed under a separate name --
ResolvedExecutionContext ("PR 1" of a follow-up review of this whole
plan's own progress). abicheck/workflows/resolved_execution_context.py
(new): a frozen ResolvedExecutionContext composing AnalysisPlan.
operation/requested_depth (via ResolvedExecutionContext.from_plan)
with an already-resolved CompatibilityEvaluationConfig and a per-side
CompileContext mapping into one typed container, plus a
resolution_digest() fingerprint of that resolved input. Does not attempt
to replace effective_config_digest.py's two-tier digest (that module's
own docstring already gives a considered reason no single object holds
every configuration axis for every run; several rich-tier fields are
themselves comparison outcomes, not inputs a pre-execution object could
carry) -- resolution_digest() is a separate, narrower, differently-named
fingerprint of the resolved input only, and deliberately excludes
evaluation_config.provenance (Codex review, PR #1027: provenance records
how a value was selected, not the value itself, so two front ends
resolving the identical values must hash identically). Pure composition: it
re-derives nothing from AnalysisPlan, CompatibilityEvaluationConfig, or
CompileContext, and (like compatibility_evaluation_frontend.py before
it) is landed with its own primitive-level test suite
(tests/unit/workflows/test_resolved_execution_context.py) and no live caller yet -- the
type, tested, before any command is migrated to build or consume one.
The "requested/effective/available depth" axis is now closed too, via a
new EvidenceView type (second slice, same PR) -- resolving without
duplicating the one existing authority for "effective depth"
(analysis_assurance.AnalysisAssurance, necessarily a post-execution
fact: what a side's resolved snapshot actually turned out to carry, not
knowable at the point a ResolvedExecutionContext is first assembled).
EvidenceView always carries requested_depth (knowable pre-execution,
via EvidenceView.for_request) and available_depths (the static
four-rung --depth ladder, model.evidence_depth_levels.USER_DEPTHS restated
as plain values -- build-time vocabulary, not a per-run computed fact, so
stating it duplicates nothing); effective_depth/depth_satisfied stay
None until EvidenceView.from_assurance() copies them verbatim off a
real, already-computed AnalysisAssurance (read via getattr, not an
isinstance check against that heavier checker-layer type, so this
workflows-layer leaf module stays import-cycle-free) -- never re-derived.
ResolvedExecutionContext.with_assurance() returns a new context (frozen
dataclasses don't mutate) whose EvidenceView is complete, for a caller
that built a pre-execution context and only later has a real
AnalysisAssurance to attach; ResolvedExecutionContext.from_plan() also
takes an optional assurance parameter to build the full view directly,
for a caller resolving the context after a run has already completed.
resolution_digest() reads only evidence.requested_depth, never
effective_depth/depth_satisfied -- an outcome has no place in a
fingerprint of the resolved input (pinned by
test_unaffected_by_effective_depth_alone).
A first real call site landed in a follow-up slice (independent
verification pass over this plan, 2026-09-03): service_compare_pipeline.
resolve_compare_request — the one resolution implementation every front
end (compare's native CLI, the typed Python API, and directory/package
fan-out) already calls through — now builds a ResolvedExecutionContext
via ResolvedExecutionContext.from_plan() from the same AnalysisPlan it
already resolves for its ADR-063 Phase 4 pre-flight check (previously
discarded), and attaches it on the new ResolvedComparePair.
resolved_execution_context field. This is additive and behavior-preserving
— the field is optional, nothing downstream reads it, and no existing
invocation's output changes — but it closes the literal "a type nothing
outside its own tests constructs" gap for the compare path specifically:
every real compare now builds one from real production inputs, not only
from a hand-built AnalysisPlan inside a test.
Still open at the time this section was first written: the context built
at this seam carried no evaluation_config/compile_contexts (this
resolution runs before ADR-049 D7's evaluation config exists for the native
CLI, and before any per-side CompileContext is captured back out of
resolve_side_snapshot's own internals — see the field's own docstring on
ResolvedComparePair), and nothing called with_assurance(). Both since
closed, on the dump side first and later on compare — see the
"Adjacent, additive infrastructure" ledger row (4B) above for the up-to-date
account: execute_dump_request now calls with_assurance() for real
(2026-09-03) and threads a per-side compile_contexts entry
(resolution.effective_compile_context, gated by
side_effective_compile_context); resolve_compare_request gained the
identical compile_contexts threading on 2026-09-04 (ADR-063 Track 3),
reusing that same shared predicate. evaluation_config remains unresolved
on both paths — that is genuinely still open. Everything the review's
"PR 2"-onward sequence describes (a semantic consumer cutover,
FactStatus-aware detectors, and the rest) is still future work.
"PR 2" (first slice) landed the query facade a consumer migration
converges on, not the migration itself. model/semantic_ir_index.py's
SemanticIRIndex wraps a SemanticIR and answers exactly the lookups the
review's Phase 6B sketch names as prerequisite — entity(EntityId),
occurrences_for(EntityId), entities_of_kind(EntityKind) (with
functions()/variables()/records() convenience filters over it), and
fact(EntityId, fact_name). references(entity) — the review's sixth
named query — is deliberately not implemented here: it names a
graph-shaped traversal that belongs with the public-surface reference
index this same plan's Phase 3/D5 amendment governs, and adding a second,
ad hoc answer to "what does this reference" here would preempt that design
rather than serve it. Same posture as ResolvedExecutionContext: pure
composition over SemanticIR.canonical_entities()/SemanticIR.
occurrences_for(), no re-derivation, its own primitive-level test suite
(tests/test_semantic_ir_index.py), and no live caller — no detector
in diff_symbols.py/diff_types.py reads through it yet. The actual
consumer cutover (routing a detector family's matching through this index
instead of the legacy AbiSnapshot.functions/variables/types
projections, and proving the two agree via a parity test before any read
path is removed) is still the next, larger, not-yet-attempted step this
paragraph's own "still open" list already named.
Preparation step, same PR-2 umbrella: real-fixture parity between
SemanticIRIndex and the legacy function-matching key. Before any
matcher rewrite, tests/test_semantic_ir_index_function_parity.py proves,
on a real compiled fixture (two overloads plus one extern "C" function,
both header-AST backends), the concrete fact a cutover of
diff_symbols._match_old_function's exact-key join would depend on:
SemanticIRIndex.functions() sees exactly the same identity set as
AbiSnapshot.functions (no function invisible to the index, no phantom
entity absent from the legacy list), and two functions the legacy
Function.mangled-keyed join keeps apart never collapse onto one
EntityId. Explicitly not claimed by this preparation step: the
extern "C" alias-fallback tier (SymbolIdentityIndex.unique_alias_match,
_match_old_function's name: join) is not shown to already agree with
EntityId equality in general — that fallback joins on a bare, unscoped
display name, while entity_id_for_function's own extern_c tag is
scope-qualified (see model/identity.py's own docstring and
test_model_identity.py::test_extern_c_ignores_param_types). A real
fixture's extern "C" function sits at global scope on both sides, so this
asymmetry does not surface in the case tested — it is recorded here as an
open question the eventual matcher must resolve (either widen the
extern-C EntityId tag to be scope-oblivious, matching production's
unscoped alias join, or keep the alias-fallback tier as a distinct,
non-EntityId join even after the exact-key tier cuts over), not silently
assumed away by this step.
Phase 5 — the fact/capability registry (generalizes change_registry.py)¶
Landed (complete, 2026-09-02): infrastructure and the full
field-by-field population. The registry, the
fact-registry-completeness gate and the generated
docs/reference/fact-registry.md landed first (schema v30), followed by
nine conversion batches (schema v31-v40) — the case-(b) population first,
then the harder case-(a) half this section's own eligibility discussion
below exists for. KNOWN_UNCONVERTED_ELIGIBLE_FACTS is now the empty set:
every field either half of the scan finds eligible carries a Fact[T]
sibling and exactly one FactDefinition, and a newly-added eligible field
fails the gate outright rather than joining a baseline. Two pieces of
shared machinery came out of the case-(a) half —
storage/fact_codec.apply_case_a_fact_backfill (one navigator + a
CaseAFactRule table, replacing Phase 0's three open-coded legacy-load
corrections) and decode_fact_with_legacy_presence — plus fixes for five
real merge-path mutation traps the conversions surfaced. Per-batch commit
detail is not separately tracked beyond this summary and the PR history for
Phase 5's landing commits; what this phase deliberately did not attempt
(no detector branches on FactStatus; no codegen of the model field, the
serialization pair, or suppression/report wiring) is recorded here, in this
paragraph, rather than in ADR-063 — its own duplicated per-phase status
block was removed (PR 0, 2026-09-02) to keep this plan the single place
that detail lives.
Goal. A new fact requires declaring the model field plus one registry entry, not nine touched files spread across serialization, diff, suppression, and hand-maintained docs.
Design. abicheck/model/fact_registry.py: FactDefinition (id, value
type, producing backends, persisted/identity-relevant/comparable/
suppressible/reportable flags, lifecycle state per ADR-063 D7). A codegen/
validation script (scripts/gen_fact_capability_matrix.py, mirroring
scripts/gen_cli_reference.py's existing pattern) emits the backend
capability-matrix doc and a serialization-completeness check from this
registry; scripts/check_ai_readiness.py gains a fact-registry-
completeness check mirroring its existing changekind-partition/
changekind-detector checks, one level up.
Scope. This phase converts the remaining model fields Phase 0 left
alone into Fact[T] + a registry entry, mechanically, field by field —
each conversion is its own small commit (not one repository-wide diff),
so a regression is attributable to one field's conversion. This is
deliberately the availability-bearing subset of ADR-063 D7's stated
"every persisted, detected, or reported fact," per that decision's own
amendment scoping its initial realization this way — an ordinary,
always-present fact with no unavailable-vs-absent ambiguity has nothing
for this registry to resolve, and registering the full, unambiguous field
population is named there as a real but separately-justified future
extension, not this phase's own bar to clear. "Remaining
model fields" means every availability-ambiguous field on every
fact-bearing model dataclass, not only the files named model/*_facts.py
— a first draft of this phase scoped itself to that filename pattern and
missed real candidates living elsewhere: RecordType.is_final (model/
entities.py), Function.contract_attributes/Variable.alignment_bits
(model/declarations.py) are exactly the same "unavailable vs. genuinely
absent" ambiguity Phase 0 exists to close, and none of them live in a
*_facts.py-named file. The completeness check below must therefore scan
every dataclass field under model/ eligible for this conversion, not
only fields already typed Fact[T] — a check that starts from "fields
already converted" is structurally blind to a raw field nobody has
touched yet, which is exactly how this phase could report complete while
the ambiguity it exists to close still exists.
"Eligible" is not the same question as "which three annotation shapes
Phase 0 happened to use," and a first draft of this phase's check scanned
only bool/list/int | None fields — a review round correctly found
real, currently-unconverted counterexamples that shape excludes:
Function.deprecated/TypeField.default (str | None, each already
guarded by their own snapshot-level reliability flag) and Variable.access
(an enum, guarded by castxml_var_access_facts_reliable). Both meet
this phase's own stated scope — "every availability-ambiguous field ...
documented as backend-dependent" — but neither is a bare bool/list/
int | None annotation, so a check enumerating those three shapes would
report Phase 5 complete while these (and any future field shaped like
them) stay raw, overloaded values with no availability distinction.
"Does this field have a snapshot-level reliability flag" is itself too
narrow a key, and a further review round found it contradicts this
phase's own required examples two paragraphs above: RecordType.
is_final/Function.contract_attributes/Variable.alignment_bits — the
fields this phase's Scope section names specifically to prove eligibility
isn't limited to *_facts.py files — have no *_facts_reliable flag
covering any of them at all. is_final is already bool | None = None,
documented tri-state at the field level (True/False = captured,
None = not captured) with no snapshot-level flag needed, since the
field's own optionality already carries the availability signal a
bases/vtable-shaped field needs a separate flag for. A scan keyed
exclusively on flag coverage would read all three of this phase's own
named examples as ineligible, which cannot be the intended rule for a
check this phase's own text introduces those three fields to motivate.
The actual scan key is therefore field-based with an optional
availability source, not flag-required: a field is eligible when it (a)
is guarded by a snapshot-level reliability flag (the REFERENCE_FLAG_
COVERAGE-tracked case below, covering bases/vtable/is_va_list-shaped
fields whose own natural resting value can't distinguish omission from
confirmed-empty), or (b) is already tri-state at the field's own
declared type (an Optional/sentinel shape whose None/sentinel value
already means "not captured," independent of any snapshot-level flag) —
is_final/contract_attributes/alignment_bits are case (b); bases/
vtable/is_va_list before their own Phase 0 conversion were case (a).
Both cases are scanned for, not only the flag-backed one — independent of
whether the field itself happens to be a bool, a list, an int |
None, a str | None, or an enum.
"Sibling" overstates what this relationship actually is, and a first
draft of this correction implied a name-based lookup a reviewer correctly
found doesn't exist. A *_facts_reliable flag is not a 1:1,
name-derivable sibling of the one field it guards — fact_provenance.py's
own real logic shows the relationship is many-to-one and non-mechanical:
clang_deprecation_facts_reliable alone gates two fields
(Function.deprecated and is_scoped), and clang_field_initializer_
facts_reliable gates TypeField.default, a field whose own name shares
no substring with the flag's. No scan that tries to derive "which field(s)
does this flag cover" by matching names or scanning annotations can
reconstruct either relationship — and the registry lookup this check
otherwise leans on cannot find a field that was never converted or
registered in the first place, so the gap this finding closes (a field
nobody has touched yet) would survive a check keyed on an undefined
"sibling" relation exactly as it would have survived the original
three-shape enumeration. The fix is an explicit, hand-maintained inventory
— a REFERENCE_FLAG_COVERAGE: dict[str, tuple[str, ...]] (flag name →
every model field it guards, covering the already-known many-to-one cases
above) living alongside fact_registry.py, not derived at scan time — this
inventory covers case (a) (flag-backed) fields only, not case (b), since a
case (b) field's availability signal is its own declared type, not a flag
to look up — with the completeness check validated in both directions
per case: for case (a), every flag in the inventory names at least one
field that is either already converted or explicitly tracked as this
phase's remaining scope (an entry with no real field would be exactly the
kind of self-congratulatory registration this plan's own D7 completeness
principle forbids), and every case-(a) model field model/'s own
eligibility sweep finds appears under some flag in the inventory (a field
with real backend-dependent prose but no inventory entry is the "field
nobody has touched yet" case this whole correction exists to catch, now
caught by a table lookup instead of an unreliable name match); for case
(b), the sweep itself is the completeness check — every already-Optional/
sentinel-typed model field carrying a documented backend-dependence
comment (the same textual marker the sweep already looks for) is eligible
regardless of flag coverage, with no inventory entry required or expected.
Building the case-(a) inventory is this phase's own first concrete task,
not an assumption its design gets to take for granted — a field with no
flag, no tri-state declared type, and no other documented backend-dependence
is out of scope, the same way it was before this correction.
Files. abicheck/model/fact_registry.py (new); every fact-bearing
model/ dataclass module with an eligible field — model/*_facts.py,
plus model/entities.py and model/declarations.py specifically (their
is_final/contract_attributes/alignment_bits fields named above, and
any sibling field matching the same shape found during the audit this
phase's first commit performs); scripts/check_ai_readiness.py (new
check); scripts/gen_fact_capability_matrix.py (new, generates what is
today a hand-maintained capability doc). serialization.py for every
field converted, not only the registry and the model dataclass — a first
draft of this phase's touch list omitted it. Every snapshot still loads
through serialization.snapshot_from_dict()'s explicit Param/
RecordType/other constructors (unchanged by the registry, which the
Design section above already states validates serialization
completeness but does not generate mappings); each field this phase
converts needs the identical encode/decode treatment Phase 0 already gave
its three — a SCHEMA_VERSION bump, the status-to-string encoding on
write, the matching decode on read, and a legacy-schema backfill path for
a pre-conversion snapshot. Without it, a persisted snapshot reloads the
newly-converted field as a plain dict (losing its Fact[...] type) or
drops the key outright — exactly the silent-regression shape Phase 0's
own round-trip tests were written to rule out, reintroduced here one
phase later for every field this one converts. The per-field touch list
a new fact needs (stated below) is a fourth item because of this, not
only the three already named.
Tests. tests/test_fact_registry_completeness.py: every Fact[T]-
typed model field has exactly one registry entry; every registry
entry's declared producing backend is checked individually against a
real parser, not merely "at least one of them is real" — a first draft
of this test accepted an entry naming one genuine producer plus any
number of nonexistent ones, which would let the generated capability
matrix falsely advertise a backend that never actually produces the fact.
The check also runs in the other direction: for each backend, every fact
that backend's own parser actually populates has a registry entry naming
that backend — an unregistered real producer is exactly the kind of
silent drift a registry meant to be the single source of truth cannot
tolerate either. A third direction closes the gap this finding raised:
the check also scans every dataclass field under model/ for the
eligible-but-unconverted shape — a field named in the
REFERENCE_FLAG_COVERAGE inventory (per the Design section's own
corrected eligibility rule above) with no matching Fact[...] sibling,
not restricted to any particular annotation shape, and not derived from
the flag's own name by a match this codebase's real many-to-one flag/field
relationships don't support — and fails
if any exists once this phase claims completion — not only auditing the
fields the registry already knows about, so a field the conversion missed
entirely (not just one the registry forgot to register) fails this check
too. The regression fixture for this check includes at least one str |
None field (Function.deprecated/TypeField.default-shaped) and one
enum field (Variable.access-shaped) alongside the bool/list/int |
None cases, confirming the check catches a field the annotation-shape
enumeration would have missed, not only the three shapes Phase 0 happened
to use. A direct serialization.py round-trip test per converted field (the
same shape Phase 0's own tests already pin for its three fields) is
required for each field this phase converts — a freshly-extracted
snapshot round-trips through snapshot_to_dict()/snapshot_from_dict()
with the Fact[...] value and status intact, and the completeness check
above is additionally confirmed to fail (not pass vacuously) against a
converted field whose serialization pair was skipped, so the check
actually exercises the gap this finding raised rather than only the
registry-entry gap it was originally written for. Re-run the
full FP-rate/mutation-score gates once after this phase's field-by-field
conversion is complete (not per-field — the mechanical conversions don't
individually risk detector-logic drift, but the cumulative change to every
Fact[T]-typed field's representation is worth one full re-verification).
Acceptance criteria. PR #734's exact touch list (model, ELF dumper,
serialization, Change, diff, suppression, capability matrix, docs,
fixtures — nine files) shrinks, for a comparably-scoped new fact added
after this phase, to: the model dataclass field itself + one registry
entry + serialization encode/decode + parser + detector + test — six
items, not five. A first draft of this phase's acceptance criterion
named five and omitted serialization entirely, the same per-field,
hand-written snapshot_to_dict()/snapshot_from_dict() pair this
section's own serialization.py Files entry above just established is
still required per field (confirmed against the real code: the existing
ElfMetadata-enum encoding is per-path, hand-written, not a generic
tree walk a future field inherits for free) — leaving it out of the count
is the same class of gap as the already-caught "registry doesn't generate
the model field" omission below, just for a different file. The
registry does not generate the
model field — FactDefinition describes and validates an existing
Fact[...]-typed field on a model/*_facts.py dataclass; it is not a
schema from which that field is code-generated, so adding the field by
hand is still required and is explicitly counted in this acceptance
criterion rather than silently omitted from it, per this corrected draft
(a reviewer caught an earlier version of this criterion listing only four
items). Nor does it generate the serialization encode/decode pair —
the completeness check the Design section above adds only validates that
encode/decode exists for every Fact[...]-typed field, the same way it
validates registry entries, it does not write the per-field code itself.
Designing and validating real generation of the model field
itself from the registry — which would shrink the list further, to
registry entry + parser + detector + test — is out of scope for this
phase; it would need its own dataclass-field-codegen design (interacting
with from __future__ import annotations, dataclasses.field(kw_only=
True) placement, and the "new field appended last" convention every
public dataclass in this repo already follows) and is not attempted here.
The identical reasoning applies to generating the serialization pair
itself from the registry — also out of scope, also a separate, real
codegen design, not a follow-on to this phase's validator.
Demonstrate the stated (six-item) reduction directly — the phase's own PR
adds one new, real fact end-to-end as a worked example, including its
serialization round-trip, and states the
old-vs-new touch-list diff in its description.
The six-item count is also an understatement for any fact flagged
suppressible/reportable — not just the Phase-8 persisted-fact case
below — and a first draft of this phase's worked example did not say so.
A registry entry's suppressible/reportable flags are validated by
the completeness check (a flagged-suppressible fact whose ChangeKind has
no matching suppression-selector path, or a flagged-reportable fact whose
field is absent from the JSON schema, fails the check) — they are not
generated from, for the identical reason the registry does not generate
the model field, the serialization pair, or (post-Phase-8) the DTO
mapping: suppression.py's selector grammar and reporter.py's
schema/JSON emission are each their own real implementation, not a
derivable function of a boolean flag. Concretely, for a fact shaped like
AGENTS.md's own elf_binding entry (suppressible via a binding: weak
rule, reportable in the JSON changes list) the real touch list is eight
items, not six: model field + registry entry + serialization encode/
decode + parser + suppression selector/matcher entry + report/
schema field + detector + test — the two added items exist today
(suppression.py's existing per-ChangeKind matchers, reporter.py's
existing per-field JSON emission) and are unaffected by this phase; they
are simply not shrunk by it, and this plan states that explicitly rather
than letting a reader assume "suppressible"/"reportable" flags alone wire
those consumers up. Designing codegen that derives a selector/schema entry
from the registry's own flags is the identical kind of out-of-scope
follow-on as model-field/serialization/DTO-mapping generation above, not
attempted in any of the three phases.
This six-item count holds only up to Phase 8; it gains one more item
once storage v2 lands, and this plan does not hide that. Phase 8's own
design explicitly requires a distinct to_dto()/from_dto() mapping per
persisted field (that is the whole point of D8 — no asdict-based
mirror), and nothing in this registry generates that mapping either, for
the identical reason it does not generate the model field or the
serialization pair. So for a fact
added after Phase 8 ships, the real touch list is seven items — model
field + registry entry + serialization encode/decode + DTO mapping +
parser + detector + test — not
six, and this plan states that explicitly rather than letting the
six-item claim quietly go stale the moment Phase 8 lands. Registry-driven
DTO-mapping generation is the same kind of out-of-scope follow-on as
model-field/serialization generation above, not attempted in any of the
three phases.
Phase 6 — canonical SemanticIR between backends and the checker¶
LANDED (first slice): the IR, its persistence, and the hybrid merge's
reconciliation of it — no parser narrowing. Exactly the ordering this
phase's own Design section requires ("this model file, and a
primitive-level test suite pinning its shape directly ... land as their own
first step in this phase, before any of the following narrowing work"), so
the per-backend migrations have one concrete target to converge on.
Landed: abicheck/model/semantic_ir.py (SemanticIR/CanonicalEntity/
canonical_cv_qualification/semantic_ir_conflict_key);
AbiSnapshot.semantic_ir/semantic_ir_conflicts;
abicheck/storage/semantic_ir_codec.py (schema v38, the list-of-entries
encoding this section specifies) called from serialization.py's two
chokepoints; abicheck/extract/semantic_ir_merge.py wired into
dumper_hybrid.merge_snapshots(); and four test modules
(test_model_semantic_ir.py, test_semantic_ir_merge.py,
test_semantic_ir_serialization.py, test_dumper_hybrid_semantic_ir.py).
Three deliberate deviations from this section's letter, each in service of its own stated reasoning:
encode_semantic_ir/decode_semantic_irlive instorage/, not inserialization.pyitself. The reason this section moved them off the domain types was themodel -> storageedge ADR-061 forbids; astorage-resident codec has no such problem (storage -> modelis the allowed direction), matches this package's two sibling codecs (entity_id_codec.py,surface_graph_codec.py), and keepsserialization.py— already at itsarchitecture/debt.yamlno-growth baseline — to the call-site plumbing.- An ambiguous group is not "unioned verbatim" for an occurrence both
sides key identically — that phrasing has no meaning for a key
collision, and taking it literally means
setdefaultsilently discarding the overlay's entity, losing exactly the facts and conflict records this step exists to preserve (Codex review, PR #991). OneOccurrenceIdnames one occurrence by that type's own definition, so an exact key match is not a guessed pairing: it merges under the ordinary base-plus-backfill rule. What stays fail-closed is what the ambiguity is actually about — two occurrences pair only when at most one side supplies a disambiguator, never when both supply one and they disagree. (Differing keys are not themselves the refusal: castxml supplies no disambiguator at all, so the ordinary match pairs an empty key with a non-empty one.) - The "two occurrences on one side sharing a non-empty disambiguator"
ambiguity is unreachable from a real
SemanticIR, because that pair is oneOccurrenceIdand therefore one dict key. The guard stays (the matcher takes plain lists, and the invariant belongs to the caller's key type), is tested at that entry point directly, and the limit is recorded rather than papered over with a test that only looks like it covers the case.
LANDED (second slice): extract/semantic_normalizer.py, wired through the
shared ELF header-AST assembly path -- semantic_ir is now real on an
ordinary dump/compare. A re-scoping this slice makes explicit rather
than silently assuming: the plan's original Design section (below, kept
verbatim) specified normalize(raw: RawCastXmlFacts | RawClangFacts | ...)
-> SemanticIR operating on pre-canonicalization facts, from parsers
narrowed to stop resolving identity themselves. That narrowing turned out to
be unnecessary for identity specifically, because Phase 2's implementation
PR chose option (a): EntityId computed once, at parse time, and
carried as a field on the parsed declaration itself. By the time this slice
landed, both header-AST backends (dumper_castxml.py, dumper_clang.py)
already attached a real, canonically-scoped entity_id to every
RecordType/EnumType they produce, and an identically-keyed EntityId
sidecar to every typedef -- confirmed directly (entity_id_for_type/
entity_id_for_enum/entity_id_for_typedef calls exist in both backends'
records.py/enums.py and parse_typedef_entity_ids()), not assumed from
this section's own earlier text. What remained genuinely duplicated per
backend -- this phase's actual "happens once, not once per backend" goal --
is the payload canonicalization CanonicalEntity holds, not a second
identity-resolution pass. normalize_header_ast is therefore a normalizer
over each backend's own already-parsed, already-identified output, not a
raw-fact interpreter: it computes nothing about identity, only reads the
entity_id each backend already resolved and projects each declaration's
already-canonical spelling (RecordType.qualified_name or .name,
EnumType.qualified_name or .name, a typedef's own resolved underlying-type
string) into a CanonicalEntity. Backend-agnostic by construction -- both
backends expose the identical parse_types()/parse_enums()/
parse_typedefs_qualified()/parse_typedef_entity_ids() shape, verified
directly, so one function serves both.
Scope of this slice: records, enums, and typedefs only. Functions and
variables are deliberately not normalized yet: a function's canonical
signature spelling and a variable's canonical type spelling are exactly the
still-open "two backends, two readings of canonical" problem (return/
parameter type rendering, not just identity) -- reusing either backend's own
current spelling here would not unify anything, it would just carry each
backend's pre-existing disagreement into SemanticIR under a name that
claims otherwise. Constants are omitted for a different reason:
CanonicalEntity.canonical_spelling is specified as a declaration's own
type spelling, and parse_constants() captures only a constant's value
expression, never a captured type string, to canonicalize. Both gaps are
named here, not silently deferred, per this repository's own bug-class
discipline (AGENTS.md's "Fix the cause, not the instance").
Wired at one shared choke point, not per format handler.
dumper_manifest.ElfHeaderAstResult gained a semantic_ir field, computed
once inside resolve_header_ast_result() from the already-merged
types/enums/typedefs_qualified/typedef_entity_ids (after
tu_merge's own cross-TU reconciliation, so this normalizer never has to
re-implement cross-TU merge logic itself) -- this single function backs
both the legacy single-header ELF dump and a real --dump-manifest dump,
which turned out to already share this one result type and never construct
AbiSnapshot from a second, independent call site the way the plan's
original Design section (written before that consolidation) assumed. Only
dumper.py's _dump_elf (ast_result.semantic_ir) reads the new field
today; _dump_pe/_dump_macho in the same module call
parser.parse_types()/parse_enums()/... directly and are not wired in
this slice, deliberately -- dumper.py sits exactly at its
architecture/debt.yaml no-growth baseline (raised by 6 lines for Phase 2's
own closing slice already), and adding a second, inline
normalize_header_ast() call per PE/Mach-O construction site would grow it
again for a slice whose whole point is staying small. BTF/CTF/PDB are
untouched, as before -- those backends do not populate entity_id at all
yet, so this normalizer has nothing to read from them.
Real, non-empty exercise of the first slice's hybrid-merge reconciliation,
for the first time. With both castxml and clang now populating
semantic_ir, an --ast-frontend hybrid dump's merge_snapshots() step
(landed inert in the first slice, since semantic_ir was always empty)
runs its real reconciliation logic against real data. A concrete fixture
(a namespaced record, a namespaced enum, and a typedef spelling a
partially-qualified nested type) confirms the two backends resolve the
record's/enum's EntityId identically (this phase's actual point: identity
canonicalized once) while genuinely disagreeing on the typedef's
underlying-type spelling (castxml resolves it to the bare "Point"; clang
resolves it to the partially-qualified "inner::Point") -- exactly the
kind of cross-backend canonicalization gap this phase exists to make
visible, which the hybrid merge now correctly keeps castxml's value as base
and records under semantic_ir_conflicts rather than silently picking a
winner. See tests/test_semantic_ir_end_to_end.py.
Cache/versioning. No serialization.SCHEMA_VERSION bump: the v38 wire
shape for semantic_ir (list-of-entries encoding, landed in the first
slice) is unchanged -- only its content is now non-empty for a real dump.
snapshot_cache._SNAPSHOT_CACHE_VERSION is bumped (23 -> 24): a
snapshot cached by an older abicheck build would otherwise silently keep
serving semantic_ir=None forever for identical cache-key inputs, per that
constant's own documented purpose ("bumped whenever a change to the
dumping/provenance pipeline could alter a snapshot's content without
changing any of the caller-supplied cache-key inputs").
qualified_name_segments._LAMBDA_IDENTITY_FIELDS is extended to walk
semantic_ir (Codex review, PR #1001) -- the first slice's own note had
flagged this as an open question once occurrences are real, and this slice
closes it: SemanticIR.occurrences' dict KEYS (an OccurrenceId wrapping
an EntityId, a reach path the generic walk's dict-handling previously
only rewrote for a plain string key) and values are both renumbered in
step with the flat types/enums/typedefs spelling they describe.
semantic_ir_conflicts -- packed-key text a naive in-place rewrite would
corrupt -- gets its own dedicated re-key/re-value function
(model.semantic_ir.renumber_conflict_keys), including a marker present
only in a conflict's discarded value (never among the retained
declarations), assigned an ordinal as a pure continuation that never
disturbs an already-assigned real ordinal. See
tests/test_lambda_identity_semantic_ir.py.
LANDED (third slice): functions and variables are normalized, and
_dump_pe/_dump_macho are wired. extract/semantic_normalizer.
normalize_header_ast gained functions/variables parameters (default
(), so a caller that has not migrated needs no change). Unlike
records/enums/typedefs, a function's return/parameter types and a
variable's own type are not already canonical the way a resolved qualified
name is -- castxml and clang genuinely spell an identical type differently.
This slice does not invent a new canonicalization for that gap: it reuses
the two primitives entity_id_for_function/resolve_function_identity
already apply for the identical cross-backend problem --
model.signature_normalization.canonicalize_function_signature_param_type
for each parameter type (also dropping a top-level by-value cv-qualifier,
matching the mangling rule that such a qualifier is not a discriminator) and
name_classification.canonicalize_type_name for the return type (matching
entity_id_for_function's own choice for that position). A function's
canonical_spelling is "<return>(<param>, ...)" built from those two
canonicalizations; is_const/is_volatile (member-function
cv-qualification) is carried via CanonicalEntity.cv_qualification, the
field this IR already reserves for exactly that purpose. Deliberately
excluded from this slice, named rather than silently dropped: a function's
ref_qualifier/variadic status are structural facts both backends already
compute identically (not a spelling problem), and CanonicalEntity has no
dedicated slot for either -- adding one is a model-shape decision for a
future slice. A variable's canonical_spelling is
canonicalize_type_name(variable.type).
A variable's cv_qualification is NOT read from Variable.is_const
(Codex review, PR #1012, fourth round, fresh evidence) -- an earlier
revision did, mirroring the function treatment, and that mirroring was
wrong. Both header-AST backends compute is_const via a bare
word-boundary search for "const" over the WHOLE type spelling
(dumper_castxml.py's/dumper_clang.py's parse_variables()), which is
correct for that field's own narrower, pre-existing question ("would
writing through this pointer/reference SIGSEGV") but conflates a mutable
pointer to const data (const int *g -- the pointee is const, the pointer
itself is not) with a genuinely const declaration. cv_qualification is
this IR's own STRUCTURAL top-level qualification, the same distinction
model.signature_normalization's "outermost vs. pointee position"
discipline and extract/headers/castxml/type_resolution.
cv_qualifies_pointer_value already treat as load-bearing elsewhere in this
codebase -- so reusing the legacy boolean reintroduced exactly the
conflation those primitives exist to avoid. _variable_top_level_cv_
qualification derives it structurally instead: finds the last top-level
(nesting-depth-0) pointer/reference sigil in the type string, then reads
const/volatile keywords only from the text after it (or from the WHOLE
string when there is no top-level sigil at all, the by-value case) --
mirroring model.declarator_qualifiers._extract_top_level_cv's identical
depth-aware discipline, so a const inside a template argument
(vector<const int> *g) is never mistaken for this declaration's own.
No function is excluded from normalization at all -- not gated on
Function.is_compiler_generated, nor on a synthetic ctor/dtor mangled key
(Codex review, PR #1012, three rounds, fresh evidence each time). The
first revision of this slice skipped every compiler-generated function --
too broad (a compiler-generated function with a real mangled name, e.g. a
synthesized operator=, has no cross-backend hazard at all: AbiSnapshot.
functions already includes it, so excluding it from semantic_ir was
itself the representation-disagreement this IR exists to avoid) and it also
missed the real hazard. The second revision re-gated on the actual hazard --
a function whose mangled is castxml's own synthetic ctor/dtor snapshot key
(model.synthetic_key.is_synthetic_ctor_key/is_synthetic_dtor_key --
assigned whenever castxml "may omit a constructor/destructor's real mangled
name even for a public user-declared" one, per extract/headers/castxml/
functions.py's own function_mangled_name docstring, so this can hit a
genuinely hand-written, non-implicit constructor too, not only an implicit
one) is NOT a stable cross-backend identity, since dumper_hybrid.
_merge_functions can rewrite it to a real clang-matched one during a hybrid
merge, a rewrite this per-backend normalizer (which runs before that merge
step) cannot see -- but that revision still unconditionally excluded such a
function even from a plain, non-hybrid dump, where no rewrite step exists
at all and the occurrence is exactly as real as any other function's.
The third round closes the hazard at its actual source instead of
dodging it in the normalizer: _merge_functions now returns (via an
output-param dict) every old_entity_id -> new_entity_id substitution its
own ctor/dtor structural match makes, and dumper_hybrid.merge_snapshots()
applies the identical substitution to castxml_snap.semantic_ir (through
_rewrite_semantic_ir_entity_ids, the same primitive the Mach-O
mangled-name fix below introduces) before reconciling it with clang's --
so a matched declaration is never left keyed under its retired synthetic
identity in one representation while merged_functions already carries the
real one. With the hazard closed at the merge step, the normalizer excludes
nothing: every function is normalized unconditionally, synthetic-keyed and
compiler-generated ones included. Confirmed as a real, not merely
theoretical, gap at every round by the real castxml/clang end-to-end
fixture in tests/test_semantic_ir_end_to_end.py (first round: Point's
implicit constructors/destructor, genuinely synthetic-keyed, surfaced with
an empty leaf_name; its real-mangled copy/move assignment operators were
then wrongly excluded too until the second round) and by a new unit test in
tests/test_dumper_hybrid_semantic_ir.py pinning the third round's fix
directly (a matched synthetic ctor's semantic_ir occurrence ends up under
the real, rewritten key, not duplicated or left stale under the synthetic
one).
The unresolved-type-sentinel check tracks nesting depth, not a plain
substring test, and definitely not exact equality (Codex review, PR #1012,
second and third rounds, fresh evidence each time). castxml's own type
resolver (extract/headers/castxml/type_resolution.py's
type_name_uncached) composes an unresolved nested type into the enclosing
spelling -- a pointer/reference/array wrapping an unresolvable pointee
renders as "?*"/"?&"/"?[]", a cv-qualified one as "const ?" -- so an
exact-equality check (correct only for the typedef branch's underlying
value, always the outermost type_name() result with nothing further
wrapped around it) misses every composite shape for a function/parameter/
variable type, and shares the identical gap for a typedef whose underlying
type is itself one of these composite shapes (second round: fixed with a
plain substring test). The third round found a plain substring test is
itself unsafe: a real, fully-resolved type spelling can legally contain a
literal "?" -- clang emits one verbatim for a dependent, unevaluated
ternary expression inside a decltype(...) (e.g. a non-type template
argument's own spelling, "S<decltype(flag ? A{} : B{})>"), which a
substring test wrongly marks Fact.failed(...), discarding real canonical
evidence. _has_unresolved_component now tracks nesting depth over
()/[]/<> instead: castxml's own sentinel-composing recursion never
emits a "?" inside such a grouping -- it only ever prepends/appends a
bare pointer/reference sigil, array brackets, or a cv keyword directly
beside it -- while a ternary's "?" is, by C++ grammar, only reachable
inside an expression context that for a type spelling means already being
inside decltype(...)'s parens or a template argument list's angle
brackets. So a "?" found at nesting depth zero is the sentinel; one found
at depth > 0 is real, resolved evidence. Replaces the exact-equality check
everywhere, typedef branch included.
One wrapper is a deliberate, named exception to plain depth-tracking
(Codex review, PR #1012, fourth round, fresh evidence): castxml's own
_Atomic(...) composition. type_name_uncached's AtomicType branch
renders an unresolved wrapped type as the literal "_Atomic(?)" --
genuine sentinel output, using a REAL parenthesis pair as part of the
resolver's own grammar, not an expression context a real, resolved "?"
could ever be found inside instead. Depth-tracking alone treats that "("
exactly like a decltype(...)'s, hiding the sentinel at depth 1 and
wrongly reporting the composite as resolved. "_Atomic(" is recognized as
a transparent token instead -- skipped without incrementing depth -- so a
sentinel directly inside it is still caught at its effective depth 0, the
same treatment a bare "?" already gets; _Atomic(...) is also real,
valid C11 syntax for an otherwise fully-resolved type ("_Atomic(int)"),
which this special-casing does not disturb.
**A declarator-grouping paren is transparent to _variable_top_level_cv_
qualification's own sigil search, not depth-increasing (Codex review, PR
1012, fifth round, fresh evidence).** A const function-pointer/¶
pointer-to-array/pointer-to-member-function variable wraps its own sigil in
a real, syntactic (...) group -- clang spells "int (*const)(int)" for
int (* const fp)(int) -- which an ordinary opaque-paren depth count
treats exactly like a parameter list, hiding the sigil at depth 1 and
reporting no qualification at all. Reuses model.declarator_qualifiers.
_is_declarator_group, the identical classifier signature_normalization.
canonicalize_function_signature_param_type already applies for this exact
shape. The same round also extended the cv-keyword matcher to recognize
restrict -- reverted the very next round (see below).
Sixth round, two further fixes, one addition and one revert (fresh
evidence each). (1) A pointer-to-member-function's own trailing parameter
list now ends the region _variable_top_level_cv_qualification reads
qualifiers from, reusing _split_at_trailing_param_list (the identical
primitive canonicalize_function_signature_param_type already uses for
this split): for "void (C::*)(int) const", the const after the
parameter list qualifies the POINTED-TO member function, not the pointer
variable itself, and the fifth round's naive whole-suffix scan wrongly
attributed it to the variable, reporting an identical ("const",) for both
a mutable and a genuinely const member-function pointer. (2) The fifth
round's restrict recognition is reverted: CanonicalEntity.
cv_qualification's vocabulary names restrict alongside const/
volatile, but recognizing it via a plain text scan is backend-asymmetric
-- clang's own qualType spells it verbatim ("int *restrict"), while
castxml's type_name_uncached never emits the word at all, by castxml's
own deliberate choice (unlike a function parameter's Param.is_restrict,
which both backends already populate structurally). A castxml-produced
entity therefore claimed a CONFIRMED () for a qualifier its own backend
structurally cannot see, and merge_semantic_ir's backfill (which only
ever backfills a non-present base fact) treated that false confirmation
identically to a genuine, deliberate absence -- a hybrid dump then
downgraded clang's real ("restrict",) to a mere two-sided disagreement
against castxml's structurally-blind (), discarding it as the merged/
authoritative value instead of backfilling it. Fact[tuple[str, ...]]
cannot express "confirmed for two of three qualifiers, blind to the third"
within one fact; a real fix needs a structural, reliability-tracked
Variable.is_restrict fact populated by both backends the way Param.
is_restrict already is (resolve_cv_restrict/clang_param_is_restrict,
both directly reusable for a variable's own type id/node), almost
certainly with its own AbiSnapshot reliability flag mirroring
clang_restrict_facts_reliable -- a model-shape decision for a future
slice, not a normalizer-only change, the same reasoning already given for
leaving a function's ref_qualifier/variadic status out of this IR.
restrict is therefore left unrecognized (never reported) for a variable
today, rather than reported unreliably.
dumper.py's _dump_pe/_dump_macho (the two PE/Mach-O construction sites
the second slice deliberately left unwired, per that slice's own note) now
populate semantic_ir too, via a new shared choke point,
extract/header_ast_fields.parse_header_ast_fields -- a new leaf module
(not inlined in either already-capped dumper.py/dumper_manifest.py)
that runs a single-parser's own parse_*() methods once each and projects
the result through normalize_header_ast, structurally typed
(_HeaderAstParser, a Protocol) rather than importing
dumper_castxml._CastxmlParser/dumper_clang._ClangAstParser directly,
which would be an extract -> (unclassified flat root module) edge the
architecture gate forbids for a migrated package. dumper.py's
architecture/debt.yaml no-growth baseline is raised by 7 for this (see
that entry's own rationale for the exact accounting). extract/
semantic_ir_merge.py's reconciliation needed no code change to handle a
function-populated IR -- it is generic over CanonicalEntity's Fact
fields, not specialized to any entity kind -- and this slice's own real
castxml/clang/hybrid fixture (a function taking a record by const reference)
is the first non-synthetic proof of that: the hybrid merge matches the
function's shared, real-mangled-name EntityId one-to-one and records the
identical kind of namespace-qualification-spelling conflict the typedef
case already has, not a new failure mode.
LANDED (fourth slice): constants. extract/semantic_normalizer.
normalize_header_ast gained constants/constant_entity_ids parameters
(default {}, so a caller that has not migrated needs no change), mirroring
typedefs_qualified/typedef_entity_ids's own pairing. Both header-AST
backends already attach a real EntityId to every public constant
(parse_constant_entity_ids(), Phase 2) -- the identity half of this
slice's work was already done before it landed; what remained was wiring
the existing maps into this normalizer at all, plus the two call sites
(dumper_manifest.resolve_header_ast_result, extract/header_ast_fields.
parse_header_ast_fields) that already compute both maps for the legacy
AbiSnapshot.constants/constant_entity_ids fields. Deliberately no
canonicalization is applied to the value text. Every prior slice's
canonical_spelling closes a real, observed cross-backend type-spelling
disagreement ("char const*" vs. "char const *", a bare "Point" vs.
"inner::Point"); a constant's parsed representation carries only its value
expression, never a captured type string, and no comparable disagreement in
value spelling has been observed between the two backends. canonical_
spelling is therefore the raw parse_constants() string, unchanged --
mirroring diff_symbols._diff_constants's own CONSTANT_CHANGED detector,
which has always compared the two backends' raw value strings with a plain
!= and never canonicalized either side. Inventing a value-spelling
canonicalizer with no known target divergence to fix would be exactly the
kind of speculative heuristic this codebase's own bug-class discipline warns
against elsewhere (see _type_index_items's/_diff_constants's own
docstrings on an identity heuristic falsified twice, and the "attempted
twice, reverted twice" discipline that follows from it) -- a canonicalizer
in search of a bug to fix, not a fix for an observed one. A constant carries
no cv_qualification/template_arguments: it has no captured type for
either fact to describe. snapshot_cache._SNAPSHOT_CACHE_VERSION is bumped
(25 -> 26) for the same reason every prior slice bumped it: a snapshot
cached by an older abicheck build would otherwise silently keep serving a
semantic_ir with no constant occurrences forever for identical
cache-key inputs. dumper_scoping._scoped_semantic_ir needed no change:
both backends already scope parse_constants()/parse_constant_entity_ids()
to the public-header surface at parse time (the same _have_public_set
filtering pass Phase 2 already applies), so there is no dependency-header
constant occurrence for that function to additionally exclude, unlike
functions/variables/types/enums which are scoped post-hoc.
Fourth slice, three fixes (Codex review, seventh round, fresh evidence
each). (1) merged.constants deliberately stays castxml_snap.constants
verbatim (a pre-existing exception, unrelated to this slice -- see that
field's own comment), but merge_semantic_ir is generic over every
EntityKind and appends an unmatched clang-only CONSTANT occurrence into
the merged IR unconditionally, the same as it correctly does for a
clang-only function/variable/type/enum (all of which genuinely ARE unioned
into their own flat fields, unlike constants). Left unfiltered, a
clang-only constant would surface through semantic_ir with no
corresponding entry in merged.constants/constant_entity_ids at all --
dumper_hybrid._drop_unmatched_constant_occurrences closes the gap,
filtering the merged IR's CONSTANT occurrences (and their conflict-key
entries) to the identical kept-entity-id set constant_entity_ids's own
union-then-filter already computes. (2) dumper_clang_expr._expr_
fingerprint stamps a compound constant initializer (anything beyond a
lone literal) with a build-stable STRUCTURAL fingerprint
("expr:" + sha256(...)[:16]), not a spelling of the source text -- that
module's own docstring is explicit that cross-backend constant values
are not expected to match for this case. Publishing the fingerprint as
Fact.present(...) made merge_semantic_ir report a spurious conflict
against castxml's real initializer text for every unchanged compound
constant; normalize_header_ast now recognizes the fingerprint's own
literal prefix structurally (by the value's own shape, not by branching on
producer == "clang") and marks it Fact.unsupported() instead -- the
same "state genuinely incomparable evidence honestly" discipline ADR-063
Phase 0 established. (3) _has_unresolved_component's depth tracker is now
a bracket-KIND-aware stack, not a flat counter: for a resolved dependent
type like "S<(N >> 1 ? 1 : 2)>", a flat counter decrements once per >
character, so the real right-shift operator's ">>" wrongly drops the
running depth to zero WHILE STILL inside the (...) grouping, misreading
the ternary's own "?" as the sentinel at real depth > 0 (confirmed
against a real clang++ -Xclang -ast-dump=json repro for exactly this
qualType shape on a dependent function return/variable type). A ">"
now only legitimately closes a template level when the innermost
still-open bracket is itself a "<" (vector<vector<int>>'s own ">>"
still correctly closes two); when the innermost open bracket is a
"("/"[" instead, a ">" is a real, resolved operator character
belonging to that expression and is left untouched. A genuinely ambiguous
bare "<" (a real less-than operator, not a template open) remains an
accepted, pre-existing residual this fix does not attempt to solve -- doing
so needs real expression parsing, and no concrete evidence of that
specific shape has been found the way this ">>" shape was.
Known, accepted limitation, documented rather than fixed (Codex review,
eighth round, fresh evidence): a top-level qualifier hidden behind a
typedef is not detected. For typedef int * const ConstPtr; extern
ConstPtr p;, both backends pass _variable_top_level_cv_qualification
the ALIAS spelling ("ConstPtr") -- neither castxml's type_name() nor
clang's plain (sugared) qualType resolves through the typedef the way
this function's text scan would need to see the real int * const
underneath, so it finds no sigil/keyword and reports a confirmed ()
even though the variable's real top-level qualification is ("const",).
A real fix needs new structural evidence this function has no access to
today: clang's separate desugaredQualType string (already used
elsewhere for the identical problem on a field), and castxml's own
structured, typedef-following resolver (resolve_cv_restrict, which
already walks Typedef/ElaboratedType nodes) -- neither is currently
threaded onto Variable for this function to read. Adding either is a
new, per-backend model field (mirroring Param.is_restrict's own
structural-fact treatment), not a normalizer-only change -- the identical
"model-shape decision for a future slice" conclusion this section already
reached for restrict. Left undetected rather than guessed at from text
that cannot see through the alias; pinned by a dedicated regression test
(test_normalize_header_ast_typedef_hidden_qualifier_is_a_known_limitation)
so a future fix has something that starts failing once it lands.
A castxml opaque FunctionType tag is Fact.unsupported(), not
Fact.present(...) (Codex review, ninth round, fresh evidence).
castxml's resolver has no dedicated rendering for an anonymous
FunctionType node (unlike Struct/Class/Union/Typedef/... which
all have one), so a direct function-pointer parameter/variable/return type
resolves to the literal opaque tag string "FunctionType" (wrapped in
whatever sigil surrounds it, e.g. "FunctionType*"), never a real
declarator spelling the way clang's own "void (*)(int)" is -- the
identical shape idioms._is_callback_type already checks for elsewhere in
this codebase ("FunctionType" in type_str). This is NOT
_has_unresolved_component's unresolved-type sentinel case: the resolver
ran and produced a real, structurally-final answer, it just cannot express
one for this shape -- exactly FactStatus.UNSUPPORTED's own definition
("this producer cannot express this family at all... a different producer
might"), not FAILED. Publishing the opaque tag as Fact.present made a
hybrid merge report a spurious conflict against clang's real, useful
spelling for an unchanged callback declaration; _function_spelling_fact/
_variable_spelling_fact now check for the marker (on the RAW components,
same as the unresolved-sentinel check) and mark Fact.unsupported()
instead, which merge_semantic_ir's backfill treats as absent and
correctly prefers clang's real evidence with no conflict recorded.
Two follow-up fixes to the FunctionType/expr: artifact checks, plus a
file split (Codex review, tenth/eleventh rounds, fresh evidence each).
(1) The clang expr-fingerprint check now matches the FULL shape
(^expr:[0-9a-f]{16}$) rather than merely the "expr:" prefix -- a plain
prefix test also matched castxml's own raw, verbatim source-text
initializer whenever it happened to spell a qualified name whose next
component is literally expr ("expr::NAMESPACE_VALUE"), mirroring
diff_default_value_reliability._is_expr_fingerprint's identical fix for
the identical mistake (PR #720). (2) The FunctionType check is anchored
to the WHOLE (cv/sigil-stripped) string and gated on producer ==
"castxml", not a bare substring test: a naive "FunctionType" in
raw_type also matched a real, legitimately-named type
("MyFunctionTypeWrapper*") and fired for clang too, even though clang
never emits this literal tag text at all. Neither fix fully eliminates
every theoretical collision (a real type named EXACTLY "FunctionType"
still collides, the same accepted residual idioms._is_callback_type
already carries) -- closing that needs real structural evidence from the
parser, not a normalizer-only text fix. These three artifact-recognition
functions (has_unresolved_component, is_castxml_opaque_function_type,
CLANG_EXPR_FINGERPRINT_RE) were split out into a new sibling leaf module,
extract/semantic_normalizer_artifacts.py, once their accumulated
docstrings pushed semantic_normalizer.py itself past the AI-readiness
gate's 800-line cap for a new file -- mirroring model.declarator_
qualifiers.py's own split from model.signature_normalization.py for the
identical reason.
The sigil-finding scan in _variable_top_level_cv_qualification gets
the identical bracket-KIND-aware stack fix (Codex review, twelfth round,
fresh evidence), for the same underlying primitive bug found in a
different function. For clang's own spelling of template<int N> extern
S<(N < 0)> * const gp ("S<(N < 0)> *const"), a flat depth counter
treats the comparison < as another template opener; after the real
)/> closers the running depth never returns to zero, so the sigil
search never finds the real top-level * at all, silently reporting no
qualification for a genuinely const pointer. Fixed symmetrically with the
existing ">" rule: a "<" only pushes a new bracket level when the
innermost still-open entry is NOT itself a "("/"[" -- a real
comparison < sitting inside an already-open paren/bracket expression is
left untouched the same way its own closing > already is, so the two
never spuriously push a level a later real ) would then incorrectly pop
instead of the paren it actually closes. Applied proactively to
has_unresolved_component too (same file split, extract/semantic_
normalizer_artifacts.py), since it carries the identical latent bug for
the identical shape even though no concrete failure was reported there --
"fix the cause, not the instance" applied to a shared primitive weakness
rather than only the one call site that happened to be caught.
The opaque-FunctionType regex gets one more shape (Codex review,
thirteenth round, fresh evidence): a cv-qualifier can also appear AFTER a
pointer/reference sigil, not only before the tag. type_resolution.py's
own CvQualifiedType branch renders a cv-qualified POINTER VALUE (not a
cv-qualified pointee) as a SUFFIX, matching this codebase's own "T *
const" convention elsewhere -- so a const function-pointer's opaque
fallback resolves to "FunctionType* const", not "const FunctionType*".
An earlier revision of the anchored regex only recognized a LEADING
cv-keyword, so this suffix-qualified shape was wrongly published as
present, conflicting with clang's real spelling in a hybrid dump of an
unchanged const callback. The regex now allows a cv-keyword run after
EVERY sigil, since castxml's recursive wrapping can in principle nest more
than one ("FunctionType** const volatile").
A clang Python-bool-derived literal constant is also Fact.unsupported(),
not Fact.present(...) (Codex review, fourteenth round, fresh evidence).
clang's compound-initializer-expression parser stringifies a captured
Python bool constant value with plain str(...), which spells True/
False with a capital first letter -- not the C/C++ source spelling
(true/false) either backend's own literal text would show, and not a
spelling castxml ever produces for the same declaration. Left as
Fact.present("True"), an unchanged boolean constant read identically by
both backends' real evidence still reported a spurious hybrid conflict the
moment clang's own producer-specific stringification diverged from
castxml's genuine source-text capture -- the identical class of bug the
expr: fingerprint and opaque-FunctionType fixes above already closed
for their own producer-specific artifacts, just for a third one this
normalizer had not yet recognized. The fix checks for the exact two-value
set {"True", "False"} rather than a case-insensitive comparison or a
substring test, since a real declared string/identifier constant that
happens to spell exactly True or False as its own source text is
legitimate, comparable evidence, not an artifact to discard. The
decimal-integer/character/float-literal residual this normalizer still
cannot distinguish from a genuinely spelled literal (e.g. castxml
composing 0x10 and clang composing 16 for an unchanged constant) is
a known, accepted, documented limitation, not attempted here: unlike the
boolean case, there is no structural signal in the value text alone that
marks a normalized-but-equivalent numeric spelling as producer-specific
rather than a real value difference worth reporting, and closing it
correctly needs either a shared literal-grammar normalizer both backends
route through or a new structural fact recording each backend's original,
un-normalized token -- a model-shape decision for a future slice, the
same conclusion already reached for restrict and the typedef-hidden
top-level qualifier above. Pinned by a dedicated regression test (a
decimal integer constant continues to report Fact.present(...)
unchanged) so a future fix has something that starts failing once it
lands.
tests/test_semantic_normalizer.py split a second time, mirroring the
production-code split (Codex review round, same principle as the
eleventh-round production split above). Adding the two regression tests
for the boolean-literal fix pushed the test file to 1217 lines, past the
AI-readiness gate's 1200-line cap for a new test file. The 21 tests
exercising semantic_normalizer_artifacts.py's own primitives (the
unresolved-type sentinel, the opaque FunctionType tag, the clang
expression fingerprint, and the two boolean-literal tests just added) moved
to a new sibling file, tests/test_semantic_normalizer_artifacts.py,
leaving the general projection/canonicalization tests in the original file.
The bool-literal exception is gated on producer == "clang" (Codex
review, fifteenth round, fresh evidence): "True"/"False" are legal
C++ identifier spellings, not exclusively a clang artifact. The
fourteenth-round fix above discarded any constant value spelling exactly
"True"/"False" regardless of which backend produced it, but
constexpr bool True = true; constexpr bool k = True; is real,
case-sensitive, compilable C++ -- castxml's verbatim init text for k
genuinely reads "True" too, and the producer-agnostic check discarded
that real castxml evidence outright (in a castxml-only snapshot) or left
no way for a hybrid merge to backfill it (clang's own value for the same
identifier reference is itself an unsupported expression fingerprint, not
a bare "True"/"False" the old check could match). Restricting the
exception to producer == "clang" closes the gap: the capitalization
signal is safe once scoped to clang's own str(bool)-derived
stringification specifically, since no clang-parsed boolean literal is
ever spelled this way -- only a clang-parsed identifier REFERENCE could
theoretically collide, and that case already routes through the
expression-fingerprint branch instead (dumper_clang_expr._initializer_value
only stringifies a bare bool for a literal, not an identifier
reference). Pinned by test_normalize_header_ast_castxml_true_named_identifier_stays_present.
A Mach-O plain-C hybrid dump's entity_id mismatched castxml's own
extern "C" tag with clang's mangled-name tag (Codex review, fifteenth
round, fresh evidence) -- fixed at the identity source, not by patching
the Mach-O semantic_ir rewrite. A genuinely plain-C compilation unit
has no LinkageSpecDecl at all (that AST node exists only in C++'s
grammar), so dumper_clang.py's entry.extern_c -- set only by walking
into one -- never becomes True for a plain-C declaration. The existing
raw_mangled == name fallback still recovers this on most platforms
(clang's mangledName for a plain-C declaration is otherwise the bare
source name), but Mach-O is the one platform where it doesn't: Darwin's
linker prepends a leading underscore to every global symbol ("_foo" for
source-level "foo"), so clang's own mangledName for the identical
plain-C declaration is "_foo", and the bare-equality check never
matches. Left unfixed, such a declaration's entity_id stayed tagged
("mangled", "_foo") while castxml -- which observes no mangledName
XML attribute at all for a plain-C Function/global Variable -- tags
the identical declaration ("extern_c",), so merge_semantic_ir's
bare-EntityId matching never recognized the two as one declaration and
retained it TWICE in the merged semantic_ir, even though the flat
functions/variables lists (matched on the bare mangled string, not
EntityId, via dumper_hybrid._merge_functions's own clang_by_mangled
lookup) already unified it. dumper_hybrid.py's own Mach-O
underscore-stripping rewrite (_macho_normalize_semantic_ir) could not
have closed this even in principle: it only re-spells the "mangled"
tag's VALUE, never its KIND, so a ("mangled", "_foo") identity stays
tagged "mangled" no matter how the trailing string is normalized. The
real fix is upstream, in both extract.headers.clang.functions.
parse_functions and dumper_clang._ClangAstParser.parse_variables:
raw_mangled == name becomes name in symbol_candidates(raw_mangled),
reusing extract.headers.clang.context.symbol_candidates -- the identical
tolerant-match helper visibility() already uses for this exact Mach-O
underscore quirk -- instead of a second, independently-spelled check.
Pinned by tests/test_dumper_clang_extern_c_identity.py (a new, dedicated
file rather than added to test_dumper_clang.py, which already sits at
its architecture/debt.yaml no_growth baseline).
Two more real findings, sixteenth round, fresh evidence each: the
Darwin de-prefixing fallback needs a target gate, and the opaque-
FunctionType regex needs a sized-array suffix.
First: the fifteenth round's name in symbol_candidates(raw_mangled)
fallback was target-agnostic -- it fired on every platform, not only
Darwin. On a NON-Darwin target, a real, explicit asm("_foo") label
genuinely produces raw_mangled == "_foo" while name == "foo" and
entry.extern_c stays False -- that is a real, distinct mangled
identity (an asm label), not a linker-decoration artifact, and castxml's
own resolver keeps the identical declaration tagged ("mangled",
"_foo"). The ungated fallback misread this as C linkage too, discarding
the genuine mangled identity clang correctly observed and producing the
identical class of cross-backend EntityId mismatch the fifteenth
round's own fix was meant to close -- just with the two backends'
tags swapped. The fix adds extract.headers.clang.context.
is_darwin_target(target_triple) (checking for "apple" in the lowercased
triple, covering both the ...-apple-darwin... and ...-apple-macosx...
spellings this codebase's own test fixtures already use) and requires it
alongside the de-prefixed match -- but NOT alongside the pre-existing
plain raw_mangled == name equality, which holds on every platform
regardless of Darwin and must stay ungated (an early revision of this
very fix wrapped the WHOLE is_extern_c expression in the Darwin gate,
which broke the ordinary, non-underscored plain-C case,
test_parse_functions_extern_c_via_mangled_equals_name, on every
platform including Darwin itself -- caught immediately by that
pre-existing regression test, not shipped). symbol_candidates itself
stays target-agnostic, since it also backs visibility()'s pure
export-table-membership check, where trying the de-prefixed form is
always safe regardless of platform; only the identity decision built on
top of it needed the gate. Pinned by two new sibling tests proving the
non-Darwin asm-label case stays ("mangled", "_foo") (function and
variable), plus a target_triple=None case (a synthetic/unprobeable AST
must default to the same conservative "not Darwin" answer, never assume
Darwin from an absence of evidence) and a small parametrized unit test for
is_darwin_target itself.
Second: castxml's ArrayType renderer spells a fixed-size array of
function pointers (void (*callbacks[3])(int)) as "FunctionType*[3]"
-- a SIZED array suffix, not only the unsized "[]" the anchored regex
already recognized (an unsized array, e.g. a function-pointer array
PARAMETER, decays to a pointer with no bound). An earlier revision of
_CASTXML_OPAQUE_FUNCTION_TYPE_RE matched only "[]", so a sized
function-pointer array's own opaque-tag fallback was wrongly published as
Fact.present, conflicting in a hybrid dump against clang's real,
complete declarator for an unchanged callback array -- the identical
class of "the opaque fallback's own contribution is always exactly the
bare tag text with nothing else glued onto it" reasoning the pointer/
reference/cv-suffix branches above already apply, just for a bracket
shape not yet covered. Fixed by widening \[\] to \[\d*\] in the regex.
Pinned by a new regression test using a sized-array Variable.
A third wrapper shape, seventeenth round, fresh evidence: _Atomic(...)
can enclose the whole opaque spelling too. castxml's resolver composes
_Atomic(void (*)(int)) callback into "_Atomic(FunctionType*)" -- the
identical "outer wrapping node, not glued onto the tag" shape the
pointer/cv/array branches above already accept, just realized as a real
paren pair instead of a sigil or keyword (mirroring
has_unresolved_component's own pre-existing _ATOMIC_WRAPPER_PREFIX
transparent-wrapper treatment for the identical _Atomic(...)
composition on the unresolved-sentinel side). An earlier revision of
_CASTXML_OPAQUE_FUNCTION_TYPE_RE had no _Atomic(...) branch at all,
so this shape fell through as a real, present spelling, conflicting in a
hybrid dump against clang's complete _Atomic(void (*)(int)) spelling
while keeping the opaque, useless base value. Fixed by splitting the
regex into a reusable inner pattern
(_CASTXML_OPAQUE_FUNCTION_TYPE_INNER) and matching it either bare or
wrapped in _Atomic(...), rather than trying to fold the wrapper into
one already-dense pattern. Pinned by a new regression test using an
_Atomic-wrapped Variable.
The _Atomic(...)-wrapped form can itself be wrapped again, eighteenth
round, fresh evidence -- fixed by factoring the wrapping pattern out and
applying it uniformly around both atoms, not by chasing each new shape
with another alternative. const _Atomic(void (*)(int)) callback,
_Atomic(void (*)(int)) *callback, and an array of atomic callbacks
render as "const _Atomic(FunctionType*)", "_Atomic(FunctionType*)*",
and "_Atomic(FunctionType*)[3]" respectively -- a cv-prefix/sigil/array
wrapper OUTSIDE the _Atomic(...) parens, on top of the wrapper already
recognized INSIDE them. The seventeenth round's fix treated _Atomic(...)
as only ever the WHOLE string (its own two-alternative regex had no
wrapping outside either branch), so none of these further-wrapped shapes
matched, and the normalizer published castxml's opaque fallback as
present -- the same false-conflict-in-a-hybrid-dump failure mode every
prior round in this thread has fixed, just for another wrapping position.
Rather than adding a THIRD alternative (and inevitably a fourth once a
combination of wrapper positions surfaces next), the fix restructures the
regex around two reusable fragments -- _CASTXML_OPAQUE_FUNCTION_TYPE_
CV_PREFIX (the leading const/volatile run) and _CASTXML_OPAQUE_
FUNCTION_TYPE_WRAPPING (the repeating sigil/array-with-trailing-cv
suffix) -- and an _CASTXML_OPAQUE_FUNCTION_TYPE_ATOM alternation
(FunctionType bare, or _Atomic(FunctionType + the SAME wrapping
pattern + )) that the cv-prefix and wrapping are then applied around
UNIFORMLY, once, regardless of which atom matched. A nested SECOND
_Atomic(...) is deliberately not modeled -- this normalizer has no
observed evidence castxml ever emits one -- so the wrapping pattern used
inside the parens is the same one used outside them, just not applied
recursively. Pinned by a parametrized regression test covering all three
newly-reported shapes.
Nineteenth round, fresh evidence, two independent findings on the SAME commit -- both narrowing the Darwin-gated extern-"C" de-prefix fallback further.
First: is_darwin_target checked only for an "apple" VENDOR substring
in the whole triple, missing a valid triple like
"x86_64-unknown-darwin" -- clang genuinely accepts this triple and
mangles for it exactly like a real Mach-O target, since the Darwin
underscore-decoration behavior this whole gate exists to recognize is an
OS-level linker behavior, not anything vendor-specific. Fixed by
splitting the triple on "-" and checking each COMPONENT against a set
of known Darwin OS names (darwin/macos/ios/tvos/watchos) via
startswith (tolerating a trailing version suffix like
"darwin20.6.0"), in addition to the pre-existing "apple" vendor
check -- rather than a bare substring test over the whole string, which
would also risk a false match from an unrelated component that merely
CONTAINS one of these tokens.
Second, a materially different and more subtle gap: the Darwin gate
ALONE is not sufficient, because a real, explicit asm("_foo") label is
just as possible ON Darwin as off it -- the sixteenth round's own fix
already established this for the non-Darwin case, and this round is the
identical failure mode surfacing on Darwin itself. This fallback's whole
justification is "a genuinely plain-C compilation unit has no
LinkageSpecDecl", and that justification only holds for a declaration
with NO enclosing scope at all: C has no namespaces, so a plain-C
declaration is always global-scope. A NAMESPACED Darwin C++ declaration
(namespace n { void foo() asm("_foo"); }) is never plain C regardless
of platform, so the fallback needs entry.scope as a THIRD, independent
gate alongside the target check -- mirroring how entry.extern_c itself
is only ever set by walking into a real LinkageSpecDecl, which (per
this same normalizer's own commentary elsewhere) always resets scope
too. Both extract.headers.clang.functions.parse_functions and
dumper_clang._ClangAstParser.parse_variables now also require not
entry.scope. Retagging such a declaration ("extern_c",) would have
been a double loss, not merely a wrong tag: entity_id_for_function/
entity_id_for_variable's is_extern_c branch always resolves
scope=(), so the fix also protects the namespace itself from being
silently discarded, not only the genuine asm-label mangled identity.
Pinned by two new regression tests (is_darwin_target's own parametrize
list gains the "x86_64-unknown-darwin" case, plus ios/tvos/
watchos OS-component cases; a new namespaced-Darwin-declaration test
proves the scope gate for both functions and variables), all in
tests/test_dumper_clang_extern_c_identity.py.
Landed (fifth slice, 2026-09-02): DWARF, the first non-header-AST
producer. ADR-063 Phase 2's "fourteenth slice" (2026-09-02, same day)
already gave dwarf_snapshot.py a real, typed ScopePath and a populated
entity_id on every RecordType/EnumType/Function/Variable/typedef
it produces -- this normalizer's own "reads identity, never resolves it"
contract meant DWARF needed no new identity work to become a caller, only
a new call site: dumper_elf_fallback._dwarf_semantic_ir (a thin wrapper
around the same normalize_header_ast, called from _try_dwarf_snapshot
right after build_snapshot_from_dwarf returns, rather than inside
dwarf_snapshot.py itself, which sits at its own architecture/debt.yaml
no-growth line-count baseline) passes the builder's
types/enums/typedefs (already namespace-qualified-keyed, matching
typedefs_qualified's own convention)/typedef_entity_ids/functions/
variables straight through, with producer="dwarf" and
constants={}/constant_entity_ids={} (DWARF carries no constexpr
initializer evidence at all).
Records/enums/typedefs needed no DWARF-specific handling. Functions and
variables each needed one producer-specific cv_qualification carve-out
(extract/semantic_normalizer_dwarf.py, split into its own leaf module to
keep semantic_normalizer.py under the 800-line production cap, the
identical reason semantic_normalizer_artifacts.py was split out one slice
earlier): a function's is unconditionally Fact.not_collected(), since
dwarf_snapshot._build_function never reads a method's own const/volatile
qualifier from the DIE at all (Function.is_const/is_volatile are always
their dataclass default here, never a confirmed reading -- reusing the
castxml/clang branch's Fact.present(...) would misrepresent "never
looked" as "confirmed not const"); a variable's is read from the
already-extracted, structurally-sound Variable.is_const field instead of
_variable_top_level_cv_qualification's text scan, which castxml/clang
need specifically because their own is_const is computed with a bare
whole-string word search that conflates a mutable pointer to const data
with a genuinely const pointer -- DWARF's is_const is not computed that
way at all (dwarf_snapshot._process_variable sets it from whether the
variable's own outermost type DIE is DW_TAG_const_type), so the same
conflation this normalizer exists to avoid for the other two backends does
not apply to DWARF, and reading is_const there is correct rather than a
regression. Verified against a real compiled fixture covering all four
cases (const int g, int* const g, const int* g, const int* const
g): int* const and const int* render as the IDENTICAL text
("const int *") by dwarf_snapshot._compute_type_name's own
const/pointer composition order, so a text scan could never have told them
apart for DWARF even in principle -- only the structural field can, which
is exactly why the DWARF branch reads it instead of reusing the text
scanner. DWARF extracts no structural volatile fact for a variable at all
(no backend has an is_volatile field on Variable), so a DWARF
variable's cv_qualification can only ever contain "const", never
"volatile" -- a documented, accepted gap, not a claimed absence.
snapshot_cache._SNAPSHOT_CACHE_VERSION bumped (26 -> 27), the same
"a stale cache entry would otherwise silently keep serving semantic_ir=
None forever" reasoning every prior slice's own cache bump gives. New
tests: tests/test_semantic_normalizer.py gained a producer="dwarf"
unit-test section (hand-built objects, no compiler needed, mirroring every
other producer's tests in that file); tests/test_dwarf_semantic_ir.py
(new) exercises the real production wiring end to end against gcc/g++
compiled fixtures, the same lightweight skipif-gated pattern
test_dwarf_entity_id.py uses (no integration marker, since this needs
no castxml/clang).
Landed (sixth slice, 2026-09-02): CanonicalEntity.template_arguments,
for records. Backend-agnostic and extraction-free, unlike every prior
slice's own per-producer work: a concrete class-template specialization's
RecordType.qualified_name/name already embeds its full Name<Arg1,
Arg2> compound spelling on every backend that surfaces one at all
(confirmed with real castxml AND DWARF output -- castxml's type_name_
uncached resolves a specialization to an ordinary, indistinguishable-
from-non-template <Struct name="Box<int, 3>"> element; a
compiler's own DWARF DW_AT_name for an emitted instantiation is the
identical compound spelling), so decomposing it needs no new identity
work and no producer-specific branch at all: extract/semantic_normalizer_
template_args.py's split_template_arguments (a new leaf module, split
out purely for line budget) is a pure bracket/paren/angle-aware text
splitter over whatever text canonical_spelling already reads -- it finds
the record's own leaf segment's (after its last top-level ::, so a
nested specialization's own arguments are never confused with an enclosing
scope's) top-level <...> and splits its contents on top-level commas,
verbatim. Deliberately never runs an argument through canonicalize_type_
name: a plain text split cannot tell a type argument from a non-type one
(a literal value, an enumerator) apart from its own text alone, and no
concrete cross-backend value-spelling divergence is observed to justify
guessing -- the identical "no canonicalizer without a known target
divergence to fix" discipline the fourth slice's constants already
established. Wired into normalize_header_ast's existing types loop
(one line, template_arguments=Fact.present(split_template_arguments(rt_
name) or ())) -- Fact.present(()) (a confirmed, not merely absent,
non-template) for every record that isn't one.
A closure-typed argument's own raw "(lambda at <path>:<line>:<col>)"
marker is stored UNrenumbered by this function -- deliberately, since
template_arguments was never added to qualified_name_segments.
_PAYLOAD_FIELD_EXCLUSIONS, so the pre-existing renumber_anonymous_
closure_identities walk (already reaching every string in AbiSnapshot.
semantic_ir, confirmed by grep before assuming it) canonicalizes it
post-hoc to the identical stable ordinal it already gives the SAME marker
embedded in the record's own canonical_spelling/EntityId -- confirmed
end to end against a REAL compiled fixture, not assumed: a template
instantiated with a real lambda's closure type, run through the actual
dump() production pipeline, shows the decomposed argument and the
record's own identity key converging on byte-identical renumbered text
(tests/test_semantic_ir_end_to_end.py's new closure-parameterized-
template test -- exactly this slice's own named acceptance-criteria
fixture).
clang is a confirmed, named exception, and it is a missing OCCURRENCE,
not a wrong FACT. dumper_clang.py's categorizing walk collects
CXXRecordDecl/RecordDecl nodes for self._records (and therefore
parse_types()) but never a ClassTemplateSpecializationDecl (confirmed
directly -- extract.headers.clang.templates.build_specialization_
index's own docstring states this exactly, for an unrelated vtable/
base-lookup reason), so a concrete specialization is never itself a
RecordType on that backend at all; only the UNINSTANTIATED PATTERN
(bare "Box", never "Box<int, 3>") is. Every clang-produced record this
normalizer sees is therefore, unconditionally and confirmedly, NOT an
instantiation -- Fact.present(()) is the CORRECT, confirmed answer for
every one of them, not a gap. What clang cannot do is produce ANY
occurrence at all for a concrete specialization, so this phase's own
acceptance-criteria fixture cannot show cross-backend AGREEMENT on that
specific entity -- confirmed with a real end-to-end test
(test_semantic_ir_template_arguments_end_to_end) that asserts exactly
this asymmetry (castxml: one real, decomposed occurrence; clang: none at
all) rather than assuming parity, squarely within this phase's own stated
acceptance bar ("not... an identical SemanticIR regardless of source
backend").
Functions/typedefs/variables/enums are deliberately untouched: none of
them can themselves be a template instantiation the way a record can in
this codebase's model. A function TEMPLATE's own instantiation is real
(identity<int>), but neither backend's Function.name embeds the
argument spelling the way a record's compound name does -- confirmed with
real castxml output: an instantiated function's own name stays the
bare, unparameterized "identity", with only the Itanium-MANGLED name
carrying the argument, needing a real demangler to decode it back into
argument spellings -- a materially different, larger project (a
mangled-name argument decoder, not a compound-spelling text split) than
this slice attempted. An enum/typedef/variable can never itself be a
template entity at all in the vocabulary this codebase's model tracks.
Seventh slice landed: PDB, types only (RecordType/EnumType via TPI).
The identical Phase 2 EntityId treatment DWARF's fifth slice needed, but
structurally harder to apply: PDB's own TPI type records carry no scope
tree at all to walk (unlike DWARF's DIE tree or the two header-AST
backends' AST trees) -- CodeView names each struct/class/union/enum with
its own FLAT, already-"::"-qualified spelling as one string
("NS::Outer::Inner"), with no separate parent-scope reference. The new
extract/pdb_scope.py therefore does the reverse of extract/
dwarf_scope.py's/the header-AST backends' own scope-construction modules:
it PARSES a qualified name string back into typed ScopePath segments,
rather than building one while walking a tree.
That parsing itself needed a shared primitive extract could actually
import: qualified_name_segments.raw_segments already had the exact
bracket-depth-aware "::"-splitting algorithm needed, but that module
belongs to the compare layer, which extract may not depend on (ADR-061
-- the identical constraint extract/headers/scope_segments.py's own
docstring already documents for that module's version_suffix). Rather
than reimplementing the same fiddly splitting loop a second time -- exactly
the "two independently constructible representations of the same fact"
shape this codebase's own governing invariant forbids elsewhere -- the
primitive itself moved down to a new leaf module,
model/qualified_name_split.py, which both qualified_name_segments.
raw_segments (now a one-line delegate) and extract/pdb_scope.py build
on. raw_segments's own existing 152-test coverage (lambda-identity,
scope-segments, semantic_ir-template-args, qualified-type-matching) all
passed unchanged against the delegating version, confirming the move
preserved behavior exactly.
The one genuinely new judgment call: given a qualified name's flat text
alone, is an enclosing segment a namespace or a nested class? CodeView's
spelling carries no direct signal either way. pdb_scope.py resolves it
by checking whether the ACCUMULATED prefix up to that segment is itself a
name this same PDB separately recorded as a struct/class/union (the
caller's own DwarfMetadata.structs key set, already computed and
in hand -- no second pass needed) -- a Record segment if so, Namespace
otherwise. This heuristic is UNVERIFIED against real MSVC output: this
environment has no MSVC/cl.exe toolchain (the windows-msvc CI lane's
own msvc-marker gate is the only place that exists), so it is covered
only by hand-built qualified-name-string tests -- the identical "no
compiler needed" discipline tests/test_pdb_parser.py's synthetic
MSF/TPI/DBI byte-stream fixtures already establish for this backend,
applied one layer up (post-parse qualified-name strings, not raw TPI
bytes). Its own documented, accepted limitation: a purely
forward-declared-and-never-defined enclosing class would not appear in
the known-record-names set (pdb_metadata._is_user_visible filters
forward-ref-only entries out entirely) and would misclassify as a
namespace -- an edge case the qualified-name text alone cannot resolve,
matching the DWARF/header-AST backends' own practice of documenting an
evidence gap explicitly rather than guessing past it. No anonymous-type
handling either: CodeView synthesizes an internal name for an unnamed
struct/union/enum rather than leaving it genuinely anonymous the way a
DWARF DIE or a Clang/castxml AST node can be, and _is_user_visible
already filters those compiler-internal names out before they ever reach
this module.
pdb_model.py's _record_from_layout/_enum_from_info now stamp a real
entity_id via this module, and the new pdb_model.pdb_semantic_ir()
calls the existing, UNMODIFIED normalize_header_ast -- no PDB-specific
Fact-carve-out module needed for this types-only slice (unlike DWARF's
fifth slice, which needed extract/semantic_normalizer_dwarf.py for its
own cv_qualification gaps): the normalizer's types/enums loop never
touches cv_qualification at all (a functions/variables-only fact this
slice does not populate for PDB), so producer="pdb" needs no new
producer-dispatch branch there. Wired into the real production call chain
(service_dump_native_pe._dump_pe's existing header-scoping fallback
branch -- the same narrow "headers requested, castxml could not resolve a
surface" path pdb_model.py's own pre-existing docstring already
describes, not a new call site) and verified end to end through that real
chain (tests/test_pdb_provenance.py::TestDumpPeFallbackBuildsPdbTypes),
not only in isolation.
Still not landed on the PDB side: PDB's own function/variable
identity. The DBI module symbol streams -- where CodeView's
S_GPROC32/S_LPROC32/S_GDATA32/S_LDATA32 records live, the PDB
analogue of DW_TAG_subprogram/DW_TAG_variable -- are parsed by
NOTHING today (pdb_parser.py's own DBI header fields for these streams
are read but never acted on), so there is no PDB-native Function/
Variable at all to attach an EntityId to yet. Giving PDB function/
variable identity is therefore not "add an entity_id= field to an
existing extraction path" the way DWARF's third slice was -- it needs
genuinely new parser work first, a materially larger, separate project
this types-only slice does not attempt.
Ninth slice landed: BTF/CTF, types only. Both btf_metadata.
parse_btf_metadata and ctf_metadata.parse_ctf_metadata reduce their own
richer, format-specific parse to the shared DwarfMetadata shape (via
BtfMetadata.to_dwarf_metadata/CtfMetadata.to_dwarf_metadata) purely so
the checker's pre-existing _diff_dwarf/_diff_advanced_dwarf detectors
work against any of the three debug formats unmodified -- but that
conversion carries only flat, name-keyed StructLayout/EnumInfo dicts,
never a RecordType/EnumType model object, so nothing along that path
had ever had an entity_id to build an occurrence from. New
extract/debug_layout_semantic_ir.py bridges that shared shape into
transient RecordType/EnumType objects carrying a real entity_id, then
feeds them through the same normalize_header_ast every other producer
uses. Wired into the ELF headerless (symbol-only) fallback path
(dumper_elf_fallback._build_symbol_only_snapshot, the only place a
BTF/CTF-resolved dumper.py run reaches -- a real DWARF resolution instead
goes through dwarf_snapshot.build_snapshot_from_dwarf, whose own
entity_id-bearing types have existed since Phase 2).
No PDB-style scope-resolution heuristic applies at all: BTF (the Linux
kernel's BPF Type Format) and CTF (illumos/Solaris's Compact C Type
Format) are both pure-C debug formats with no namespace/class nesting
whatsoever, so every ScopePath this slice builds is unconditionally
empty -- none of PDB's own documented limitations (namespace-vs-record
ambiguity, forward-declared enclosing classes, function-local scopes,
nested anonymous aggregates) have a BTF/CTF analogue, and none needed
inventing here.
Deliberately does not widen AbiSnapshot.types/.enums. Every prior
Phase 6 slice (DWARF and PDB) only ever adds entity_id/SemanticIR
normalization on top of a model-type bridge that already existed
independently of Phase 6, for other reasons (DWARF's own DIE-walk
builder predates this phase entirely; PDB's own pdb_model.py bridge,
wired into the PE header-scoping fallback so those types reach
AbiSnapshot.types/.enums and, from there, surface.py/vtable/
internal-leak detection, likewise predates this phase). BTF/CTF never had
such a bridge at all, so building one that newly feeds .types/.enums
would be a genuinely new, larger change -- newly exposing BTF/CTF structs
to every other .types-consuming detector for the first time, a
materially larger, separately-scoped design question this slice does not
attempt. This slice's own RecordType/EnumType values are therefore
transient: consumed only by normalize_header_ast and discarded
immediately after, with AbiSnapshot.types/.enums left exactly as they
already are for a BTF/CTF-sourced snapshot (empty, on the current
headerless fallback path). A future slice giving BTF/CTF real model-type
population (mirroring what PDB's own PE-fallback wiring already does) can
reuse this same entity_id assignment without redoing it.
Function/variable/typedef identity is a further, unattempted gap:
BtfMetadata/CtfMetadata's own func_protos/typedefs fields are not
even carried across either format's own to_dwarf_metadata() conversion
(see that method's own docstring) -- there is no matching
EntityId-bearing evidence reaching the normalizer for those kinds at
all, not a case this slice silently drops.
service.py's own BTF/CTF dispatch (a third production assembler this
phase's own "not the only production assembly call sites" note above
names -- the separate call site around where it parses a raw BTF/CTF blob
and constructs an AbiSnapshot directly from btf.to_dwarf_metadata()/
_typeinfo_functions(btf.func_protos)/dict(btf.typedefs), and the
identical CTF branch beside it, NOT the dumper_elf_fallback.py path this
slice wires) has since been wired too (Codex review, fresh evidence): that
dispatch lives on today as workflows/input_resolution.py::
_resolve_raw_typeinfo (ADR-061 Phase 4 relocated it out of service.py
verbatim, before this slice's own review round), and now calls
semantic_ir_from_debug_metadata for both its BTF and CTF branches, the
same as dumper_elf_fallback.py's own call site.
Still not landed, and therefore this phase is not complete:
the phase's own acceptance criteria (a
closure-parameterized template fixture) remain only PARTIALLY met --
castxml's own occurrence now really does decompose and canonicalize a
closure-typed template argument end to end, but clang produces no
comparable occurrence to agree with it at all (above), and a function
template's own template-argument list remains a separate, unattempted
gap (also above). A manifest (--dump-manifest) dump's own occurrence-
detail loss (Codex review, PR #1001) is now closed -- see the corrected
account below, kept in full for the institutional-memory reasons the
paragraph itself explains.
Multi-TU manifest occurrence detail: two wrong analyses, then the real
fix (Codex, PR #1024) -- recorded precisely rather than quietly
corrected a third time. An earlier revision of
this section argued the obvious close (normalize each TuFragment before
merge_fragments collapses identities, keyed by a real per-TU
disambiguator) was a no-op, reasoning that a RecordType's forward-
declaration/full-definition split produces two byte-identical
CanonicalEntity PAYLOADS (canonical_spelling/template_arguments both
derive from the qualified name text alone; cv_qualification is
NOT_COLLECTED for every record regardless) and concluding genuinely
closing the gap needed CanonicalEntity to grow a new completeness/
availability field. That conflated the payload with the occurrence: two
occurrences with identical payloads are still two distinct declarations,
and SemanticIR.occurrences (keyed by OccurrenceId, not EntityId)
exists precisely to preserve occurrence COUNT independent of whether two
payloads happen to coincide -- so the "pure duplication" framing, and the
model-extension conclusion drawn from it, were both wrong.
A second analysis then claimed the blocker was sharper still: that
nothing today distinguishes a genuine cross-TU declaration split from the
ordinary, far more common case of the identical declaration observed
redundantly because many TUs #include the same header, and that closing
it needed tu_merge.py to expose a new per-entity trivial-vs-genuine-
variance signal it does not have today. This was also wrong, and a
second Codex review round on PR #1024 pointed at the fix directly: the
distinguishing signal already exists, on the pre-merge candidates
themselves, and does not need tu_merge.py to expose anything new.
RecordType/EnumType/Function/Variable (both header-AST backends)
already carry their own source_location ("file:line") from parsing --
a genuine cross-TU split (a public header's forward declaration, a
private header's full definition) reports two different locations,
while the ordinary redundant-#include case reports the identical
location from every including TU, since it is literally the same file and
line. That is exactly the distinction the second analysis said did not
exist.
The actual fix, landed: extract/manifest_semantic_ir.py's new
manifest_semantic_ir(fragments) normalizes each contributing TU's own
raw, pre-merge TuFragment independently (never reading
merge_fragments's already-folded output at all) and unions the
resulting per-fragment SemanticIR.occurrences maps (first-fragment-wins
on a key collision, fragments ordered by tu_name to stay deterministic).
extract/semantic_normalizer.normalize_header_ast gained a
disambiguate_by_source_location: bool = False parameter (default
preserves every existing caller's behavior unchanged); when true, each
type/enum/function/variable occurrence's OccurrenceId.disambiguator is
set to that declaration's own source_location. Typedefs/constants carry
no source_location in this codebase's model at all (neither has an
"incomplete" form to begin with), so this pass leaves them exactly as
merge_fragments's own flat fields already do. dumper_manifest.
run_tu_loop calls manifest_semantic_ir(fragments) and attaches the
result to MergedTuFragments.semantic_ir (a new field);
resolve_header_ast_result prefers that richer value when present,
falling back to the legacy single-pass normalize_header_ast call only
when a caller's MergedTuFragments never set it. Verified against real
clang output in tests/test_dumper_manifest_semantic_ir.py: a genuine
two-TU forward-declaration/full-definition split produces exactly two
occurrences with two distinct source_location-derived disambiguators,
and three TUs sharing one unmodified header collapse to exactly one
occurrence -- confirming both the fix and that it does not regress the
overwhelmingly common shared-header case into per-TU noise. A single-TU
manifest dump is unaffected either way (nothing for the merge to collapse
ahead of it, and the resulting IR matches a single-header dump's own
shape). See extract/semantic_normalizer.py's and
extract/manifest_semantic_ir.py's own docstrings for the same account.
The original Design/Files/Tests/Acceptance-criteria sections below are
kept verbatim as the phase's full target shape -- each slice is a step
toward that target, not a redefinition of it.
Goal. Type-spelling, scope, template-argument, anonymous/lambda, and CV-qualification canonicalization happens once, not once per backend.
Design. SemanticIR is defined and tested before any parser is
narrowed to feed it — an earlier draft of this phase specified only the
normalizer function's signature (normalize(raw) -> SemanticIR) and
never the type itself, which would have let each backend's migration
converge on a different ad hoc shape behind the same name, defeating the
whole point of "one canonical IR." abicheck/model/semantic_ir.py (new):
SemanticIR is keyed by OccurrenceId, not collapsed to one entry
per EntityId — a first draft of this phase kept one entity per
EntityId, which would silently discard exactly the distinction Phase 2
introduces OccurrenceId to preserve: a complete definition and an
incomplete/ODR-duplicate declaration can legitimately share one
EntityId while carrying different availability, origin, or producer
facts, and a one-entry-per-identity map would overwrite or merge that
evidence away before comparison ever sees it. SemanticIR.occurrences:
dict[OccurrenceId, CanonicalEntity] holds every occurrence; a derived
SemanticIR.canonical_entities() -> dict[EntityId, CanonicalEntity]
projection (resolving which occurrence wins when a consumer genuinely
wants one canonical view, not every occurrence) is a separate, explicit
method for the callers that actually need that reduction, rather than
the only shape SemanticIR offers. CanonicalEntity itself carries no
ScopePath/EntityId of its own — a first draft of this phase gave it a
resolved ScopePath field alongside the dict's own OccurrenceId key,
and a reviewer correctly flagged that as the exact "two independently
constructible representations of the same fact" shape the Governing
Invariant exists to forbid: OccurrenceId.entity_id.scope_path already
is the resolved scope, so a second, separately-settable copy on the
value means a normalizer or deserializer bug could produce a mapping
whose key names one scope and whose value reports another, with nothing
short of a dedicated equality test to catch the disagreement. Identity
— ScopePath included — lives exclusively in the key; CanonicalEntity
holds only the non-identity payload: canonical type spelling,
template-argument list, and CV-qualification, independent of which
backend produced it, plus the Fact[...]-wrapped per-field availability
Phase 0 established, so a canonicalized entity can state "this backend
didn't produce this particular fact" rather than only "here is the
value." A caller that needs an entity's own scope reads it off the key it
was retrieved with (occ_id.entity_id.scope_path), never a field on the
value — a function that needs to hand a CanonicalEntity to another
caller without its key in scope returns the pair ((OccurrenceId,
CanonicalEntity) or equivalent), not a CanonicalEntity carrying a
second copy of what the key already states. This model file,
and a primitive-level test suite pinning its shape directly (construct a
few entities by hand, including two sharing one EntityId with
differing availability, and assert both the OccurrenceId→entity mapping
and the canonicalization rules independent of any real backend), land as
their own first step in this phase, before any of the following
narrowing work.
Only once SemanticIR itself is real does abicheck/extract/
semantic_normalizer.py's normalize(raw: RawCastXmlFacts | RawClangFacts
| RawDwarfFacts | ...) -> SemanticIR have a concrete target to produce.
Each backend's existing parser (dumper_castxml.py, dumper_clang.py,
dwarf_snapshot.py, pdb_metadata.py) is narrowed to produce only its
own RawXFacts — today's parse_types()/parse_typedefs()-style
functions stop doing their own ad hoc namespace-joining, anonymous-marker
handling, and closure-identity stripping, and instead emit the backend's
literal output for the normalizer to canonicalize via the EntityId/
ScopePath primitives Phase 2 already built, converging on the one
SemanticIR shape just defined rather than each backend's own
reading of "canonical."
Why this phase is ordered after Phase 2, not before. Every
cross-backend disagreement AGENTS.md records in this area (the lambda-
closure-identity entries, the MSVC-vs-Itanium mangling-scheme entries, the
Outer::Inner partial-qualification entry) is a canonicalization
disagreement about identity specifically — Phase 2's EntityId/
ScopePath is the primitive this normalizer is built on, not a parallel
concern.
Files. abicheck/model/semantic_ir.py (new — SemanticIR itself,
landed and tested before any of the files below are touched);
abicheck/extract/semantic_normalizer.py (new);
dumper_castxml.py/dumper_clang.py/dwarf_snapshot.py/pdb_metadata.py
(narrowed to raw-fact production, each losing its own copy of
anonymous-marker/closure-identity/namespace-join logic as that logic moves
to the shared normalizer); btf_metadata.py/ctf_metadata.py (their own
BtfType/CtfType/_TypeResolver pairs narrowed the same way — included
per ADR-063 D9 on the same architectural grounds as the other
type-declaration-producing backends, even though neither has a specific
AGENTS.md incident motivating it yet); name_classification.py (its
_ANONYMOUS_TYPE_MARKERS and sibling helpers become the normalizer's,
used once); dumper.py/dumper_manifest.py (the assembly call sites —
call semantic_normalizer.normalize() on each backend's raw facts,
project into the existing AbiSnapshot field shapes, and attach the
SemanticIR itself on the new semantic_ir field, per the Design section
above). dumper.py/dumper_manifest.py are not the only production
assembly call sites — service.py has two more of its own, each
independent, and a first draft of this phase's Files list named neither.
service.py's own BTF/CTF dispatch (around where it parses a raw BTF/CTF
blob and constructs an AbiSnapshot directly from btf.to_dwarf_metadata()/
_typeinfo_functions(btf.func_protos)/dict(btf.typedefs), and the
identical CTF branch beside it) is a third production assembler —
narrowing btf_metadata.py/ctf_metadata.py to raw-fact production
without also routing this call site through semantic_normalizer.
normalize() would leave it assembling an AbiSnapshot from facts whose
shape just changed out from under it, breaking every BTF/CTF-backed
dump/compare rather than merely leaving SemanticIR inert.
service.py's PE/PDB path is a fourth: it calls pdb_model.
model_types_from_dwarf_metadata(dwarf_meta) to convert PDB-derived DWARF
metadata into RecordType/EnumType objects before assembling the
snapshot — narrowing pdb_metadata.py alone, as the first draft's Files
list already named, leaves this second, PDB-model-specific conversion
step untouched and still producing the pre-normalization shape.
A fifth site is not a raw-fact assembler at all, and a later review
round correctly found that this phase's own "attach semantic_ir
identically" treatment does not describe what it actually needs:
dumper_hybrid.merge_snapshots(), the --ast-frontend hybrid path.
service.py recursively produces a CastXML snapshot and a Clang snapshot
and hands both to merge_snapshots(), which reconciles them via
dataclasses.replace(castxml_snap, ...) — a pairwise merge of two
already-assembled AbiSnapshots, not a single backend's raw facts
going through the normalizer once. Each sub-snapshot, per this phase's own
per-backend wiring, already carries its own semantic_ir by the time
merge_snapshots() receives it — but that function's real logic only
reconciles the legacy functions/types/... projections (folding in
Clang-only entities and Clang-backfilled facts onto the CastXML base), and
has no step that does the equivalent reconciliation for semantic_ir
itself. Left as stated, the merged snapshot would keep the CastXML-only
semantic_ir unchanged while its own functions/types fields include
exactly the Clang-only and Clang-backfilled data that reconciliation adds
— the two representations disagreeing on a single, freshly-produced
snapshot, which is the Governing Invariant's one forbidden outcome, not a
legacy-compatibility accommodation. merge_snapshots() therefore needs
its own, fifth reconciliation step for semantic_ir.
"Merging (or reconstructing) the two sub-snapshots' SemanticIR.
occurrences maps the same way the legacy fields already merge" is not
itself a rule, and a review round correctly pressed on what that
parenthetical was actually supposed to mean — it does not define how a
matching EntityId with a different OccurrenceId disambiguator is
reconciled, nor what happens when both backends produced a real,
disagreeing fact for the same entity. The actual rule is the identical
base-plus-backfill-with-provenance discipline merge_snapshots()'s own
docstring already states for every other field it touches — "castxml
remains the base... only the facts documented in this module's docstring
are actually reconciled/backfilled," with fact_provenance recording
which backend's value won, per declaration, per fact — applied to
occurrences instead of invented fresh for it:
- Matching is keyed on
EntityId, not the fullOccurrenceId. Both backends parse the identical headers, so the common case is two independently-derivedEntityIds that are structurally identical (sameScopePath, same kind, same leaf name) with an empty disambiguator on both sides — exactly the "globally unique identity" case the graph-key fix elsewhere in this phase already reduces to plainEntityIdequality for. Matching on the bareEntityIdfirst, and only reading each side's own disambiguator afterward to decide whether a match is safe.
The safety check itself had a real bug, and a review round caught
it precisely: "non-empty... on either side" is not the same condition
as "both sides assert something, and they disagree" — and the
difference breaks the ordinary case, not just an edge case.
OccurrenceId's disambiguator carries no neutral, producer-independent
value — it is populated from whichever USR/TU-context signal that
one backend's own parse actually derived, and CastXML has no USR
concept at all (entity_identity.py's own rule: "an absent USR/mangled
name degrades the tier, it is never guessed at"), so CastXML's side of
an ordinary, genuinely-matching declaration is routinely empty while
Clang's side is routinely non-empty — not because the two parsers
disagree about identity, but because only one of them has a TU-context
signal to report at all. The originally-stated rule ("a non-empty,
disagreeing disambiguator on either side... left unmerged") reads that
as a disagreement, which would leave the common hybrid case
permanently unmerged — exactly the failure the reviewer traced: no
Clang backfill ever reaches the base entity, and semantic_ir
disagrees with the legacy fields merge_snapshots() already
reconciles, on every ordinary declaration, not only a genuinely
ambiguous one. The corrected rule needs both sides to carry a
non-empty disambiguator before comparing them at all — an empty
disambiguator on either side is "no additional signal from that
backend," never itself a disagreement — and only withholds the merge
when both are non-empty and unequal (the real TU-collision case this
mechanism exists for: two backends that can both derive a TU-context
signal and that signal genuinely differs).
That corrected rule is still not enough on its own, and a further
review round found the gap it leaves: it silently assumes each
EntityId maps to at most one OccurrenceId per side, which this
same phase's own Design section explicitly says is false. Phase 6's
own text states SemanticIR.occurrences is "keyed by OccurrenceId,
not collapsed to one entry" precisely because a real ODR-duplicate
pair, or an incomplete-declaration/complete-definition pair, shares one
EntityId across multiple occurrences on a single backend's own
snapshot. "Match first by bare EntityId" is therefore not
automatically a one-to-one match whenever either side's candidate set
for that EntityId has more than one member — an arbitrary CastXML
occurrence could be paired against the wrong Clang occurrence sharing
the same identity (or vice versa), backfilling facts from a
declaration that is not actually the same physical entity, or
collapsing two genuinely distinct occurrences into one. Fixed by
checking candidate-set size before attempting the disambiguator
comparison above, not only after: for a given EntityId, if either
side has more than one occurrence, matching is decided over the whole
group at once, not pair by pair.
"Exactly one compatible pair survives all-pairs filtering" is the
wrong test for "a unique matching exists," and a further review round
gave the exact counterexample: when each side independently has
more than one occurrence, a correct, unambiguous one-to-one pairing
can still exist, and the all-pairs-filter rule rejects it anyway.
Two CastXML occurrences with disambiguators {usr1, usr2} against two
Clang occurrences with disambiguators {usr1, usr2} have exactly one
correct pairing (usr1↔usr1, usr2↔usr2) — but checking every
cross pair against the disambiguator-safety rule leaves two
agreeing pairs surviving (usr1↔usr1 and usr2↔usr2), not one, so
"exactly one pair remains" incorrectly reads this as ambiguous and
unions the whole group unmerged, losing a real Clang backfill for both
occurrences even though there was never any actual ambiguity. The
correct test is uniqueness of a complete matching over the group, not
uniqueness of a single surviving pair: group each side's occurrences by
disambiguator value first — every non-empty disambiguator value present
on both sides must name exactly one occurrence per side (two
occurrences on one side sharing a non-empty disambiguator is itself a
genuine ambiguity for that value, not resolvable by this rule at all) —
pairing each such value 1:1.
The leftovers after that pass are not necessarily empty-disambiguator
occurrences, and a further review round gave the exact counterexample:
CastXML {empty, usr1} against Clang {usr1, usr2} pairs usr1↔usr1
in the first pass, leaving CastXML's empty and Clang's usr2 as the
leftovers — one empty, one genuinely non-empty, neither claimed by the
other side. A one-sided non-empty disambiguator (present on one
occurrence, simply absent because the other backend never derived one
for its matching declaration) is not itself a disagreement — the same
"no additional signal from that backend" rule the single-occurrence
case already states — so treating every leftover as if it must be
empty, and refusing to look at what it actually holds, would wrongly
leave this pairing unmerged even though it is exactly as safe as the
empty-vs-empty case. The leftover pass therefore applies the identical
single-pair disambiguator-safety rule from above, not a
narrower empty-only rule: when exactly one occurrence remains unmatched
on each side, pair them unless both remaining disambiguators are
non-empty and unequal (a real, two-sided disagreement — the one case
this rule still refuses). Any EntityId for which this process
leaves a non-empty disambiguator value claimed by more than one
occurrence on either side in the first pass, leaves more than one
leftover unmatched on either side after it, or leaves exactly one
leftover per side whose disambiguators are both non-empty and disagree,
has no unique matching — it is left entirely unmerged for that
identity: every occurrence from both sides is unioned in verbatim as its
own entry, under rule 3 below, rather than guessing at a pairing. This
is the same fail-closed
direction the disambiguator fix above already takes — an ambiguous
group produces no merge rather than an arbitrary one — applied at the
cardinality check that has to run before the pairwise comparison is
even meaningful, not a new principle invented for it. A dedicated
property test pins the uniquely-matchable multi-occurrence case
directly (two same-sized groups whose non-empty disambiguators are a
bijection, confirmed to merge both pairs correctly), the genuinely-
ambiguous case (two occurrences sharing one non-empty disambiguator on
one side, confirmed to leave the whole group unmerged), and the mixed
one-sided-leftover case above (CastXML {empty, usr1} against Clang
{usr1, usr2}, confirmed to merge both pairs — usr1↔usr1 from the
first pass, empty↔usr2 from the leftover pass — rather than leaving
the group unmerged) — the three shapes this finding's own two
counterexamples and the original ambiguity-detection requirement each
name.
2. CastXML's CanonicalEntity is the base for every matched pair,
mirroring every other reconciled field in this function: Clang's
matching occurrence backfills only the specific facts CastXML's own
entity carries as Fact.not_collected()/Fact.unsupported(...) — it
never overwrites a fact CastXML already resolved to Fact.present(...),
present-value disagreement included. A fact CastXML resolved and Clang
also resolved, disagreeing, is not silently dropped by either
direction: CastXML's value is kept (matching the base precedence every
other field already uses), and the disagreement itself is recorded.
fact_provenance itself cannot carry that record, and a review round
correctly found the reused-field claim doesn't survive checking the
real type. AbiSnapshot.fact_provenance: dict[str, str] stores only
the winning producer's name ("castxml"/"clang") per fact key —
by design, per that field's own docstring, for every pre-existing
legacy-field reconciliation this function already does — and has no
slot for the losing backend's own value or for a conflict marker at
all. Reusing it for semantic_ir reconciliation would silently lose
exactly the information this step claims to preserve: once CastXML
wins, a consumer reading fact_provenance sees "castxml" whether
Clang agreed or actively disagreed, with no way to tell the two apart.
Fixed with a new, additive field instead of widening the existing
one (which every pre-existing fact_provenance reader already depends
on staying dict[str, str]) — AbiSnapshot.semantic_ir_conflicts:
dict[str, str], valued with a repr() of the losing backend's
discarded value
(present only for a key where a real conflict occurred; absent
otherwise, same "absence means no conflict" convention fact_
provenance itself already uses).
Keying semantic_ir_conflicts identically to fact_provenance's own
fact keys is itself wrong for this specific field, and a further review
round caught exactly why: fact_provenance's keys
(func_fact_key(mangled, fact)/type_fact_key(name, fact)/...,
reading the real functions) name a declaration, not an occurrence —
correct for every pre-existing legacy-field reconciliation, which never
has more than one matched pair per identity, but this phase's own
matching rule (two sections up) explicitly allows more than one matched
pair to share one EntityId (the ODR-duplicate/incomplete-declaration
case SemanticIR.occurrences exists to represent). Two different
matched occurrence pairs sharing one EntityId, each with its own real
conflict on the same fact name, would write the same declaration-keyed
string twice — the second write silently discards the first conflict
record, with no signal that two conflicts existed rather than one.
Fixed by keying semantic_ir_conflicts on the occurrence, not the
declaration: each key is the matched pair's own canonical_key
(occurrence_id) (Phase 3's collision-free occurrence rendering, already
built for exactly this "more than one occurrence can share one
EntityId" case) joined with the fact name, rather than reusing
fact_provenance's declaration-only key — so two conflicting pairs for
the same EntityId occupy two distinct keys and neither can silently
overwrite the other. fact_provenance itself is unchanged (its
existing declaration-only key stays correct for the legacy fields it
already reconciles, none of which has this multi-occurrence shape); a
dedicated property test covers two matched occurrence pairs sharing one
EntityId, each with its own independent conflicting fact, asserting
both conflict records survive in semantic_ir_conflicts — confirmed to
fail against a version keyed like fact_provenance, where the second
pair's conflict silently overwrites the first's. fact_provenance[key]
== "castxml" (declaration-keyed, for the legacy fields) plus the
occurrence-keyed semantic_ir_conflicts together give a consumer both
which backend won a given legacy field and, per occurrence, that a real
disagreement, not mere agreement, produced a semantic_ir outcome.
3. A Clang-only EntityId (no CastXML match at all) is unioned in
verbatim, exactly mirroring how a genuinely Clang-only function/type
is appended rather than dropped in the existing legacy-field merge.
Added to the Files list below and to the parity-test requirement, since a
--ast-frontend hybrid dump/compare is a real, already-
documented production path this phase's own "every assembly call site"
bar already commits to covering, not a fifth site invented for this
finding. The BTF/CTF dispatch and PDB path are
updated the same way dumper.py/dumper_manifest.py are: call
semantic_normalizer.normalize() on the raw facts and project through the
existing AbiSnapshot field shapes, attaching semantic_ir identically
— model/snapshot.py (the new AbiSnapshot.semantic_ir field);
pdb_model.py (model_types_from_dwarf_metadata narrowed to raw-fact
production the same way pdb_metadata.py itself is, per the Design
section's own parser-narrowing rule, since it's a second conversion layer
for the identical backend, not a different one);
dumper_hybrid.py (merge_snapshots() gains the semantic_ir
reconciliation step named above, alongside its existing legacy-field
merge);
serialization.py (SCHEMA_VERSION bump, and a real encode/decode design
for semantic_ir — not the bare "field's encode/decode" a first draft
of this phase left unspecified, which understates a genuine technical
blocker. SemanticIR.occurrences: dict[OccurrenceId, CanonicalEntity]
is keyed by a dataclass, and snapshot_to_dict() calls whole-snapshot
asdict(snap) — dataclasses.asdict() recurses into a dict's keys the
same way it recurses into values, so an OccurrenceId key becomes a
nested dict before json.dump() ever runs, and a dict is unhashable —
asdict() itself raises constructing the converted mapping, for every
snapshot once semantic_ir is populated, not only on the eventual JSON
write. Flattening OccurrenceId into a string key (the same move Phase
2's storage/entity_ids.py finding rejected for EntityId's own
ScopePath) would reintroduce the identical lossy-flattening defect for
the identical structural reason — an OccurrenceId carries an EntityId
carrying a ScopePath, so a string rendering can't be reversed any more
than ScopePath alone could. The fix follows the same shape Phase 2's
v2 DTO already established: semantic_ir is excluded from the plain
asdict() walk (the same special-casing surface_graph already needs,
per Phase 3's finding) and encoded as a list of entries, not a dict —
{"occurrences": [{"occurrence": <dto>, "entity": <dto>} for each
occurrence_id, entity in self.occurrences.items()]}.
The encode/decode functions themselves do not live on the domain types —
a first draft of this phase put to_dict()/from_dict() directly on
SemanticIR/OccurrenceId/EntityId, and a reviewer correctly caught
that this reverses the dependency direction D8/ADR-061 already establish.
storage/entity_ids.py's v2 DTO conversion is owned by storage/
precisely so model -> storage never has to exist as an edge; a model/
semantic_ir.py-resident to_dict() that calls into storage/entity_ids.
py's DTO functions would create exactly that edge, and reimplementing the
identical structured-segment encoding directly inside model/ would
create the duplicate encoding this same paragraph already rejects one
paragraph up. The actual owner of this conversion is serialization.py
itself — the one module in this codebase already positioned to depend on
both model and storage — via new, free (not method) functions,
encode_semantic_ir(semantic_ir) -> dict/decode_semantic_ir(data) ->
SemanticIR, which call storage.entity_ids.to_dto()/from_dto() for each
OccurrenceId/EntityId and assemble/take apart the list-of-entries shape
above. snapshot_to_dict()/snapshot_from_dict() call these two
functions for the semantic_ir field instead of either a domain-type
method or a second, storage-importing branch living in model/.
elf_metadata.py/pe_metadata.py/macho_metadata.py are
explicitly not touched by this phase — see ADR-063 D9's own
"deliberately excluded, not an oversight" note: binary-symbol-table
extraction has no type spelling/scope/template-argument concern for this
normalizer to canonicalize in the first place.
Narrowing the parsers is not, by itself, a complete migration: dumper.py
and dumper_manifest.py are the production call sites that invoke
parser.parse_functions()/parse_types()/... today and assemble their
return values directly into AbiSnapshot's functions/types/...
fields, and checker.compare() consumes that AbiSnapshot shape
unchanged. Once a parser method returns only RawXFacts, neither call
site has anywhere to route the normalizer through — dumper.py/
dumper_manifest.py (both named explicitly in the Files list below, not
only in this prose) are updated in this same phase to call
semantic_normalizer.normalize() on each backend's raw facts and project
the result back into the existing AbiSnapshot field shapes. This
projection must not go through SemanticIR.canonical_entities() — a first
draft of this phase specified exactly that, and it silently reintroduces
the evidence loss OccurrenceId-keying exists to prevent, one step later
than the earlier fix closed it. canonical_entities() is defined above
as "resolve which occurrence wins" — a genuine reduction, by design, for
a consumer that explicitly wants one canonical view. AbiSnapshot.
functions/types/... are not that consumer: they are the existing
list-shaped fields the unchanged checker reads today, and today's
assembly already puts a complete definition and an incomplete/ODR-
duplicate declaration sharing one EntityId into that list as two
separate entries — routing through canonical_entities() here would
collapse them to one, an order-dependent, unspecified-winner loss of
exactly the evidence SemanticIR.occurrences (plural, keyed by
OccurrenceId) was built to keep. The correct projection iterates
SemanticIR.occurrences directly — one AbiSnapshot list entry per
occurrence, the same cardinality today's assembly already produces — and
is pinned by this phase's own parity test (below) proving the legacy
fields' shape and count are unchanged for a fixture containing a real
ODR-duplicate pair, not only for the common one-occurrence-per-entity
case a less pointed test could pass by accident. SemanticIR.
canonical_entities() remains exactly what it already was: the reduction
method for a future SemanticIR-aware consumer that genuinely wants one
canonical view, reachable through AbiSnapshot.semantic_ir below, never
through the legacy fields. The adapter making SemanticIR itself
available to compare()/future SemanticIR-aware detectors, not only the
projected fields, is this same assembly step: AbiSnapshot gains a new,
optional semantic_ir: SemanticIR | None field, populated by dumper.py/
dumper_manifest.py alongside the projected fields — one assembly call
produces both the backward-compatible AbiSnapshot shape existing
detectors read and the canonical SemanticIR a future detector can read
instead.
"Rather than two independent channels that could disagree" overclaims
what one shared assembly call actually guarantees, and a first draft of
this phase left it at that — review correctly pointed out that the
guarantee is one-time, not ongoing. Both AbiSnapshot.functions/
types/... and AbiSnapshot.semantic_ir are ordinary, independently
mutable dataclass fields once construction returns — nothing stops a
direct Python caller, a post-processing pass, or a deserializer from
mutating or constructing one without the other afterward, and Phase 10
does not retire either representation (each is read by real consumers:
every existing detector reads the legacy fields, a future SemanticIR-
aware detector reads the new one). So after this phase, a snapshot that
went through any path other than the one assembly call this phase adds
can carry a legacy projection and a SemanticIR that disagree — the
one-time construction guarantee does not survive the object's own
mutability, and this plan should not claim it does. Not made read-only
or derived-on-access in this phase, and that is a real, named limitation
rather than a silent gap: a @property-derived legacy field was already
rejected twice elsewhere in this same plan (the vtable/bases Fact
bridge in Phase 0, the CanonicalEntity/ScopePath duplication in this
same phase, above) for the identical reason — dataclasses.asdict() walks
real fields, not properties, so deriving one from the other would rename
or drop a JSON key every existing asdict-based consumer reads today, the
exact compatibility break this phase's own "backward-compatible
AbiSnapshot shape" commitment exists to avoid. What this phase does
add: the end-to-end parity tests below exercise every real assembly call
site and would catch the one-time guarantee failing at construction, and
retiring the legacy fields (making them genuinely derived, or deleting
them outright) is explicitly deferred — not to an unscheduled "eventually,"
but to whichever future phase first has a real SemanticIR-only detector
population large enough that the legacy fields have no remaining reader,
the same retirement bar Phase 10's other removals already use elsewhere
in this plan. Until then, a caller that mutates one representation
directly and not the other is responsible for keeping them consistent
itself — this phase does not enforce it at every mutation/load boundary,
since doing so would mean exactly the derived-field redesign just rejected
above. This is
additive to AbiSnapshot (another serialization.SCHEMA_VERSION bump,
same shape as Phase 0/Phase 3's), not a replacement for the existing
fields, so checker.compare() itself needs no change in this phase —
every existing detector keeps reading AbiSnapshot.functions/types/...
exactly as it does today; only a detector written to consume SemanticIR
directly (none exist yet) would read the new field. Without this wiring,
landing the Files list above either breaks every dump/compare
invocation (the parsers stop returning what dumper.py/service.py
assemble from) or
leaves SemanticIR fully built and fully inert beside an unchanged
production pipeline — neither is an acceptable state to merge this phase
in. An end-to-end parity test (dump/compare over a real fixture,
before and after this phase, asserting identical AbiSnapshot output) is
required for each of the five assembly call sites — dumper.py,
dumper_manifest.py, service.py's BTF/CTF dispatch, service.py's
PDB path via pdb_model.model_types_from_dwarf_metadata, and
dumper_hybrid.merge_snapshots() — not only the
first two, alongside the per-backend unit tests below, to prove the
normalizer-mediated assembly path is behavior-preserving for the existing
pipeline rather than asserted — and that fixture must include a real
ODR-duplicate or incomplete/complete declaration pair sharing one
EntityId, not only the common one-occurrence-per-entity case, per the
canonical_entities() finding above: a fixture without that shape could
pass this parity test even with the wrong (collapsing) projection, since
the collapse is only observable when more than one occurrence exists for
some identity. A separate, direct test covers semantic_ir's own
save/load round trip: construct a SemanticIR with multiple occurrences
sharing one EntityId (the same ODR-duplicate shape above), attach it to
an AbiSnapshot, write it via snapshot_to_dict(), read it back via
snapshot_from_dict(), and assert the reloaded SemanticIR.occurrences
has the same keys and values as the original — confirmed to fail against
a version of snapshot_to_dict() that still relies on plain asdict()
for this field (it raises before the assertion is even reached, per the
unhashable-key defect above) and against a version using a string-keyed
encoding (it loses the shared-EntityId, multiple-occurrence shape the
test specifically constructs). A dedicated test covers dumper_hybrid.
merge_snapshots()'s own semantic_ir reconciliation directly: a CastXML
sub-snapshot and a Clang sub-snapshot each carrying a real, distinct
SemanticIR (one entity backfilled from Clang-only facts, matching the
legacy-field reconciliation this function already performs), asserting
the merged snapshot's semantic_ir reflects that same Clang-only entity
— confirmed to fail against a version of merge_snapshots() that carries
the CastXML sub-snapshot's semantic_ir through unchanged, the exact
drift this finding caught.
Tests. Every existing per-backend regression test that currently proves "backend X handles construct Y" is kept and re-targeted at the normalizer's output for that backend's raw facts — this is a large, mechanical re-pointing, not new test design, and is the natural place to retire now-redundant backend-local duplicates of the same assertion (e.g. two nearly-identical closure-identity tests, one per backend, collapsing into one normalizer test parameterized over both backends' raw fixtures).
Acceptance criteria. Not "an identical SemanticIR regardless of
source backend" — backends genuinely differ in what evidence they can
produce (DWARF may see only emitted template instantiations where a
header AST sees uninstantiated declarations too; a given backend may be
structurally unable to produce a given fact at all, which is exactly
Fact.unsupported()'s job from Phase 0), and requiring bit-identical
output across backends could only be satisfied by discarding real
backend-specific evidence or fabricating a fact a backend never actually
observed — the opposite of what Fact[T] exists to prevent. The real bar
is narrower and is what this phase actually fixes: for the subset of
facts two backends both produce for a shared fixture, canonical
identity and spelling (EntityId/ScopePath, template-argument/
anonymous-marker/CV-qualification rendering) must agree exactly — a single
shared test fixture (one closure-parameterized template, one
partially-qualified nested type, one using-re-exported constant) asserts
that agreement on the intersection, and separately asserts each backend's
expected FactStatus for the facts only one of them can produce (e.g.
dumper_castxml.py genuinely reporting Fact.unsupported() for a fact
only the clang backend extracts) — stated as one parameterized test with
two assertions per fixture, not one assertion claiming full equality.
RESOLVED (2026-08-29, PR #943): Phase 2's implementation PR chose option
(a), so everything this paragraph and the next conditionally assign to
Phase 6 stays in Phase 2 and is NOT this phase's work. The two
paragraphs are kept verbatim below rather than deleted, because they still
record why the dependency exists and what it would have cost; read them as
the branch that did not happen. (Phase 2's own third-slice finding does
re-sequence the consumer migration within Phase 2 -- behind persistence
and the finding_identity.py move -- but nowhere near Phase 6.)
If Phase 2's implementation PR resolves its own open option-(a)-vs-(b)
question (above) as option (b), this phase's Files/Tests/Acceptance
criteria above do not by themselves cover what that choice defers here —
a review round correctly found the dependency stated in Phase 2 has no
corresponding landing task in this phase, which would leave diff_
filtering.py's/type_reachability.py's deferred post-parse consumers
permanently unmigrated, completing neither D3 nor this phase's own
acceptance bar, with nothing in either phase's checklist to catch the
gap. Stated here explicitly, conditional on that choice rather than
asserted as this phase's work unconditionally: under option (b), this
phase's own SemanticIR assembly is exactly where every declaration/type
first receives a real, resolved EntityId (CanonicalEntity's own
identity, built from the typed scope data Phase 2 establishes), so the
Files list above additionally migrates diff_filtering.py's ambiguity-
tracking helpers and type_reachability.py's remaining post-parse
consumers (named in Phase 2's own finding) to read that resolved
EntityId off the assembled SemanticIR instead of re-deriving ambiguity
from bare qualified-name strings, with their existing bespoke trackers
deleted in the same PR (folding Phase 2's own Phase-3-deletion-checklist
row for these two modules into this phase's PR when, and only when,
option (b) is the one actually chosen).
That covers only the two consumers Phase 2's own finding named —
Phase 3 itself is a third, and the larger one, and a further review
round found it still wasn't rescheduled anywhere. Phase 2's own
"not contained to Phase 2" paragraph already states the consequence for
Phase 3 directly: under option (b), Phase 3's graph builder has no
resolved EntityId to key declaration/type nodes by, since it is a
post-parse consumer in the identical position type_reachability.py's
other consumers are in, and that paragraph names "move Phase 3's
identity-dependent parts to land with or after Phase 6" as one of the two
ways to resolve it — but named the obligation without any phase's own
Files/Tests/Acceptance section actually carrying it out, leaving option
(b) a choice with no landing task for the larger of the two things it
defers. Under option (b), this phase's Files list gains Phase 3's own
identity-dependent work too, not only the two narrower consumers above:
the public-surface graph builder (compare/surface_graph.py's node/edge
construction, keyed by EntityId/canonical_key(occurrence_id)),
PublicSurfaceQuery.resolve()/resolve_public_domain()/
resolve_export_domain() (which need that same graph to query), and the
model/graph.py/AbiSnapshot.surface_graph persistence work Phase 3
otherwise lands on its own — each moves to land with or after this phase
instead of before it, since SemanticIR assembly is the first point any
of them has a real EntityId to build from under this choice. Phase 3's
own Files/Tests/Acceptance sections stay written exactly as they are
(they are still the correct description of the work, option (a) or (b))
— what changes under option (b) is purely when that work lands, stated
here rather than asserted silently resolved by a partial Phase 6 entry
covering only the smaller of the two deferred obligations — not left as a dangling forward
reference two phases back with no phase left to claim it.
Phase 7 — RunOutcome and the last inline exit-code computation¶
Landed (2026-09-02). abicheck/policy/outcome.py (new) implements
RunOutcome/PolicyGateDecision/OperationalStatus/TargetLifecycle
exactly per this section's corrected design below (severity.GateDecision
untouched). Every writer this section's Files list names emits the new
run_outcome block additively: reporter.py's four JSON entry points
(via a new leaf, report_run_outcome.py, threaded through
reporter_contract_blocks.render_json_with_side_facts's shared tail —
split out purely to keep reporter.py/service_scan.py under the
file-size hard cap, not a design change); the three buildsource/
check_report.py synthetic builders; scan_engine.ScanOutcome.to_dict()
and service_scan.ScanResult.to_dict()/ScanSetResult.to_dict(); and all
three real callers of the not-comparable refusal document
(report/not_comparable.py's two functions, cli_compare_helpers.py's
JSON branch, and cli_compare_release_pairwise.py's per-library refusal
branch — this last one is the file the plan's own Files list named
cli_compare_release.py for, corrected here against the real call site).
workflows/aggregate/gate.py reads structured-first with legacy decode as
the named fallback, exactly as designed; fold.py needed no change, as
predicted. buildsource/check_report.py's _neutralize_gate()/
_escalate_removed_library_severity() both gained the identical
run_outcome.gate treatment described below. Report schema bumped to
2.48 (compare/release)/1.24 (scan); the package schema gained the
run_outcome definition, republished to the docs mirror via
scripts/publish_schemas.py. The no-inline-gate-computation AI-readiness
check landed as scripts/no_inline_gate_computation.py (WARN), scoped per
this section's own Acceptance Criteria text (does not flag fold.py's
GateInfo.exit_code aggregation). All of this section's own Tests are
implemented in tests/test_run_outcome.py/tests/test_no_inline_gate_
computation.py. action/run.sh is untouched, exactly as this section's
own explicit exception states.
Goal. The multi-target aggregate path (gate.py/fold.py) stops
decoding and re-aggregating raw exit_code integers as semantic data;
every front end encodes RunOutcome's independent axes exactly once, at
the boundary. junit_report.py's own per-finding _is_failure is
explicitly not in this phase's scope — see the Design section's own
correction below for why a per-finding field on Change is the wrong
shape for what _is_failure answers, regardless of which layer would
stamp it.
Design. abicheck/policy/outcome.py: RunOutcome (compatibility,
assurance, gate, operational, lifecycle — each axis's underlying concept
is already real today as Verdict/AnalysisAssurance/various ad hoc
operational-status values/ADR-053's target lifecycle, just not yet one
object). The gate axis is a new type, PolicyGateDecision, not a
reuse of the existing severity.GateDecision — per ADR-063 D6's own
note: severity.GateDecision carries exit_code: int/blocking: bool/
blocking_categories, exactly the scheme-encoded data D6 bans from domain
objects, so RunOutcome.gate cannot simply be one. PolicyGateDecision
is an ordered, exit-code-free value (mirroring the IssueCategory
ordering severity.compute_exit_code already uses internally — NONE <
ADDITION_QUALITY < POTENTIAL_BREAKING < ABI_BREAKING) that the boundary
encoders convert to severity.GateDecision/a raw integer, never the
reverse. RunOutcome is report-level, not per-finding — it does not
replace junit_report.py's per-test-case classification, and this phase
does not attempt to make it. junit_report.py's _is_failure decides,
per Change, whether that individual finding fails its JUnit test case,
after contract evaluation, scoped-finding filtering, policy overrides, and
severity mapping have already run on it — exactly the per-change
granularity ADR-042 already records _is_failure as needing, and an
aggregate whole-report gate/compatibility value cannot answer "does
this change fail" for a report where only some category blocks. The fix
this phase makes is narrower than "read RunOutcome instead of changes",
and it does not reuse Change.compatibility_decision for this — a
first draft of this phase proposed exactly that and a reviewer correctly
rejected it: compatibility_decision is None/NOT_EVALUATED by design
on an ordinary, non---contract run, and for a --contract run's
excluded findings, meaning "policy did not run on this finding," per
ADR-049 D1's own documented contract — reading it as a pass/fail signal
for JUnit would either read every ordinary breaking change as an
unclassified non-failure, or require universally populating a field whose
whole point is to distinguish "evaluated" from "not evaluated," silently
erasing that distinction from the existing JSON/SARIF output every
external consumer already relies on.
No stored Change field at all — this is the resolution to the
stamping-layer question three prior rounds of this plan each tried to
answer a different way, and it closes by removing the thing being argued
over rather than picking a fourth layer to stamp it at. Every prior
attempt (compatibility_decision reuse, a checker.compare()-stamped
field, an unspecified later-layer field) shared one assumption this round's
review finally named directly: that _is_failure's per-finding answer is
a property of the Change, fixed once and read many times. It is not —
the identical DiffResult can legitimately be rendered twice with two
different SeverityConfigs and relevant_ids sets (an info-only render
and a strict-severity render of the same comparison, say), and
_is_failure is supposed to answer oppositely for the same Change in
each case. A single always-resolved field baked onto the shared Change
object can only ever be correct for the render context that stamped it;
every other render context reads a stale answer, which is a worse defect
than any of the three per-layer placements already rejected — it is wrong
by design, not by picking the wrong layer. The fix: _is_failure stays
exactly what it is today, a per-render function of (change,
SeverityConfig, relevant_ids) — junit_report.py keeps computing it
inline, unchanged, because "inline" was never the actual problem; the
problem this phase's Goal statement should have named is the aggregate
exit-code computation (gate.py/fold.py, covered below), not this
function. RunOutcome stays exactly what it always was in this phase —
a report-level aggregate, compatibility_decision keeps its existing
meaning and existing callers completely unchanged, and this phase adds
zero new fields to Change/checker_types.py. "Stops computing
inline" is corrected to mean only what Phase 7 actually closes: the
multi-target aggregate path below, not junit_report.py's per-finding
logic, which this phase now leaves alone entirely.
junit_report.py is not the only remaining inline exit-code consumer —
a first draft of this phase missed the multi-target aggregate path
entirely. abicheck/workflows/aggregate/gate.py's GateInfo.
from_report_data/from_scan_report decode a persisted report's raw exit_code integer back
into blocking/severity semantics, and abicheck/workflows/aggregate/
fold.py's own exit_code() aggregates every target's gate by max()-ing
their integer codes and branches blocking/filtering directly on
t.gate.exit_code. This is exactly the PR #700 failure mode D6 targets —
an integer read and branched on as semantic data inside the system, not
only encoded once at a boundary — and it is not an instance this
phase can treat as "just another front-end encoder," because unlike the
CLI's own _exit_with_severity_or_verdict, gate.py is parsing a
persisted report produced by a separate, possibly older, process —
the raw exit_code integer is a genuine external wire contract at that
boundary, not internal domain data this phase controls end to end.
The fix is additive to the report schema, not a behavior change to what
already-published reports mean: the report JSON gains RunOutcome's
structured axes (compatibility/assurance/gate/operational/
lifecycle) alongside the existing exit_code field — never replacing
it, since exit_code is the documented external contract
(docs/reference/exit-codes.md) and stays exactly as it is for every
external consumer. GateInfo.from_report_data reads the structured fields
when a report carries them, and falls back to decoding the legacy
exit_code only for a report that predates this change (the same
"read once, decode for legacy, never for fresh" backfill shape Phase 0
already established for Fact[...] against the old reliability flags —
this phase is the second place that exact shape applies, not a new
pattern).
Reading RunOutcome.gate alone is not enough, and a first draft of
this phase's fold stopped there. PolicyGateDecision (D6, above) only
orders compatibility categories (NONE/ADDITION_QUALITY/
POTENTIAL_BREAKING/ABI_BREAKING) — it has no slot for a scan
report's budget-overflow or not-comparable failures, which are exactly
why scan's own legacy exit-code scheme is 0/2/4/5/6, not 0/1/2/4: 5
and 6 are real, independent blocking conditions today's raw-code fallback
(gate.py::from_scan_report's existing discriminated-on-raw-code branch)
correctly keeps blocking, per that function's own docstring. RunOutcome
carries this as the separate operational: OperationalStatus axis for
exactly this reason.
A later draft of this phase's fix routed the operational axis through a
new field on fold.py/TargetReport, reviewed and found to be the wrong
layer — fold.py does not need to change at all, and TargetReport does
not need a new field, because GateInfo is already the single
representation fold.py's exit_code() reads (max(t.gate.exit_code for
t in gated ...), plus .blocking/.blocking_categories elsewhere in the
same module) and the existing codebase already folds conditions outside
PolicyGateDecision's compatibility categories into exactly that same
representation — load.py's own loader already synthesizes a blocking
GateInfo(blocking_categories=("operational_error",))/("not_comparable",)
for a report that never arrived or carried an ADR-050 D2 verdict: null
result, and from_scan_report's raw-code branch maps scan's 5/6 onto a
blocking GateInfo the identical way. The operational axis is simply a
third source feeding the one representation those two already populate,
not a second channel fold.py additionally has to consult. The fix:
gate.py's own readers — GateInfo.from_report_data/from_scan_report —
fold RunOutcome.operational into the GateInfo they return, for a
fresh report that carries the new structured fields: a blocking
OperationalStatus value (BUDGET_OVERFLOW/NOT_COMPARABLE/
EVIDENCE_CONTRACT_ERROR/EXTRACTION_ERROR, per ADR-063 D6's own
grounded definition of the type) is combined with
PolicyGateDecision's own compatibility contribution by max() over the
exit-code scheme both already share — the same orthogonal-axes shape
ADR-049 Phase 7's contract-coverage axis already uses elsewhere in this
codebase for "two independent failure axes, neither allowed to mask the
other," resolved once, inside gate.py, rather than carried as two
values for every later consumer to remember to fold themselves.
fold.py is therefore unchanged by this phase beyond no longer being
fed a GateInfo whose exit_code came from decoding a raw integer for a
fresh report — it already aggregates whatever GateInfo it is handed,
which is exactly what makes this the right layer: every existing
consumer of TargetReport.gate (fold.py's blocking_targets/
coverage_blocking/exit_code(), the CLI summary's blocking_categories
join) sees the operational axis automatically, with no second read path to
keep in sync.
Files. abicheck/policy/outcome.py (new — RunOutcome and the new,
exit-code-free PolicyGateDecision ordered type, per the Design section
above; severity.GateDecision itself is untouched, since it remains
exactly what the boundary encoders convert to). checker_types.py/
Change gain nothing in this phase — the Design section's own
correction above replaces the earlier gate_classification-field plan
entirely; junit_report.py is correspondingly not touched either,
since its _is_failure stays the unchanged, per-render function it
already is. html_report.py's CI Gate card (already RunOutcome-shaped
per ADR-042 — confirm it reads the new object directly rather than a
precursor shape, closing ADR-036 Increment 3 as a side effect if it
hasn't landed separately by then); abicheck/workflows/aggregate/gate.py
(GateInfo.from_report_data/from_scan_report read structured
RunOutcome.gate/.operational fields first, folding both into the one
returned GateInfo by max() over the shared exit-code scheme; legacy
exit_code decoding becomes the named fallback path, not the only path).
abicheck/workflows/aggregate/fold.py needs no change in this phase —
per the Design section's own correction above, it already aggregates
whatever GateInfo each target's reader returns, so folding the
operational axis into GateInfo at the gate.py layer is what makes
max()-over-raw-integers disappear everywhere downstream at once, not a
second deletion this file list has to separately track.
The report-writing side of reporter.py/aggregate.py (emit the new
structured fields alongside the unchanged exit_code, and bump
REPORT_SCHEMA_VERSION/AGGREGATE_SCHEMA_VERSION — an additive schema
change needs its version bumped the same way every prior
report_schema_version-gated field addition already did, per that
constant's own changelog comments; a first draft of this phase named the
field addition without the version bump, schema-file edit, or
regeneration that addition requires); abicheck/schemas/
compare_report.schema.json/aggregate_report.schema.json (the new
fields — the authoritative package schemas, not their
docs/reference/schemas/v1/ mirror, which a first draft of this row
named instead. scripts/publish_schemas.py's own docstring states the
direction plainly — "the package copy is the source of truth" — copying
abicheck/schemas/*.schema.json onto the docs mirror, never the reverse;
editing the mirror directly would have its hand-added fields silently
overwritten the next time anyone runs the publisher, and in the meantime
leaves the actual schema package validates fresh reports against
unchanged, so a freshly-generated report carrying RunOutcome fields
would fail validation against its own schema rather than the mirror
merely drifting). Edit the two package schema files, then run
scripts/publish_schemas.py to regenerate the
docs/reference/schemas/v1/ mirror from them — never the other order.
Three more writers are not report-reading fallback paths but
synthetic report builders, each stamping REPORT_SCHEMA_VERSION
directly and independently of reporter.py/aggregate.py — a gap a first
draft of this phase's Files list didn't catch, since these aren't shaped
like the other named writers. buildsource/check_report.py's
build_operational_error_report()/build_bootstrap_report()/
build_new_target_report() each hand-build a report dict from scratch
for exactly the three non-EXISTING cases RunOutcome.lifecycle/
.operational exist to represent (EXTRACTION_ERROR, BOOTSTRAP, and
NEW_TARGET respectively) — once REPORT_SCHEMA_VERSION is bumped, these
three would stamp the new version number on a document that still omits
the very structured axes that version bump is for, for precisely the
cases those axes were built to cover. Each gains the identical structured-
field emission the other writers in this phase add — build_operational_
error_report() emits RunOutcome.operational = EXTRACTION_ERROR;
build_bootstrap_report()/build_new_target_report() emit
RunOutcome.lifecycle = BOOTSTRAP/NEW_TARGET respectively — alongside
their existing legacy sentinel fields, unchanged. None of the three
ever computed a real compatibility verdict, which is exactly why
ADR-063 D6's compatibility field is CompatibilityVerdict | None
rather than required: all three construct their RunOutcome with
compatibility=None, the honest "no comparison ran" value, rather
than inventing one. scan_engine.
ScanOutcome.to_dict() (a separate, independent report writer a first
draft of this phase's file list missed entirely — not a sibling of
reporter.py's compare-report writer, and gate.py's GateInfo.
from_scan_report is the matching separate reader already in this
phase's file list, so leaving the writer unmigrated would mean every
freshly-generated scan report still lacks the structured fields and
keeps forcing from_scan_report onto the legacy-decode path D6 means to
reserve for genuinely old reports, not new ones). scan_engine.
ScanOutcome is not the only scan-report writer — a later review round
found two more, independent of it and of each other, missed by the first
fix in turn: service_scan.py's ScanResult.to_dict() (the typed-API
single-binary scan result) and ScanSetResult.to_dict() (the
--artifact-set sibling, ADR-056) each build their own dict directly —
verdict/exit_code as raw fields, no call into ScanOutcome or any
shared writer — so migrating scan_engine.ScanOutcome alone would leave
exactly these two typed-API paths (and ScanArtifactResult.to_dict(),
which wraps ScanResult.to_dict()'s output unchanged) emitting the old,
unstructured shape while stamping the newly-bumped SCAN_SCHEMA_VERSION
— a document claiming a schema version it doesn't actually carry the
fields of, which is a worse state than not bumping the version at all.
Both gain the identical structured RunOutcome-axis fields
ScanOutcome.to_dict() adds, alongside their existing verdict/
exit_code fields (additive, same as every other writer in this phase).
scan reports carry
their own independent SCAN_SCHEMA_VERSION (schemas.py) — genuinely
separate from REPORT_SCHEMA_VERSION/AGGREGATE_SCHEMA_VERSION, not
another name for one of them, since scan_engine.py/service_scan.py
stamp it on their own report shape. Emitting the new structured fields
from this writer needs SCAN_SCHEMA_VERSION bumped too, for the identical
reason the compare/aggregate counters are bumped above — a freshly
regenerated scan report with the new fields but an unbumped
scan_schema_version reads as the old schema to any version-aware
consumer and defeats the whole point of versioning the additive change.
Two more writers, found after the synthetic-writer correction above,
are the ones this phase exists for most directly and were still missing:
the two paths that actually produce a verdict: null NOT_COMPARABLE
report. report/not_comparable.py's not_comparable_document() — the
ADR-050 D2 comparability-refusal document checker.compare's own gate
raises before any DiffResult exists to build a report from — and
cli_compare_release.py's per-library refusal branch (the release
fan-out's own inline report_schema_version: REPORT_SCHEMA_VERSION
construction for the identical refusal, independent of the shared
document builder) both stamp REPORT_SCHEMA_VERSION directly with no
RunOutcome fields at all. Once this phase's schema bump lands, a freshly
produced refusal report from either path claims the current schema while
omitting RunOutcome.operational = NOT_COMPARABLE — the exact axis this
whole phase exists to make structured-first — forcing GateInfo.
from_report_data's new reader back onto the legacy-decode fallback this
phase means to reserve for genuinely old reports, on a report that isn't
old at all. Both gain RunOutcome.operational = NOT_COMPARABLE alongside
their existing verdict: null/reason fields; not_comparable_document()
takes the value as an explicit parameter the same way it already does for
report_schema_version (per that function's own stated reason — a report
schema version it does not itself own — which applies identically to a
RunOutcome axis it likewise must not hardcode), and cli_compare_
release.py's refusal branch passes it through the identical way it
already threads report_schema_version.
Making not_comparable_document()'s new parameter required breaks a
third path neither of the two writers above is the caller for, and a
review round correctly traced the real call chain to find it: the
normal, single-pair compare --format json refusal. cli_compare_
helpers.py's own comparability-gate handler calls render_not_comparable_
json() — not not_comparable_document() directly — which itself calls
not_comparable_document() one layer down; neither
render_not_comparable_json()'s own signature nor its one real call site
in cli_compare_helpers.py gained the new parameter, so an ordinary
compare invocation that hits a profile/scope mismatch (ADR-050 D1/D2,
the most common way a user actually reaches this refusal path, not only
the release fan-out) would either raise TypeError on the now-required
argument or — if the parameter were merely added without being threaded
here — silently keep stamping the current schema version with no
RunOutcome.operational at all, the identical gap this paragraph exists
to close, just one call deeper than where the first fix stopped.
render_not_comparable_json() gains the identical parameter, threaded
straight through to not_comparable_document() the same way it already
threads report_schema_version; cli_compare_helpers.py's own call site
passes operational=NOT_COMPARABLE explicitly, the same value the
release fan-out's own refusal branch now passes. Added to this phase's
writer inventory and to the schema-version parity tests alongside the
other writers above — three real producers of this document now, not
two, each verified through its own actual call chain rather than only at
the shared builder's own signature.
A fourth writer-adjacent call site needs this phase's attention, and it
is not a new writer — it is an existing neutralizer, missed by the
first draft because its own subject is mutating an already-written
report, not producing one: buildsource/check_report.py's
_neutralize_gate(). check-project.yml's gate-mode: advisory path
zeroes a report's legacy severity/exit_code contribution in place
(that function's own docstring: "Only advisory reports are rewritten
this way"), and its own accumulated review history already lists three
prior rounds of exactly this shape of bug — zeroing only the top-level
field left a nested diff.severity block, or the orthogonal
contract-coverage contribution, still driving the trailing aggregate
job to a nonzero exit, each caught and fixed in turn. RunOutcome.gate is
a fourth axis this function does not know about yet, and GateInfo.
from_report_data's own new structured-field-first reading (this phase's
own change, a few paragraphs above) is exactly what makes the omission
land: once a fresh report's structured fields are preferred over the
legacy ones this function does zero, an unchanged, still-blocking
RunOutcome.gate value overrides the neutralization entirely, and an
explicitly advisory check blocks the trailing aggregate anyway — the
identical failure mode this function's own history keeps rediscovering,
reached through the one axis this phase adds rather than one of the three
it already covers. _neutralize_gate() gains the identical treatment for
that one axis: zero RunOutcome.gate's own blocking contribution the same
way it already zeroes severity/exit_code/the nested scan-shaped block/
the coverage contribution — one more axis added to a function whose whole
job is "every compatibility-policy axis that can block, zeroed for
advisory."
RunOutcome.operational is deliberately excluded from that
treatment, and a first draft of this paragraph said to zero it
alongside .gate — review correctly caught that as reproducing the exact
class of bug this function exists to prevent, just on the opposite axis.
check_report.final_exit_code()'s own docstring states the invariant in
so many words: "Operational errors... always fail the job regardless of
gate-mode... resolve-baseline's failure taxonomy is never silently
degraded to a passing/neutral outcome either," and its implementation
returns 1 on operational_error unconditionally, before it even
branches on gate_mode. RunOutcome.operational (BUDGET_OVERFLOW/
NOT_COMPARABLE/EVIDENCE_CONTRACT_ERROR/EXTRACTION_ERROR) is exactly
this same signal in the new structured shape — an analysis that never
produced a real compatibility verdict at all, as opposed to one that did
and scored it ABI_BREAKING. Zeroing it under advisory would let a scan
that hit a budget overflow or a hard evidence-contract error read as a
clean, non-blocking pass once a consumer prefers the structured fields —
silently degrading exactly the failure taxonomy final_exit_code() says
must never be degraded, and confirming the reviewer's point that .gate
and .operational are not one axis that happens to share a dataclass:
.gate is the thing advisory is for deferring, .operational is the
thing no gate-mode may ever defer.
A fifth mutator needs the identical structured-field treatment, found
alongside _neutralize_gate() but pulling in the opposite direction — it
escalates rather than neutralizes, and the same
structured-field-preferred reading can silently erase an escalation the
same way it could silently keep a neutralized gate blocking.
buildsource/check_report.py's _escalate_removed_library_severity() is
augment_report()'s own fold for --fail-on-removed-library's exit 8 —
the one case, per that function's own docstring, where the composite
Action's real process exit code can diverge from what the report body
itself persisted, since compare_release_cmd's _exit_compare_release
applies exit 8 "in preference to the severity code." It writes only the
legacy severity dict (exit_code/blocking/blocking_categories) —
augment_report() has no RunOutcome.gate field to write yet at all, so
once GateInfo.from_report_data is reading the structured fields first,
a report this exact function escalated reads RunOutcome.gate = NONE
(never populated, defaulting to non-blocking) while its legacy severity
block correctly reads blocking: true — a deferred aggregate job reading
the preferred, structured field sees no escalation at all and passes a
release whose removed-library gate the caller explicitly asked for,
silently, with no warning and no test currently covering this exact path.
_escalate_removed_library_severity() gains the matching structured
write — RunOutcome.gate = ABI_BREAKING (the same tier its existing
legacy write already encodes: "a whole library disappearing is
unambiguously an ABI break," per that function's own docstring) — folded
in at the identical call site augment_report() already has
(analysis_exit_code == _REMOVED_LIBRARY_EXIT_CODE), alongside the
existing severity write, never replacing it. A dedicated regression
reproduces the exit-8 path end to end through augment_report() itself
(not a hand-built report dict) and asserts GateInfo.from_report_data
reads a blocking result from the escalated report under every gate mode
except advisory — confirmed to fail against a version of
_escalate_removed_library_severity() that writes only the legacy block,
which is exactly today's code.
Tests. tests/test_junit_report.py's existing suite needs no changes
at all — _is_failure is untouched, so this is the test that proves the
Design section's own correction is real: if any of these tests needed to
change, this phase would have reintroduced a Change-level field by
another name. A
second parity test for the aggregate path: fold.py's exit_code()/
blocking output is unchanged for every existing tests/test_aggregate.py
fixture, run twice — once against a report carrying only the legacy
exit_code field (proving the fallback decode path reproduces today's
behavior exactly) and once against a report regenerated with the new
structured fields (proving the new path agrees with the old one on every
existing fixture, not only on fixtures written after this phase).
Explicitly included in that fixture set, not left to be covered
incidentally: the two scan-specific exit codes PolicyGateDecision
alone cannot represent — a fresh scan report carrying exit 5 (budget
overflow) and one carrying exit 6 (not-comparable), each constructed with
the new structured RunOutcome fields, asserted to still produce a
blocking GateInfo from gate.py's own reader and to still aggregate as
blocking through fold.py's unchanged exit_code() — confirmed to fail
against a gate.py reader that folds only RunOutcome.gate into the
returned GateInfo and ignores .operational entirely, which is the
exact regression this finding caught (fold.py itself needs no
corresponding test change, since it is not touched by this phase). A third
parity test covers the writers this phase adds — all three of them, not
only scan_engine.ScanOutcome: a freshly-generated scan report
(ScanOutcome.to_dict()), a freshly-run typed-API ScanResult.to_dict(),
and a freshly-run --artifact-set ScanSetResult.to_dict() each carry
the new structured fields, and GateInfo.from_scan_report() reading any
of the three fresh reports takes the structured-field path, not the
legacy-decode fallback — confirmed by
asserting which path actually ran (not only that the output matches),
since a test that only checks the output could pass with the writer
changed and the reader still silently falling back. A parity test for
_neutralize_gate() pins the new-axis gap directly, and deliberately
pins the two axes to opposite outcomes rather than asserting one blanket
"non-blocking" result — asserting the same thing for both would silently
reproduce the .operational bug this same finding's own fix exists to
prevent, just inside the test instead of the implementation. Case one: a
fresh report carrying a blocking RunOutcome.gate value (no operational
failure), run through _neutralize_gate() under gate-mode: advisory,
then read back through the same GateInfo.from_report_data/
from_scan_report this phase's reader changes use — asserting the
aggregate sees a non-blocking result, confirmed to fail against a version
of _neutralize_gate() that zeroes only the pre-existing legacy axes and
leaves .gate untouched, reproducing the exact "advisory check blocks the
trailing aggregate anyway" failure mode this function's own prior review
rounds already fixed three times for other axes. Case two: a fresh report
carrying a non-blocking RunOutcome.gate alongside a real
RunOutcome.operational failure (e.g. EVIDENCE_CONTRACT_ERROR), run
through the identical _neutralize_gate()/gate-mode: advisory path —
asserting the aggregate still sees a blocking result, confirmed to
fail against a version of _neutralize_gate() that zeroes .operational
alongside .gate (this finding's own rejected first draft), which is
precisely what final_exit_code()'s "operational errors always fail the
job regardless of gate-mode" invariant forbids. A fourth test
validates every regenerated fixture report against the regenerated
docs/reference/schemas/v1/compare_report.schema.json/
aggregate_report.schema.json (the same validation
scripts/verify.py's fair-metadata step already runs for generated
files), so the new fields are provably reflected in the published schema
mirror, not only in the Python writer. A fifth test covers the three
synthetic builders directly: calling each of build_operational_error_
report()/build_bootstrap_report()/build_new_target_report() and
asserting the returned document carries the correct, non-EXISTING
structured axis (RunOutcome.operational/.lifecycle respectively) —
confirmed to fail against a version of each builder that stamps the
bumped REPORT_SCHEMA_VERSION without adding the matching structured
field, the exact "claims a schema version it doesn't carry the fields of"
defect this phase's own ScanOutcome/ScanResult/ScanSetResult
migration above is already written to avoid.
Acceptance criteria. Zero remaining inline exit-code/severity
computation outside the one designated encoder per front end — enforced
by a new check_ai_readiness.py check (no-inline-gate-computation,
WARN) flagging a severity/exit-code literal compared against Change
data, or a .gate/.operational-shaped RunOutcome axis decoded by
max()/comparison against a raw integer, outside policy/outcome.py and
the per-front-end encoders (the widened check is what actually closes the
gap the first draft's narrower, Change-only check left open: gate.py's
decode of a persisted report's raw exit_code never touches Change at
all, so a check scoped to Change comparisons alone would never have
flagged it). fold.py's own max(t.gate.exit_code for t in gated ...)
is not a violation this check should flag — per the Design section's
own correction above, GateInfo.exit_code is by that point already the
output of gate.py's structured decode (both axes already folded by
max() over the shared exit-code scheme), not a raw integer read back
off the persisted report a second time; the check distinguishes the two
by scope (gate.py and policy/outcome.py are where a RunOutcome axis
may be decoded from raw fields at all) rather than by forbidding every
max() over a .exit_code attribute outright, which would also flag
fold.py's own legitimate aggregation-by-max() over many targets'
already-decoded gates. Stated explicitly, matching ADR-063 D6's own
restated encoder list: gate.py reads structured RunOutcome.gate/
.operational fields first and folds both into the one GateInfo it
returns (legacy exit_code decoding as the named fallback, never the
only path for a fresh report); fold.py aggregates those already-folded
GateInfo values across targets, unchanged by this phase; and
fold.py::exit_code() is the one place that aggregated value converts to
the integer aggregate's own JSON output and process exit code need —
two decode/encode steps (gate.py in, fold.py::exit_code() out), not
three, because folding the operational axis into GateInfo at the read
boundary means there is no longer a third, separate step left over to
name.
The acceptance check above cannot actually establish the "zero remaining
inline exit-code/severity computation" bar it states, because it never
touches action/run.sh at all — a review round correctly found this
phase's Files list never migrates it, even though ADR-063 D6 already names
"the Action's own encoder" as one of exactly four boundary encoders this
decision covers. check_ai_readiness.py's no-inline-gate-computation
check is a Python AST walk; action/run.sh is bash, structurally invisible
to it. Reading the real script confirms the gap is not cosmetic:
_severity_gate_exit() still reads severity.exit_code from the JSON
report directly, and the main case $ABICHECK_EXIT in ...) blocks (the
scan/dump/deps paths) reconstruct which message/annotation to emit
from the bare process exit integer, not from the report's own structured
RunOutcome.gate/.operational fields — exactly the "semantic decision
computed by branching on an integer exit code" shape D6's own "no domain
or workflow code" rule targets, just in a language this phase's tooling
cannot enforce against. ADR-063 D6's framing — "the Action's own encoder...
already owns exactly this conversion... this is a new input type for an
existing function" — overstates what landing Phase 7 alone actually
changes for the Action: nothing in this phase's Files list touches
action/run.sh, so its raw-exit-code branching is exactly as unmigrated
after this phase as before it. Not migrated in this phase, named here as
an explicit, scoped exception rather than a silent gap in this phase's own
accounting — matching this plan's own established pattern for a real,
separately-justified residual (the binary-less dump --sources path in
Phase 1; the two cli_compare_release.py branches in Phase 4): a correct
migration means rewriting action/run.sh's several hundred lines of
case/annotation logic to read RunOutcome.gate/.operational from the
JSON report via jq instead of branching on $ABICHECK_EXIT, re-verified
against every one of its existing GH Action annotation/step-summary
behaviors end to end — a real, large, separately-scoped rewrite of a
shell script this phase's Python-only tooling cannot even test-cover, not
a drive-by addition to this phase's Files list. Until that future phase
lands, action/run.sh stays a raw-exit-code consumer for its own
messaging logic, and the "zero remaining inline exit-code/severity
computation" acceptance bar above is scoped to the Python front ends
(cli.py, service.py, aggregate) this phase's own check can actually
see.
Phase 8 — wire storage v2's writer/reader to the domain layer (closes ADR-062 Phase 1, jointly with D8)¶
Update (2026-09-02, landed — full D8 section split plus default CLI
wiring as a single-file shape; not the whole phase). ADR-062 Phase 0's
primitives are no longer inert: every legacy document field is split
across D8's section vocabulary, and dump/compare/scan --against
now read/write that split by default — packaged as one JSON document
(storage.sectioned_document), not the directory-backed package this
phase originally targeted (see the CLI-wiring paragraph below for why).
abicheck/storage/dto.py is the D8-constrained
SectionDTO envelope this phase's Files section asked for, built on
storage/semantic_ir_codec.py's existing explicit SemanticIR encoding
(extracted into a pure semantic_ir_to_document/semantic_ir_from_document
pair this DTO layer calls, rather than duplicating it). abicheck/storage/
legacy_sections.py (new) is what closes this slice's previously-open
gap: split_legacy_document/join_legacy_document partition every
remaining document field across D8's binary/declarations/types/
layout/debug/build/graph/provenance sections via one explicit,
per-section field allowlist (an unassigned or misfiled field is a hard
error, never silently dropped into a catch-all) — a completeness test
enumerates AbiSnapshot's real dataclass fields and fails until each has a
section. abicheck/storage/import_v1.py's import_legacy_snapshot/
export_legacy_snapshot (the second now new too — the exact inverse) write/
read one SectionDTO per present section. abicheck/project_snapshot_store.py
(flat-root — this phase's directory-backed writer, kept outside storage/
for the same import-layering reason package.py's own docstring already
gives) is a real DirectoryObjectStore plus a manifest/ref writer and
reader implementing D6's directory layout (everything except the .tar.zst
transport form); the new legacy-document round-trip functions
(write_legacy_snapshot_package/read_legacy_snapshot_document/
is_project_snapshot_package_dir) live in a sibling,
abicheck/project_snapshot_legacy.py, since project_snapshot_store.py
itself had only two lines of headroom under the 800-line architecture cap.
scripts/check_ai_readiness.py's project-snapshot-dto-no-asdict check
now also watches legacy_sections.py.
CLI wiring — redesigned (2026-09-02) as the default single-file shape,
not a directory package. The directory package's real value (content
dedup, independent per-section objects) only pays off once a project shares
content across multiple artifacts, which nothing produces yet; for the
single-artifact case every dump performs today it was pure storage-UX
cost. storage.sectioned_document packages the identical D8 split as one
JSON document instead, wired into serialization.snapshot_to_json/
write_snapshot (write) and snapshot_from_dict/load_snapshot (read,
transparently unwrapping either shape) — every dump/compare/scan
invocation gets it by default, no flag, and an older flat .abi.json a
prior build wrote stays fully readable. The directory writer/reader
(project_snapshot_legacy.write_legacy_snapshot_package/
read_legacy_snapshot_document) remain available as typed-API primitives;
compare/scan --against still accept a directory package as an input
path (workflows.input_resolution.resolve_input's directory branch,
cli_resolve.classify_compare_operand/frontends/cli/scan_against.py's
validation distinguishing it from a plain directory-of-libraries operand
and a BuildSourcePack's identically-named manifest.json) — but no
dump CLI flag writes one today. What this slice does not attempt:
decoding a section's own internal shape into a typed domain object beyond
semantic_ir (each section still carries the pre-existing JSON encoding
for its fields); multi-artifact packages (a real multi-library
ProjectSnapshot); folding baseline sets/BundleFacts into sections, the
.tar.zst transport form, bundle_variants: config wiring (A1.4-A1.7),
and non-ELF membership specifics beyond ArtifactRef.kind (A1.8) remain
open. See docs/contribute/plans/
storage-format-v2.md's "Landed in Phase 1" section and
docs/contribute/adr/062-project-snapshot-storage-v2.md's Status for the
authoritative, jointly-maintained account — this note exists so a reader of
this plan doesn't have to cross-reference the ADR to learn Phase 8 is no
longer purely a proposal.
Goal. ADR-062 Phase 0's primitives stop being inert. A real
ProjectSnapshot can be written and read, using Fact[T]/EntityId from
Phases 0/2 as its domain representation rather than a second identity/
availability scheme invented at the storage layer.
Design. This phase is ADR-062 Phase 1 (the v1-v25 import adapter, the
directory-backed ObjectStore, folding baseline sets/BundleFacts into
sections) executed with this plan's D8 constraint already in force:
every DTO is a distinct, versioned class from the domain SemanticIR/
Fact[T]/EntityId objects, with an explicit to_dto()/from_dto()
(never asdict/a 500-line mirror deserializer) and a migration adapter per
DTO version. Doing this jointly with ADR-062 Phase 1 (rather than landing
Phase 1 first, unconstrained, and retrofitting D8 after) avoids writing a
throwaway first version of the writer/reader.
Files. abicheck/storage/package.py (already has the object model —
PackageManifest/VariantRef/ArtifactRef/ObjectRef/ObjectStore;
this phase adds the directory-backed implementation and the writer);
abicheck/storage/dto.py (new — the SnapshotDTO/ProjectSnapshotDTO
classes D8 requires); serialization.py (the legacy asdict-adjacent
snapshot_from_dict path is the one this phase's D8 constraint exists to
prevent from growing a ProjectSnapshot-shaped sibling).
Tests. Per ADR-062's own validation-corpus plan, plus a D8-specific test: renaming an internal domain field (a synthetic identity key, a reordered dataclass field) must not change any persisted DTO's bytes — stated as a property test generating domain-object mutations outside the DTO's own declared field set and asserting the serialized bytes are unchanged.
Acceptance criteria. Matches ADR-062 Phase 1's own acceptance
criteria (see that ADR and the storage-format-v2.md plan) plus: zero
direct asdict/mirror-deserializer call sites for any ProjectSnapshot-
related type, enforced by the same AI-readiness-style check this plan's
earlier phases already establish as the pattern.
Phase 9 — selector/suppression/reclassification consolidation (D10)¶
Landed (complete). abicheck/policy/selectors.py's SelectorSet is the
one selector grammar Suppression.__post_init__/ReclassifyRule.
__post_init__ each construct internally and delegate all validation/
matching to, exactly as designed below — including both selectors this
section's own history flags as easy to omit (binding, finding_id) and
the finding_id-matcher upward-dependency fix this section's own Design
section calls out by name (a first draft did make that mistake again here;
it was caught and corrected the same way before landing). The namespace-glob
compilation machinery (_SegmentGlobMatcher and everything it depends on)
moved into a sibling leaf, abicheck/policy/selectors_namespace_glob.py,
purely to keep both files under the architecture gate's 800-line ceiling —
not part of the original design, a mechanical split forced by the moved
code's own size (the same reason diff_types_vtable.py exists). Every
pre-existing suppression.py/reclassify.py selector test is unmodified
and green; a new tests/test_policy_selectors.py exercises the shared leaf
directly. reclassify.py's importlib.import_module workaround and its
_suppression_cls() helper are both deleted, replaced by a static
from .policy.selectors import SelectorSet — confirmed via a repo-wide
grep that no importlib.import_module call remains in that module.
scripts/check_architecture.py gained the selector-leaf-purity check
this section's own Files subsection specifies, exercised in
tests/test_architecture_check.py against a deliberately-reintroduced
cyclic-import fixture (one test per denylisted module, plus the
submodule-import and no-op-when-absent cases) to confirm it fails closed.
The paragraph above is this phase's own user-facing summary — ADR-063
itself no longer restates per-phase detail (its duplicated status block was
removed by PR 0, 2026-09-02); see this plan's own pointer paragraph at the
top of the document for the current set of status sources.
Goal. suppression.py and reclassify.py share one selector-matching
primitive instead of two independent grammars kept in sync by hand, and
reclassify.py's importlib.import_module workaround for an import cycle
is removed because the cycle it works around no longer exists.
Design. abicheck/policy/selectors.py (new, leaf — zero dependency on
checker_types, suppression.py, reclassify.py, or reporter, per
ADR-063 D10): the selector grammar (symbol/symbol_pattern/
type_pattern/member_name/namespace/entity_namespace/
cause_namespace/source_location/change_kind/binding/finding_id/
expires) — binding and finding_id are listed explicitly here
because a first draft of this phase omitted both, in two different ways.
binding is shared by both Suppression and ReclassifyRule (ELF
symbol-linkage matching, Suppression._matches_binding), and existing
tests cover weak/global binding rules. finding_id is narrower —
Suppression.finding_id only, not ReclassifyRule, matched via
_matches_finding_id() against finding_identity.
report_canonical_finding_id(change) — but it is a standalone-sufficient
selector (an exact match on the producer-agnostic canonical finding
identity needs no other field to narrow it), so it is still part of the
shared grammar suppression.py's matcher must keep evaluating even though
reclassify.py never uses it; the shared module's leaf-matcher contract
covers the union of both classes' fields, not their intersection, and a
consumer that doesn't use a given field simply never sets it.
The finding_id matcher itself must not call finding_identity.
report_canonical_finding_id from inside the leaf module — a first draft
of this phase's Files section did exactly that, and it is the same
upward-dependency mistake Phase 2 caught and corrected for
model/identity.py, recreated here. finding_identity.py is
comparison-layer logic that imports checker_types/model entities to
compute its answer, so policy/selectors.py calling into it would depend
upward on compare/-level code — precisely the edge this leaf module's
own "zero dependency on checker_types/suppression.py/reclassify.py/
reporter" contract exists to forbid, and the existing architecture-gate
check this phase adds (see Files below) would not even catch it, since
that check's denylist names those four modules specifically, not
finding_identity.py. The fix follows the same shape Phase 2 already
established for exactly this situation: the leaf matcher never computes
the canonical finding id itself — it only compares a string. The
caller (Suppression.selector_matches()/suppression.py, which already
imports finding_identity.py today and is comparison-layer code, not a
leaf) computes report_canonical_finding_id(change) once and passes the
resulting string into the shared matcher alongside the Change, so
policy/selectors.py's finding_id check is a plain string-equality
comparison against an already-computed value, with no import of
finding_identity.py anywhere in the leaf module. Dropping
either from the shared grammar would either lose a supported selector
outright or leave its matching logic as a second, un-consolidated
implementation sitting next to the new leaf module — exactly what this
phase exists to remove, not reintroduce. The corrected list above is every
selector field either class currently supports, not a subset — extracted
from suppression.Suppression's existing selector_matches() — already
the real, shared logic reclassify.py calls today, just reached through
the import-cycle workaround rather than a dependency-free module. Once the
grammar itself lives in a leaf package with no edge back to checker_types
or policy_file, reclassify.py can import it statically — the cycle
policy_file -> reclassify -> suppression -> checker_types -> policy_file
that reclassify.py's own docstring names as the reason for the
importlib.import_module workaround no longer exists, because neither
reclassify.py nor suppression.py needs to import the other anymore —
both import the shared leaf instead. Suppression/ReclassifyRule keep
their own, distinct outcomes (delete the finding vs. reclassify its
verdict) — D10 consolidates the matching grammar, not the two rule types'
different actions, which remain genuinely different decisions and are not
an instance of the "one concept, two representations" problem this plan
otherwise targets.
Files. abicheck/policy/selectors.py (new — includes the
finding_id matcher as a plain string-equality comparison against an
already-computed canonical id, per the Design section above; the leaf
itself never imports finding_identity.py); suppression.py
(Suppression.selector_matches() becomes a thin wrapper calling the
shared matcher — computing finding_identity.report_canonical_finding_id
itself, as it already does today via _matches_finding_id(), and passing
the resulting string into the shared matcher — or is removed in favor of
direct calls, whichever keeps Suppression's own public method surface,
including its existing parse_finding_id-based construction-time
validation, intact for existing callers);
reclassify.py (drops the importlib.import_module workaround and its
own docstring's cycle justification, replaced by a static import of
policy/selectors.py); scripts/check_architecture.py's import-direction
gate (ADR-061) gains a check that policy/selectors.py itself imports
nothing from policy_file.py/checker_types.py/suppression.py/
reclassify.py/finding_identity.py — finding_identity.py is added to
the denylist explicitly, not assumed covered by the other four, since it
is exactly the module a first draft of this phase tried to import from
the leaf and the module name alone gives no hint it belongs on this list
unless named — so a future change cannot silently reintroduce the same
cycle through the new leaf module.
Tests. Every existing suppression.py/reclassify.py selector test is
kept and re-targeted at the shared matcher — this phase's acceptance bar
is that no selector-matching behavior changes, only where the grammar
lives. A new test asserts reclassify.py contains no
importlib.import_module call at all (confirmed to fail against the
pre-phase code, which has exactly one, per that module's own docstring),
and scripts/check_architecture.py's widened gate is exercised directly
against a deliberately-reintroduced cyclic import in a throwaway fixture
module to confirm it actually fails closed.
Acceptance criteria. reclassify.py's importlib.import_module
workaround is deleted, not kept "for safety" alongside the static import —
per the Governing Invariant, a workaround for a cycle that no longer
exists is itself a stale second path. suppression.py's selector grammar
is the shared leaf module's, not a second copy. FP-rate/mutation-score
gates show no regression (this phase moves matching logic, it does not
change what matches).
Phase 10 — delete the superseded representations¶
Goal. Every phase above is only complete once its "before" state is removed, not left as a second path. This phase is the accounting pass, not new design.
Checklist (one row per phase, each a real PR removing code):
- Phase 0: the domain-side
AbiSnapshot.clang_*_facts_reliableboolean attributes are removed once every consumer reads theFact[...]field instead. Not removed, ever, per Phase 0's own corrected design: the wire-level decode of those same keys for a pre-Fact[...]persisted snapshot —serialization.py's legacy-schema backfill path is a permanent reader, the same way every other schema-version branch in that module is, for as long as ADR-062's v1-v25 import adapter promises to keep importing that version at all. A second, separate row for the same phase: the retained legacy compatibility-bridge attributes themselves —RecordType.vtable/bases/virtual_bases/vptr_offset_bits,Param.is_va_list— are removed from the public dataclasses once the widened, repository-wide legacy-attribute-read check Phase 0's own Acceptance criteria adds reports zero remaining readers outside__post_init__/serialization, closing the "kept for one release" window that section's own design states rather than leaving it open-ended; a first draft of this plan said this removal happened in Phase 5, which never touches these four fields at all (see Phase 0's own corrected text above). - Phase 1:
cli_dump_helpers.render_dump_dry_run()'s independent resolution logic; the legacy-p/--compile-dbauto-match's standalone code path once the fold fully subsumes it (already partly done per AGENTS.md's "legacy-match overlap" record — this is closing the remainder). Not in this accounting, and not silently assumed closed by it:cli_buildsource.dump_source_only(), the binary-lessdump --sources/--build-infopath — per ADR-063 D1's own named exception, it remains a third, independent dump assembler with no phase in this plan scheduled to migrate or retire it; closing it is a real, separately-justified future phase, not a residual of Phase 10's cleanup. - Phase 2:
diff_filtering.py/type_reachability.py's bespoke string- suffix ambiguity trackers. Closed (2026-09-11), in the shape the two prior "Consumer 1"/"Consumer 2" migrations already established rather than a further rewrite: a re-audit for this row found a third, previously-unauditeddiff_filtering.pyopaque-suppression path,_downgrade_opaque_struct_changes(the DWARF-oriented asymmetric- existence sibling of the already-migrated_downgrade_opaque_type_changes), still comparing barec.symbol in truly_opaquestrings — missed by every earlier slice because those slices' own text names only_find_opaque_types/_downgrade_opaque_type_changes. Migrated ontocompare/opaque_types.OpaqueTypeIndexvia a newOpaqueTypeIndex.build(declarations)classmethod (stableEntityIdfirst,RecordType.namespelling second, alwaysstrict=Falsesince this index is not a pairedintersect()and carries no completeness proof). The function's ownopaque_types/embedded_types/truly_opaqueset construction stays plainset[str]— both are rendered-text spelling questions, not identity ones, the same distinctionfind_by_value_typesalready draws. Re-verified, not re-litigated, the three prior slices' (ninth/twelfth/fourteenth) finding that the rest oftype_reachability.py's own machinery (_spelling_index/_stripped_signature_spelling/_typedef_spelling_targets/_namespace_suffix_spellings) stays on raw strings by design: those strings must appear inside rendered signature text rather than stand for an entity, and the module performs no cross-snapshot pairing at all, so there is no old/new identity question for a stable tier to improve. See theidentityconcept'sremoval_gateindocs/_meta/one-semantic-pipeline-status.yamlfor the full account andtests/test_opaque_identity_tiers.py'sTestDowngradeOpaqueStructChangesIdentityTiers/TestOpaqueTypeIndexBuildPropertiesfor the tests. - Phase 3:
surface.py's pre-graph traversal implementation andexport_surface.py's independent closure walk, oncePublicSurfaceQuery.resolveis the only path either one calls; the original, in-place copies ofGraphNode/GraphEdge/GraphFact/FactConflict/merge_graph_factsinbuildsource/graph_facts.pyonce every caller reads them frommodel/graph.pyinstead of the re-export shim. A second, separate row for the same phase:BuildSourcePack. source_graph's own live-alias mechanism is removed once the five named readers (internal_leak.py,buildsource/cross_source_checks.py,buildsource/ evidence_report.py,evidence_depth.py,cli_graph.py) are migrated to readAbiSnapshot.surface_graphdirectly — a review round correctly found the alias's own Phase 3 text named this migration as "real, scoped, follow-up work" with no phase ever actually scheduled to do it, and no Phase 10 row removing the alias once it was; left as stated, the two attribute paths for one mutable graph persist indefinitely, which is exactly the kind of drift risk ("a later assignment to either path alone could make old and new consumers diverge again") the alias's own one-object guarantee was built to avoid, not accept permanently. This row's migration is the same five-reader audit Phase 3's own text already deferred, made concrete instead of left open-ended: migrate each reader (verified against its own existing tests, per Phase 3's own reasoning for why this wasn't attempted in that phase), then delete the in-memory alias assignment in the L5 builder — the one piece this row can actually remove once every reader stops going through it.
Five-reader migration landed, with a security correction to the
preference order this row's own text originally specified (PR #1216
review). All five named readers (internal_leak.py,
buildsource/cross_source_checks.py's two call sites,
buildsource/evidence_report.py, evidence_depth.py, cli_graph.py) now
read graph = (snap.build_source.source_graph if snap.build_source else
None) or snap.surface_graph — build_source.source_graph first, not
surface_graph first as this section originally specified. A security
reviewer correctly found that a real --sources/--build-info embed can
leave build_source.source_graph a strictly richer, real L3-L5 evidence
graph than the always-on, header-only-only surface_graph
(_attach_header_graph builds the latter from headers alone and never
updates it once written; buildsource/embed.py's backfill only ever
adopts the header-only graph into an empty build_source.source_graph,
never displacing a real one) — so the originally-specified
surface_graph-first order silently drops real call-graph/dependency edges
whenever the two diverge, letting a genuinely-reachable internal removal
be misjudged unreachable and suppressed under a
reachability: proven-unreachable-only policy (a BREAKING→COMPATIBLE
false negative). surface_graph is still the correct fallback for a
depth-projected snapshot, where policy/depth_projection.py clears
build_source.source_graph at the "source" depth floor while retaining
surface_graph (an L2 fact) down to the "binary" floor, and for a
pre-Phase-3 document carrying build_source.source_graph with no
surface_graph at all — the corrected order still resolves both cases
identically to the original one, since it only changes behavior when the
two graphs are genuinely different objects. The pack-aware equivalent for
a call site whose pack may be an out-of-band one the caller resolved
independently of the snapshot (evidence_report._side_source_graph/
evidence_depth._l5_payload_empty) applies the same corrected order,
still only substituting surface_graph when the pack in hand actually
is snap.build_source, never for an unrelated
--old/new-build-info/--old/new-sources pack, per those two modules'
own pre-existing "never default pack to snap.build_source" contract.
cli_graph.py's _load_source_graph has no AbiSnapshot in scope at all
(it loads a bare graph JSON file or an out-of-band pack directory), so its
migration is the dict-shaped analogue: a full embedded-snapshot JSON
document (flat or the current single-file sectioned wire format, schema
v42+ — a second review-round finding, since the first draft's hand-rolled
dict walk missed the sectioned shape every real dump --sources -o
snapshot actually uses today) is now decoded through
serialization.snapshot_from_dict and its graph read with the same
corrected preference order. tests/test_surface_graph_reader_migration.py
pins output parity between the pre-Phase-3 shape (surface_graph absent)
and the post-Phase-3 shape (surface_graph populated as the same
object) for all five, plus a dedicated "prefers the richer
build_source.source_graph" regression test per reader proving the two
graphs are read correctly when they are genuinely different objects.
Two further corrections from the same review round. (1) The
SurfaceGraphLike narrow in cross_source_checks.py/
evidence_report.py used assert isinstance(graph, SourceGraphSummary)
to satisfy mypy — but SurfaceGraphLike (model/graph_facts.py) is
deliberately structural so a typed-API caller may supply a conforming,
non-SourceGraphSummary implementation, and both call sites only ever
read .nodes/.edges (protocol members) afterward; a runtime assert
would reject such a caller for no reason. Switched to a type-only
cast(...). (2) _side_source_graph/_l5_payload_empty's
surface_graph fallback did not check whether the pack's own manifest
already recorded an explicit (even NOT_COLLECTED) L5 coverage row.
policy/depth_projection.py deliberately clears build_source.
source_graph to None for a --depth build (or shallower) comparison
while stamping that exact row and leaving surface_graph untouched (an
L2 fact, cleared at a lower floor) -- so the fallback, unguarded, silently
resurrected L5-labeled graph-diff findings and a "source" depth label
for a comparison whose own report said L5 was excluded. Both functions
now only fall back when the pack's manifest records no L5 coverage
row at all (pack.manifest.coverage_for(DataLayer.L5_SOURCE_GRAPH) is
None) -- the one case left needing the fallback is a pack that never
went through any collection/projection pipeline at all (e.g. a bare
typed-API-constructed snapshot). Regression tests added for both: a
depth-projected pack (explicit NOT_COLLECTED row) must not fall back,
while a pack with no coverage row recorded still does.
Fourth review round: two more corrections, plus a real root-cause fix
this time rather than another per-call-site patch. (1) The
coverage-row-aware guard above had only been applied to
_side_source_graph/_l5_payload_empty -- internal_leak.py's
compute_call_graph_leak_paths and cross_source_checks.py's two checks
(_check_private_header_leak, _check_public_to_internal_dependency)
still used the unguarded build_source.source_graph or surface_graph
fallback, so the identical depth-exclusion leak this phase already fixed
once was still open in three more places. (2) A deeper bug in the guard
itself: policy/depth_projection.py's _mark_layers_not_collected only
ever rewrote an L5 coverage row that already existed -- for a pack that
started with no coverage rows tracked at all (a hand-built/
typed-API-constructed BuildSourcePack), a --depth build projection
cleared source_graph but left coverage_for(L5) reading None
regardless, defeating the "no row at all" signal the guard relies on to
tell "genuinely never collected" apart from "collected, then projected
away." Fixed at the root (_mark_layers_not_collected now inserts a
fresh NOT_COLLECTED row for a layer with no existing row, not only
rewriting rows that already exist) rather than special-cased in each
reader again, closing the gap for every current and future caller of that
function at once. This round also consolidated the fallback logic itself:
four independently-maintained near-duplicates had already drifted three
times across four review rounds, so all five readers now go through one
shared evidence_depth.resolve_l5_source_graph(snap, pack) resolver
(_side_source_graph/_l5_payload_empty are now thin wrappers over it).
That consolidation also fixed a third finding from this round: the
SurfaceGraphLike-vs-SourceGraphSummary cast the previous round
introduced for _side_source_graph was actually unsound at that
specific call site (unlike cross_source_checks.py's own narrows) --
diff_source_graph_findings downstream reads concrete-only attributes
(narrowed_scope/extractor_passes/degraded_passes) the
SurfaceGraphLike protocol doesn't declare, so a merely-structural
implementation would reach it and crash. The shared resolver now requires
a genuine SourceGraphSummary for the fallback uniformly (an
isinstance check, not a blind cast), accepting a small, deliberate loss
of typed-API flexibility for cross_source_checks.py's/
internal_leak.py's own narrower needs (protocol members only) in
exchange for one call-site-independent, uniformly-safe contract.
Regression tests added: internal_leak.py/cross_source_checks.py both
now have a depth-exclusion test alongside their existing
"prefers-the-richer-graph" one, _mark_layers_not_collected has a direct
unit test for the missing-row-insertion fix, and
resolve_l5_source_graph has a direct test pinning the non-concrete
rejection.
The in-memory alias-assignment deletion is NOT done, and is being left
open rather than forced through unverified. The one place that builds
the alias — service_header_graph_attach.py's _attach_header_graph —
threads one shared SourceGraphSummary instance into
AbiSnapshot.surface_graph and a synthesized snap.build_source
(BuildSourcePack(root=Path(""), source_graph=graph)) for the
always-on, header-only (L2) graph case, purely so a legacy
build_source.source_graph reader still sees it even when no
--sources/--build-info ran. A real audit for this row (not limited to
the five named readers) found this synthesized pack has further
production readers this checklist never named: coverage reporting
(evidence_report.optional_coverage/layer_presence,
evidence_report.detect_coverage_asymmetry), cli_buildsource.py's own
_layer_payload_empty/build_source_already_satisfies, and
buildsource/embed.py's own --sources/--build-info backfill logic
(existing = snap.build_source; ... existing.source_graph is not None)
— every one of them would silently regress (reporting NOT_COLLECTED
L5 coverage for a plain header-only dump, or skipping the header-only
graph backfill entirely) if _attach_header_graph stopped populating
snap.build_source for this case. None of those call sites were part of
this row's five-reader scope, and auditing and migrating them too is a
materially larger, separately-scoped change than this row's own text
anticipated ("the one piece this row can actually remove" undersold the
blast radius). Per this file's own root-AGENTS.md-inherited
decision-making principles ("if a genuinely general fix isn't feasible
in one pass, say so explicitly and record the gap"), that deletion is
left as an explicitly named, separately-scoped follow-up rather than
performed against an incomplete audit. Until it lands, git grep -n
"surface_graph = graph" inside service_header_graph_attach.py still
finds the alias-construction site outside history — expected, and
tracked here rather than silently left implied-closed.
"Delete the legacy-document aliasing fallback in snapshot_from_dict()"
is no longer this row's to do — that fallback was itself retracted
earlier in this same phase (see the correction above), and a further
review round correctly found the deeper problem this row's first draft
didn't address: migrating the five readers to read only AbiSnapshot.
surface_graph makes historical L3-L5 evidence silently disappear from
them, not merely redundant. surface_graph is deliberately never
populated for a pre-Phase-3 snapshot (the retracted-aliasing fix's whole
point — aliasing the legacy L3-L5-only graph in there breaks
resolve_public_surface()), so a reader that reads only
AbiSnapshot.surface_graph sees nothing for exactly the old snapshots
this row's migration is supposed to leave working. The migration is
therefore not a hard cutover to a single field: each of the five readers
keeps a fallback to build_source.source_graph for a snapshot where
surface_graph is None but build_source is present — graph =
snap.surface_graph or (snap.build_source.source_graph if snap.
build_source else None), read AbiSnapshot.surface_graph first (the
canonical location for a fresh snapshot, including one re-saved through
this phase's own assembly step) and fall back to the legacy nested field
only when it's absent. This acceptance check changes accordingly: "every
reader prefers AbiSnapshot.surface_graph" is what git grep can
confirm mechanically; "no pre-Phase-3 baseline silently loses L3-L5
evidence" is confirmed by a direct regression test loading a real
pre-Phase-3 fixture (surface_graph absent, build_source.source_graph
present) through each migrated reader and asserting its output is
unchanged from before the migration.
- Phase 4: no row, by design, not by omission — AnalysisPlan/
AnalysisPlanner.resolve() is net-new pre-flight validation, not a
second implementation of something this plan is consolidating onto one
representation. The defect it closes is a silent no-op (an unsatisfiable
request dropping mid-run with no diagnostic), not a duplicate
representation with an old copy left over to delete once migration
finishes — there is no prior PlanningError-equivalent code path for
this checklist to retire. Verified instead by Phase 4's own
already-stated acceptance test: the --build-target + pre-captured
aquery gap and the -H + unsupported-collect-mode gap each raise
PlanningError rather than silently dropping the request, confirmed on
the same two scenarios AGENTS.md already documents as today's silent
failures.
- Phase 5: any hand-maintained capability-matrix doc section the
generator now produces.
- Phase 6: each backend parser's own copy of anonymous-marker/closure-
identity/namespace-join logic.
This row is deliberately narrower than this phase's "before" state in
full, and a review round correctly found that gap worth naming rather
than leaving this row read as covering it — this section's own Goal
above says "every phase above is only complete once its 'before' state
is removed," and Phase 6's own text is explicit that the legacy
functions/types/... projection is not removed by this phase, with
no later row in this checklist removing it either. Phase 6's own
text already states why: retiring the legacy fields is deferred "not to
an unscheduled 'eventually,' but to whichever future phase first has a
real SemanticIR-only detector population large enough that the legacy
fields have no remaining reader" — but no phase in this plan is that
phase, so the condition is real and named, not yet satisfied by
anything scheduled here. This is the same shape as Phase 4's own "no
row, by design, not by omission" entry above, stated explicitly for the
same reason: the legacy-field retirement is not a superseded
representation with an old copy sitting idle to delete (every existing
detector still reads it, genuinely, not merely for compatibility), it is
a migration with a real prerequisite — a detector population large
enough to retire the fallback — that this plan's own Phase 6 deliberately
does not attempt, per that phase's own "validating both 'is the IR
correct' and 'does every detector still behave identically once reading
from it' in one unreviewable pass" reasoning. Migrating the checker's
detectors onto SemanticIR and retiring the legacy projection is
therefore real, scheduled, separately-justified future work — named here
as a residual this checklist does not close, not a silent gap in its own
accounting.
- Phase 7: gate.py's raw-exit_code-decode-as-the-only-path for a
fresh report (replaced outright by the structured-RunOutcome-first
read in the same phase, not left running alongside it for new reports —
fold.py itself needs no corresponding row, since it was never the
file doing the raw decode). Not removed, ever,
per Phase 7's own corrected design: junit_report.py's own
_is_failure computation — its answer is a per-render function of each
call's own SeverityConfig/relevant_ids, not a property a finding
carries, so there is nothing for it to be superseded by and deleting it
would remove real, still-needed behavior rather than a second path.
gate.py's
legacy-exit_code-only decode fallback is a second permanent
exception, not a temporary one to be deleted once every front end has
migrated — a review round correctly caught a first draft's "removed
once no such report needs to be read anymore (every front end emits the
structured fields from this phase onward)" for conflating two different
conditions: which front ends currently write is not the same fact as
whether any pre-Phase-7 report still exists to be read, and a
persisted report (a CI artifact, an archived aggregate result) can
outlive every writer that produced it by years. This exception is
therefore kept exactly the way Phase 0's serialization.py
legacy-schema backfill is kept ("not removed, ever... a permanent
reader, the same way every other schema-version branch in that module
is, for as long as ADR-062's v1-v25 import adapter promises to keep
importing that version at all") — the same reasoning applies here
verbatim, and matches ADR-063 D6's own stated promise that this
fallback "keeps every already-published report decodable rather than
orphaned by this decision," which is a permanent commitment, not one
scoped to "until every front end migrates." Not kept as a second
current representation — every front end emits the structured fields
from this phase onward, so the fallback is reached only for a report
that predates it — but never scheduled for removal on any writer-
migration timeline.
- Phase 8: any remaining legacy baseline-set/BundleFacts-only code path
once the ProjectSnapshot import adapter covers it — per ADR-062's own
phasing, not accelerated here.
- Phase 9: reclassify.py's importlib.import_module workaround and its
own now-stale cycle-justification docstring.
Acceptance criteria. For each row: a git grep for the removed
pattern/function name returns nothing outside test fixtures/changelog
history.
That grep must be scoped per row to the specific symbol the row names,
never a blanket pattern — a first draft of this section left that
implicit, and a review round correctly found the gap: two of these
rows explicitly say the old path is not removed, and a loosely-
chosen grep pattern for a sibling row can still match that
intentionally-retained code. Phase 0's row removes the domain-side
clang_*_facts_reliable boolean attributes but explicitly keeps
serialization.py's legacy-schema backfill reading those same wire
keys; Phase 7's row removes gate.py's raw-exit_code-decode-as-
the-only-path but explicitly keeps its legacy-exit_code-only decode
fallback for a pre-Phase-7 report. A grep for a pattern as generic as
facts_reliable or exit_code would match both the removed call site
and the retained one, so each row's acceptance check names the exact
removed symbol (e.g. the domain-side attribute access on AbiSnapshot
itself, not the wire-key string the backfill still reads; the
raw-decode-as-primary-path function, not the fallback branch that
survives it) — not a substring a retained sibling could also contain.
Each of the two rows with a deliberately-retained path additionally
gets its own positive assertion, run alongside the row's negative
grep rather than instead of it: the retained backfill/fallback symbol
(serialization.py's legacy-schema decode path; gate.py's
legacy-exit_code-only fallback) is confirmed still present and still
reachable from a pre-migration-schema input, so this checklist cannot
be satisfied by accidentally deleting the compatibility path those
rows were written to keep.
This checklist is re-run, and re-verified, at the end of the last phase landed in a given release cycle — not deferred to "eventually."
What this plan deliberately does not attempt¶
- No new root CLI command or public API surface. Every new type in
this plan (
Fact[T],EntityId,AnalysisPlan,RunOutcome) is internal until a specific phase's own PR explicitly promotes it, per ADR-063's own "Explicitly not done by this ADR" section. - No
ProjectSnapshot/storage-v2 schema version bump beyond what ADR-062 already plans — this is narrower than "no schema bump at all," and an earlier draft of this section stated it too broadly, contradicting Phase 0's own design a few hundred lines earlier. Phase 8 follows ADR-062's own phase boundaries for theProjectSnapshot/DTO schema specifically; it adds no independent migration there. This plan does, deliberately, bump two schema versions ADR-062 does not own: Phase 0 bumpsserialization.SCHEMA_VERSION(the pre-existingAbiSnapshotschema — the same counter every priorclang_*_facts_reliableflag addition already bumped, v21/v23/etc.), and Phase 7 adds new report-JSON fields alongside the unchangedexit_code. Both are real, intentional, additive migrations to formats that predate and are independent of ADR-062'sProjectSnapshot— not a contradiction of this bullet, which is scoped to the one schema ADR-062 actually owns. - No attempt to resolve every AGENTS.md "Known gaps" entry. Several
entries there are accepted, permanent limitations (e.g. the reverted
linkage-blind-removal attempts, the
type_base_changedevidence gap with no independent signal) that this plan's primitives make easier to close later (Phase 0'sFact[T]onRecordType.bases, specifically) but does not itself close — closing them needs the evidence this plan doesn't add (consumer-side evidence, a captured base-layout fact), which is out of scope here and remains a tracked gap. - No toolchain-identity-probe implementation. AGENTS.md names this gap
independently (castxml/clang invoked without validating the resolved
compiler matches the real build); Phase 6's
SemanticNormalizermakes a future probe's result easier to thread through uniformly, but does not implement the probe itself.
Effort and risk summary¶
| Phase | Effort | Primary risk |
|---|---|---|
| 0 | M | Converting the wrong three fields first (pick fields with an active fabricated-finding incident, not merely "many None checks") |
| 1 | L | castxml unavailability blocking the parity-test half; mitigated by explicit clang-only first landing |
| 2 | L | Identity collision regressions are exactly the bug class this phase targets — the property-test suite is the real acceptance bar, not code review alone |
| 3 | XL | Two distinct risks, not one: migrating surface.py/export_surface.py's traversal without changing what counts as public (the kept-test-behavior acceptance bar), and relocating buildsource/graph_facts.py's GraphNode/GraphEdge/merge_graph_facts into model/ without disturbing the existing L5 source-graph suite — a wider blast radius than the phase's first draft assumed, which is exactly why review caught the parallel-graph-hierarchy defect in that draft before it shipped |
| 4 | M | Planner rejecting a request current behavior silently accepted — must ship with a migration note in CHANGELOG.md/docs, not only a changelog fragment, since it is a user-visible behavior change (a previously-silent no-op becomes an error) |
| 5 | L (mechanical, field-by-field) | Scope creep — cap each commit to one field |
| 6 | XL | Largest blast radius in this plan (every backend parser); sequence last among the "hard" phases, after 0/2/3 give it primitives to build on |
| 7 | M | Two independent exit-code consumers, not one: JUnit output is consumed by external CI systems (_is_failure stays per-finding, parity testing not redesign), and the multi-target aggregate path (workflows/aggregate/gate.py/fold.py) was missed entirely in this phase's first draft — its report-schema addition needs the legacy-decode fallback proven bit-for-bit equivalent on every existing fixture, not just new ones |
| 8 | XL | Shared with ADR-062's own Phase 1 risk profile; do not duplicate that ADR's own risk analysis here, defer to it |
| 9 | S | Low technical risk (extracting already-shared logic into a leaf module), but skippable-looking — it has no dependency on any other phase, which makes it easy to defer indefinitely rather than land; don't let "independent" read as "optional" |
| 10 | S per row, continuous | The easiest phase to skip under time pressure — explicitly called out as required, not optional, per ADR-063's decision drivers |