ADR-062: Project Snapshot Storage v2 — Content-Addressed Sections, Explicit Fact Availability, and Occurrence-Preserving Identity¶

Date: 2026-08-26 Status: Proposed — partially implemented; amended 2026-11-03 (D19 accepted and implemented: an optional new fact is a declared extension fact, not a global SCHEMA_VERSION bump); amended 2026-10-30 (D14-D18, rules accepted; D14, D15 and D17 implemented, D16 partially (evidence and transport), D18 steps 1-2 (per-section decode split; lazy declarations/types/debug layout via a deferred semantic_ir cell) implemented 2026-10-31, per-pair release and the shared decoded-section cache pending -- see known-gaps). Phase 0 primitives implemented (abicheck/storage/: fact availability, entity/occurrence identity with conflict preservation, canonical encoding and semantic digest, and the separated version axes). Phase 1's A1.1-A1.3 are implemented: abicheck/storage/package.py carries the ProjectSnapshot manifest/ref/object-store object model (PackageManifest, VariantRef, ArtifactRef, ObjectRef, the ObjectStore protocol, and the D6 path-layout functions); abicheck/project_snapshot_store.py is a real, directory-backed ObjectStore implementation (DirectoryObjectStore) plus a manifest/ref writer and reader over ADR-059's physical envelope — everything in D6's layout except the .tar.zst transport form, which remains open; abicheck/storage/import_v1.py's import_legacy_snapshot/export_legacy_snapshot are the v1-v25 import adapter and its exact inverse, expressing a single-library legacy document as a one-artifact, one-variant project that round-trips through the store at the semantic-digest level. Landed jointly with ADR-063 Phase 8, whose own D8 constraint (abicheck/storage/dto.py's SectionDTO — a distinct, versioned, explicitly-encoded class per section, never asdict) this now satisfies for every legacy document field, not just semantic_ir/semantic_ir_conflicts: abicheck/storage/legacy_sections.py's split_legacy_document/ join_legacy_document partition the rest across D8's binary/ declarations/types/layout/debug/build/graph/provenance sections, each independently versioned and explicitly allowlisted per field (an unassigned field is a hard import-time error, never a silent catch-all). What remains unsplit is each section's own internal shape — elf/dwarf/ build_source/... still carry the existing JSON encoding inside their own section, not a further per-field typed decode. A1.4 is implemented too (ADR-063 Track C 8B): abicheck/storage/import_bundle_facts.py/ import_baseline_set.py fold a persisted G38 BundleFacts document and an actions/baseline set respectively onto this same sectioned representation, calling import_legacy_snapshot once per library and attaching each container's own composition facts to a new VariantRef.sections field. Of the 2026-10-30 amendment's data-evolution rules (plan item A1.9), D17 is implemented (package format v2: immutable refs/<kind>/<id>/<ref-digest>.json documents named by {id, ref_digest} in manifest.json, an atomic manifest commit, serialized writers — abicheck/storage/package_publish.py), D14 (storage/wire_extensions.py: preserved unknown sections, header and record extensions, must_understand on section headers, the document and the manifest; a blocked section resolves its readers to not comparable through compare/section_readers.py), D15 (a field inventory generated by tests/_wire_inventory.py and diffed in CI, plus the section_availability header table), D16 partially (the evidence digest, persisted as ArtifactRef.evidence_digest and read by the same-snapshot coverage note, and the transport digest of each package-archive member; comparison_input is not), and D18 is not implemented; see the plan's per-item status.

This is now wired into dump/compare/scan as the default, single-file shape (Phase 8 redesign). The directory-backed package this ADR originally specified (manifest.json/refs//objects/sha256/...) turned out to be pure storage-UX cost for the single-artifact case every dump performs today — many small files instead of one, awkward to scp/commit/upload as a CI artifact — for none of its real payoff (content dedup, independent per-section objects), which only materializes once a project shares content across multiple artifacts. storage.sectioned_document packages the identical D8 section split as one JSON document instead; serialization.snapshot_to_json/write_snapshot write it and snapshot_from_dict/load_snapshot read it by default, with no CLI flag required. An older flat .abi.json a prior build wrote stays fully readable. The directory writer/reader (abicheck/project_snapshot_legacy.py's write_legacy_snapshot_package/ read_legacy_snapshot_document) remain available as typed-API primitives — compare/scan --against still accept a directory package as an input path — but no dump CLI flag produces one today. A1.4's document-folding slice is implemented (ADR-063 Track C 8B): abicheck/storage/import_bundle_facts.py/import_baseline_set.py fold an already-persisted G38 BundleFacts document and an actions/baseline set respectively into this sectioned representation, calling import_legacy_snapshot once per library and attaching each container's own composition facts to VariantRef.sections (a new field this track added). A1.4/A1.5's live-object slice is implemented too, and now reconciled onto the same physical layout (Track 1): abicheck/bundle_facts_store.py's write_bundle_facts_package/read_bundle_facts_package are the producer/reader of a multi-ArtifactRef PackageManifest from a live BundleFacts object — one BundleFacts becomes N ArtifactRefs under one shared VariantRef — and are now a thin wrapper over bundle_facts_serialization.bundle_facts_to_dict/bundle_facts_from_dict plus storage.import_bundle_facts.import_bundle_facts/export_bundle_facts, so a package written from a live object and one written from a persisted document share the identical VariantRef.sections["bundle_composition"] layout. PackageManifest.project_sections (abicheck/storage/package.py) remains D7's general cross-artifact-evidence slot, but this path no longer populates it. abicheck/project_snapshot_store.py's write_project_manifest/ read_project_manifest publish and read both VariantRef.sections and project_sections back through the real D6 directory layout.

Former known gap, now closed (Track 1): these two tracks briefly solved the identical plan item (A1.4, "fold BundleFacts into the sectioned representation") independently and landed with two different, non-interoperable physical layouts for the same composition facts — bundle_facts_store.py originally split variant_fingerprint/ filesystem_aliases/library_filenames onto VariantRef.captured/ ArtifactRef.native_identity and stored only the instantiation manifest in project_sections, while import_bundle_facts.py bundles all four facts into one VariantRef.sections["bundle_composition"] object — a package written by one could not be read by the other's reader. Since storage/import_bundle_facts.py already established "storage takes an already-serialized document" as this area's governing contract (matching import_v1.py), the fix rebuilt bundle_facts_store.py's live-object writer/reader as a thin wrapper over bundle_facts_serialization .bundle_facts_to_dict/bundle_facts_from_dict plus storage .import_bundle_facts.import_bundle_facts/export_bundle_facts, retiring the separate project_sections/native_identity-for-filename/aliases layout in favor of the single VariantRef.sections one — including rewriting bundle_facts_store.py's own test suite (much of it previously pinned to the internal layout being retired) with the same adversarial-review rigor the original had.

A1.7 (stored/live and stored/stored release-comparison CLI reachability) is also implemented: abicheck compare (directory/package release fan-out, cli_compare_release.py) now accepts a multi-artifact ProjectSnapshot package directory as either operand, unpacked via workflows.release_package.resolve_release_package_map into per-library single-artifact sub-packages the existing single-artifact resolution path already reads unchanged; --old-variant/--new-variant select among multiple declared variants. Directory packages only — the .tar.zst transport form (A1.1's own remainder, below) is not yet produced by anything, so nothing yet exercises this against one.

A1.6 (bundle_variants: capture wiring) has landed: the .abicheck.yml bundle_variants: block (model/bundle_variants.py, strictly validated on every config load) drives abicheck project capture-variants, which captures every declared variant into one package (workflows/bundle_variants_capture.py) — one VariantRef per variant with D9's two maps kept apart (declared from config, captured from what the dumps recorded), a required variant that cannot be captured refused before anything is written, an optional one skipped and reported with no VariantRef. Two stored packages compared through compare now report comparison_scope.variant_pairing (by variant_id; declared-coordinate changes as a variant-boundary change distinct from captured drift; an unmatched variant never read as removed). See the plan's A1.6 entry.

Digest-deduplicated shared evidence beyond what the slices above already give and the .tar.zst transport form (the remainder of A1.5/A1.1) remain not implemented, along with BuildSourcePack/project source-graph dedup (the ~57-59 MB-per-artifact finding this ADR's Context names) (non-ELF artifact membership, A1.8, has since landed — see the plan's A1.8 entry) — note this is the remainder of Phase 1, not Phase 2 (Phase 2 is the separate scale/performance work: lazy loading, streaming encode, cache migration, indexes). A full per-item design for A1.1's .tar.zst remainder and A1.5/A1.6/A1.8 (Goal/Design/Files/Tests/Acceptance criteria each) exists in docs/contribute/plans/storage-format-v2.md's "Phases" → "Phase 1" section (A1.7's own entry there is now a landing note, not only a design); the object model these items build on (PackageManifest.artifact_refs/ variant_refs, ArtifactRef.kind/.sections) already supports every one of them without a schema change, which is why each item below is scoped as a producer/CLI-wiring gap rather than a storage-format gap. All of Phase 2 is not implemented. Decision maker: abicheck maintainers Supersedes (partially): ADR-015's single-document logical model. ADR-059's physical envelope is kept unchanged and explicitly not superseded.

Context¶

The snapshot format has grown from a single-library JSON dump into the persistence layer for a whole product line. Four distinct persistence shapes now coexist:

per-library .abi.json[.zst] snapshots (AbiSnapshot, schema v25);
baseline sets (manifest.json plus per-library snapshots, sometimes with staged binaries);
BundleFacts, which embeds every per-library snapshot in one JSON document (schema version tracked by BUNDLE_FACTS_SCHEMA_VERSION in abicheck/bundle_facts.py, bumped by CLI cleanup phase two's artifact_type self-describing-marker addition; still one undivided document regardless of the exact version);
BuildSourcePack, which may be embedded inside each snapshot or stored out of band.

Each is individually reasonable. Together they give more than one answer to "what is the baseline for this project", "where does release membership live", "which file owns shared build/source evidence", and "what is the integrity root of a release".

The physical layer is in good shape. ADR-059's envelope detects compression by magic bytes, rejects suffix conflicts, writes deterministically and atomically, and enforces stored-size, decoded-size, and zstd-window limits. Real measurements on oneDAL show it working: a ~149 MB daal snapshot and a ~115 MB oneapi::dal snapshot compress from roughly 264 MB to about 7.7 MB. Most of that volume is genuine L5 graph evidence, not accidental bloat.

So the problem is not the codec, and the fix is not to store less evidence. The problems are in the logical and container model:

Whole-document construction and loading. snapshot_to_dict calls dataclasses.asdict(), which deep-copies the entire object graph before JSON encoding; the read path decompresses, decodes, and parses one complete document before reconstructing every dataclass. Several full representations can be live at once, so a 150 MB decoded snapshot costs far more than its 5 MB on disk. BundleSignatureEvidence was introduced precisely because release comparison could not afford to retain full snapshots; that is a symptom, not a general solution. One decoded BundleFacts document for a large release can also approach the reader's 1 GiB decoded-size ceiling even though every library in it is individually valid.
Identity loss. AbiSnapshot.index() is first-wins for functions, variables, and types: duplicates are warned about and then dropped from the lookup maps. typedefs is keyed by bare name (v25 added typedefs_qualified alongside it, but the lossy map remains). ElfMetadata.symbol_map maps a bare symbol name to exactly one ElfSymbol, although ELF legitimately carries several versions, bindings, and default/non-default definitions of one name. Base offsets are a base name -> offset mapping, which cannot express repeated or path-distinguished base subobjects. The multi-TU merge already documents further cases it cannot resolve (MSVC decorated names that encode return type, uninstantiated template methods with no mangled name, plain-C statics with no TU scoping).
One overloaded version integer. SCHEMA_VERSION is at 25 and encodes at least four independent facts: JSON field layout, producer implementation epoch, per-fact reliability, and comparison-critical contract compatibility. Bumps v9, v19-v23, and v25 exist because a producer began emitting a correct value where it previously emitted a real-but-wrong default — which the loader reconstructs into snapshot-level *_facts_reliable flags. A normalization or resolver change can alter meaning without changing any field; adding an optional display field should not imply a new evidence recipe.
Missing conflated with false. A plain bool, [], {}, or None cannot distinguish "positively established" from "producer never ran", "producer does not support this", "extraction failed", and "not applicable". The *_facts_reliable flags are correct compatibility patches for the specific historical cases they name, but they do not scale to a general evidence model, and they say nothing when one side was captured at L2 and the other at L4.
Incompletely specified canonical form. snapshot_to_json() does not globally sort keys, entity list order can follow producer traversal order, and the stable snapshot hash sorts dictionary keys but treats list order as significant. BundleFacts serialization cannot use recursive key sorting at all, because one template-instantiation mapping uses insertion order to carry template-argument order — structural order and incidental map order are not separated.
Duplicated shared evidence. A BuildSourcePack embedded per library, inside a BundleFacts that embeds every library, repeats project-wide source graphs, compile databases, and toolchain data once per artifact. The oneDAL graph section alone is roughly 57-59 MB decoded per snapshot.
Multibuild is modelled but not captured. bundle_multibuild.py (since deleted as unreachable; compare/variant_pairing.py is the pairing that runs) gets the semantics right — variants are paired, never unioned, and a same-side fingerprint collision is an error — but the ordinary capture pipeline still writes the default variant fingerprint, bundle_variants: is not wired into .abicheck.yml discovery, and the stored/live release comparison is not reachable from the standard CLI. A coherent cpu/gcc + cpu/clang + dpc/icx + aarch64/gcc baseline cannot be produced by a normal workflow today.
Derived graphs are rebuilt with today's semantics. BundleFacts correctly stores authoritative per-library ELF metadata and rebuilds the resolution graph rather than persisting a second copy. But provider selection, alias normalization, symbol-version handling, and reachability have each been corrected over time, so the same stored baseline can yield a different derived graph under a later abicheck — with nothing recorded about which semantics produced the original answer.

BuildSourcePack already demonstrates the shape the rest of the project should adopt: a manifest, normalized facts, optional raw output, and a content hash computed over normalized content with volatile runtime data excluded. This ADR elevates that approach from one embedded subsection to the storage model for the whole project.

Decision¶

D1 — Keep the physical envelope; replace the logical model¶

ADR-059's compression detection, atomic write path, deterministic gzip/zstd output, and decompression-bomb limits are retained unchanged and remain the only place compression is implemented. v2 changes what is stored and how it is addressed, not how bytes are framed. No new binary codec is introduced, and no evidence layer is dropped to save space: JSON compresses this content extremely well, and the remaining cost is construction, retention, and duplication rather than disk.

D2 — Separate the version axes¶

One integer stops carrying four meanings. A v2 package records, as independent fields:

Axis	Meaning
`package_format_version`	container/manifest layout
`section_schema_version`	per-section field layout (per section kind)
`normalization_recipe`	how spellings/paths/identities were normalized
`producer` (`name`, `version`, `binary_digest`)	what emitted the facts
`extractor_generation`	extraction semantics epoch
`resolver_generation`	derived-graph/resolution semantics epoch
`comparison_contract_version`	what a reader must understand to compare safely

Exactly two axes fail closed, for two different reasons: a newer package_format_version means the reader may not be able to locate the package's structures at all, and a newer comparison_contract_version means comparing without understanding the change could produce a wrong verdict. Both also fail closed when the package does not state the axis validly — absent, malformed, non-integral or non-positive alike. An axis that exists to refuse unknown semantics cannot treat "unknown" as agreement, and that reasoning is symmetric: a package that never says what layout it has is no more parseable than one whose layout is too new. Neither axis is validated only by the deserializer; a reader must re-check both where it decides, because the version object is constructible directly.

The remaining five are informational to a reader that does not recognize them, which is what lets an optional display field ship without implying a new evidence recipe.

(Three drafts of this paragraph were wrong in the same direction and each was caught by review. The first said only comparison_contract_version fails closed at all; the second granted that both do but kept the absent-value rule for the contract axis alone, so a consumer implementing the ADR could synthesize a current format version for a package whose layout is unknown; and the implementation itself had to be corrected twice before it matched — a negative version, then a fractional one, each slipping past the guard written for the previous case. Recorded rather than tidied away, because the repeated direction is the finding: it is always the container-layout axis that gets quietly weakened to match the narrower rule, and weakening it lets a reader silently misparse a package instead of refusing it.)

Capability is stated explicitly rather than derived from schema history, and an imported legacy snapshot preserves its source_schema_version and source_producer_generation so migrations and audits stay honest instead of accumulating one special case per newly discovered historical producer defect.

D3 — Fact availability is explicit, never inferred from a default value¶

Every comparison-relevant fact family carries a FactAvailability record: status (present / partial / not_collected / unsupported / failed / not_applicable), producer, producer version, recipe, scope, confidence, and diagnostics. Availability is declared once per fact family, overridden per entity only where an entity genuinely differs, and summarized in a coverage ledger.

An empty vtable list then means exactly "the producer ran and established there are no virtual entries", distinct from unsupported. A comparison that requires a family whose availability is not present/partial resolves through policy to NOT_COMPARABLE or an explicitly reduced-confidence result. It never infers safety from an empty collection. This generalizes the *_facts_reliable flags, which remain as the import-time expression of the same fact for legacy snapshots.

D4 — Preserve every occurrence; never resolve identity by dropping data¶

Two identities are modelled separately:

EntityId — the logical declaration/symbol/type believed to be the same across occurrences and releases.
OccurrenceId — one specific observation: an AST declaration, a TU occurrence, a binary symbol version, a DWARF DIE, a PDB record, a source location.

Every occurrence is stored. Resolution may group occurrences under one entity, but must never destroy multiplicity. Where occurrences cannot be reconciled confidently, both are retained and an IdentityConflict is recorded — replacing today's warn-and-keep-the-first behavior.

Concretely, an ELF symbol occurrence is keyed by artifact, name, version, default-ness, binding, type, visibility, and definition/import status rather than by bare name; a base subobject becomes a list entry carrying base type, inheritance path, offset, virtuality, access, producer, and availability rather than a name -> offset mapping.

D5 — One canonical logical encoding¶

every unordered collection has an explicit, stable sort key;
every semantically ordered collection is an array, never a map whose insertion order carries meaning (template arguments become [{"parameter": …, "value": …}, …]);
a map is used only when its keys are unique and its order is meaningless;
float normalization is specified: -0.0 and 0.0 agree, an integral float and the same integer agree, and a non-finite value is refused rather than emitted as a bare NaN/Infinity literal no conforming JSON parser accepts;
volatile capture metadata (timestamps, hostnames, wall-clock durations, absolute scratch paths) lives outside the semantic-hash domain;
semantic digests are computed from the normalized logical object, independently of pretty-printing, key order, or compression.

Randomizing producer traversal, dictionary insertion, TU completion, and parallel extraction order must not change any semantic digest or any comparison result.

Path normalization is deliberately not specified here, and is Phase 1 work. An earlier draft of this list claimed it alongside float normalization; nothing defined or implemented it, so a producer had a requirement it could not satisfy consistently and two equivalent captures could hash differently on path spelling or platform (Codex review). Stating a rule nobody can implement is worse than stating none: it reads as settled.

It is not a line item because it is not a small one — separator direction, case folding on case-insensitive filesystems, absolute versus source-root- relative form, symlink resolution, and ~-redaction (which ADR-032 D7 already applies to CompileUnit.output, for persistence rather than for hashing) each change what two captures of one build agree on, and getting any of them wrong silently merges or splits content. It belongs with the normalization_recipe axis D2 already reserves, so that a package states which rule produced its digests rather than every reader assuming the current one. Until then, a path reaching the hash domain is hashed as written.

D6 — One `ProjectSnapshot` package¶

Baseline sets, bundle facts, and per-library snapshots converge on a single content-addressed package. A one-library scan is a package with one artifact; bundle facts become one section rather than a parallel top-level format.

project.abicheck/
  manifest.json            # small; loads immediately
  refs/variants/<variant-id>.json
  refs/artifacts/<artifact-id>.json
  objects/sha256/<aa>/<digest>.json.zst
  indexes/index.sqlite     # optional, rebuildable, never canonical truth

The transport form is a deterministic .tar.zst of that directory. The SQLite index accelerates queries and is regenerable from manifests and objects, so baseline durability never depends on a database migration.

Artifact records carry native binary identity explicitly (content SHA-256, ELF build ID / Mach-O UUID / PE-PDB identity) — the existing top-level build_id means an opaque CI identifier and is not reused for this. Membership is stated for every artifact kind, including PE, Mach-O, Python-visible, and header-only targets, with bundle-level resolution declared as an ELF-only capability rather than silently excluding non-ELF entries.

D7 — Shared evidence is stored once and referenced by digest¶

Project source graphs, variant build evidence, toolchain profiles, compile database slices, common header surfaces, and raw extractor artifacts are content-addressed objects. An artifact references the shared object plus its own overlay instead of embedding a private copy.

D8 — Sections are independently addressable and lazily loaded¶

Snapshot content is split into binary, declarations, types, layout, debug, build, source_abi, graph, provenance, diagnostics, and raw_refs sections, chunked to a bounded decoded size (target 1-8 MB, tuned against real measurements). A project comparison loads two manifests, all small L0 binary sections, then one matched library pair at a time, releasing each pair's sections before the next. Peak memory must scale with the largest active section or library pair, not with the total decoded size of the release.

Implementation note (2026-09-24, evidence-entity-model Phase 5a; the first landed slice of storage-format-v2 Phase 2 A2.1). The graph section of a single-file sectioned snapshot is now loaded lazily: snapshot_from_dict leaves the section's DTO validation, migration and graph construction for the first read of AbiSnapshot.surface_graph (or of the build_source.source_graph alias, which shares one pending cell; model/lazy_graph.py). A decode failure raises at that access, with the same error an eager load raised, and never reads as an empty graph. The rest of D8 (chunking, per-pair release of sections) is not implemented.

Decision: the L5 graph diff is not gated. A default stored-vs-stored compare of two header-graph snapshots still reads the graph: the L5 diff_source_graph_findings diff, the private_header_leak and public_to_internal_dependency cross-source checks, the assurance _graph_completeness pass and the snapshot content digest all run on every such compare. Gating the L5 diff to runs that ask for it would remove its RISK-tier findings from default output, which changes findings; nothing in the Phase 5 measurements justifies that (the verdict was identical in every variant, but the findings are part of what a default run reports), so lazy loading ships without it. Lazy loading therefore saves the decode only on runs that never read the graph (a --depth binary/debug compare, which strips the header graph before anything reads it, and other snapshot-metadata-only readers); the compare-side cost is addressed by the compact graph encoding instead (Phase 5b).

D9 — Variant identity is created at capture time¶

The capture pipeline is told which variant it is producing. A package stores both declared and captured-and-verified variant coordinates. Stable variant identity (target, compiler family, feature toggles) is kept separate from build state that may legitimately change between releases (compiler version, standard, flags, artifact membership) — so adding or removing a library is a change inside a matched variant, not a new variant. Variants are never unioned; old-only and new-only variants stay explicit outcomes.

D10 — Evidence profiles define completeness¶

"Store all relevant information" means: for the selected profile, every required fact family is either present or carries a machine-readable reason why it is not.

Profile	Required
`binary`	artifact identity, exports/imports, versions, dependencies, loader facts
`abi`	`binary` plus debug/layout and signature evidence where available
`api`	`abi` plus public-header declarations and the extraction contract
`source_abi`	`api` plus L3/L4 build and source facts
`deep_graph`	`source_abi` plus L5 graph and reachability evidence
`reproducible`	the selected profile plus raw evidence references

D11 — Caches key on content, not on mtime¶

Cache identity is binary content digest + input tree content digest + extraction contract + producer/extractor generation + normalization recipe + requested profile. Size and mtime may be used to avoid recomputing a digest, but never to define semantic identity, so touching an unchanged header does not discard an expensive extraction. Eviction uses a byte quota with LRU rather than an entry count, and caching is per section so an unchanged binary, a shared graph, or an unchanged variant can be reused.

Implementation note (2026-10-31). The identity half is implemented for the whole-snapshot and header-AST caches: content digests only (extract/cache_input_digest.py), over every file under an include root, file-valued includes, the compiler's built-in search directories and CPATH-style directories; a stat signature only lets a process skip re-reading an unchanged file. Byte-quota LRU is implemented (storage/byte_quota_lru.py). Per-section caching is not (storage plan A2.3). D11 itself is still Proposed: accepting it is a maintainer decision.

Implementation note (2026-11-01): dependency manifests. Hashing the content of every file under the compiler's built-in search directories (~4,400 files) cost 30-60 ms per key derivation, about three derivations per dump and header-graph attach, and failed the header-graph perf gate. System include directories now enter every key by spelling and order only (plus the compiler's content identity, already folded). The system headers a parse actually read are recorded at store time in a dependency manifest beside the entry -- ccache's "direct mode" -- owned by extract/cache_dep_manifest.py: the file list comes from the frontend's own depfile (-MD -MF; both clang and castxml honour it), each file with its content digest. A key hit is served only while every recorded file still digests the same and no file has appeared that would shadow one from an earlier search directory; a missing or corrupt manifest, or a dependency modified after the parse started, is a miss / no store. The clang AST entry (<key>.json.deps), the castxml parse-store entry and its clang-derived vector/variable-template sidecars, and the whole-snapshot entry (<key>.deps, the union of the manifests of every parse the dump ran or was served, gathered by cache_dep_manifest.recording()) all use it. A frontend run that writes no depfile (a DPC++ multi-pass driver run) records the old full-content digest of each system directory instead. Project inputs (headers, include roots, file-valued includes, CPATH-style directories) stay content-hashed in the key.

D12 — Compression level is internal policy, not a new user knob¶

Local/extraction cache uses zstd level 3, ordinary CI baselines 10-15, archival release publication 19; already-compressed or raw objects are not recompressed. After structural deduplication the marginal value of maximum compression falls further, so this stays a policy table rather than another tuning surface.

D13 — Migration is adapter-based and non-destructive¶

Every v1-v25 snapshot stays readable through an import adapter into the v2 in-memory model. No existing baseline is rewritten in place, and no user is required to regenerate a baseline to keep comparing. Writers adopt v2 behind an explicit opt-in until the validation corpus in the plan passes.

Consequences¶

Positive. Peak memory decouples from release size. Duplicate and ambiguous entities stop disappearing silently. "Not collected" stops reading as "not present". Determinism becomes checkable rather than assumed. One package answers project, release, variant, and integrity questions. Shared graphs are stored once. Historical baselines can be compared under, or explicitly refused by, the semantics that produced them.

Negative / cost. A second storage model exists during migration, and the import adapter is real, permanent surface. Content-addressed layout is harder to inspect by hand than one JSON file (mitigated by the transport .tar.zst and the rebuildable index). Occurrence preservation makes some sections larger before deduplication makes the package smaller. Explicit availability will surface genuine NOT_COMPARABLE results that today pass silently — that is the point, but it is a visible behavior change and is therefore gated behind D13's opt-in.

What this ADR does not decide¶

The wire encoding inside an object (JSON is retained; a columnar or binary encoding is not proposed and is not expected to be the bottleneck).
Concrete chunk-size and zstd-level constants, which are set from the plan's measurement corpus rather than chosen here.
Whether the SQLite index ever becomes mandatory (it does not today).
Bundle-level resolution semantics for PE and Mach-O — D6 requires membership and capability declaration, not a new resolver.
The ChangeKind or report-schema consequences of newly explicit NOT_COMPARABLE outcomes; those are ADR-050's and ADR-042's surfaces.

References¶

ADR-015 — original snapshot format (historical)
ADR-023 — bundle layer
ADR-050 — comparability contract
ADR-059 — physical storage envelope (kept)
ADR-061 — the storage layer this package occupies
G38 — bundle facts and multibuild pairing
Storage format v2 plan — phasing, acceptance criteria, validation corpus

Amendment (2026-10-21): the source tier has its own generation axis¶

D2's extractor axis covers abicheck/extract/** and the root parse-stage modules. The L4 source extractors and the L5 graph producers under abicheck/buildsource/ were in neither generation axis, so a change to their semantics moved no stated generation and no stored package reported drift. Folding them into EXTRACTOR_GENERATION was rejected: it would drift every L2 (binary-plus-headers) package on an L4/L5 edit, which is false drift.

The source tier is therefore a third axis, with the same mechanism as the other two:

Constant. SOURCE_GENERATION in storage/versioning.py is this build's source-tier semantics epoch.
Scope. abicheck/buildsource/source_extractors/ (every module, derived from the tree) plus the explicit SOURCE_TIER_FILES in storage/generation_scope.py: the L4 replay and link producers and the L5 graph producers and folds. Comparison and finding modules (source_diff, source_graph_compare, source_graph_findings, cross_source_checks*) are out, because they decide verdicts from recorded facts rather than deciding what is recorded. source_extractors/resolver.py decides provider selection, so it stays on the resolver axis. The three scopes are disjoint, which a test enforces.
Stamping. StorageVersions.source_generation is stated only by a package that carries source-tier (L4/L5) facts. this_build_generations takes carries_source_facts for this. A package without them omits the key and stays unstated, so its serialized bytes do not change and it never drifts. Packages written before this amendment state no source generation.
Drift. A stated source generation that differs from this build's is reported as semantics_differ, the same as the extractor and resolver axes. It is never a refusal: the source tier is a derived-answer caveat under ADR-028 D3. FactGenerations widens from a pair to (extractor, resolver, source).
Record. storage/generation_fingerprints.json gains a source entry, recorded with scripts/record_generation_fingerprints.py --axis source. The extractor and resolver entries are unchanged.

The extractor axis's own scope and generation do not change, so this amendment moves no existing stored package's generation. The residual limit is the same as for every axis: a fingerprint cannot tell a refactor from a semantic change, so --no-semantic-change --reason is a reviewed claim.

Amendment (2026-10-30): data-evolution rules, digest roles, and immutable publication¶

Status of this amendment: the rules D14-D18 are accepted; D14, D15 and D17 are implemented, D16 partially, D18 only its decode split. Their implementation is tracked as plan item A1.9 in the storage format v2 plan. The rest of this ADR stays Proposed.

Context. An external review of main reported the following. Its figures were measured outside this repository and are recorded as reported, not reproduced. A1.9 turns each probe into a named regression test.

JSON + zstd stays (D1). A 3000-function library dump compresses from 25.4 MB to 107 KB. Splitting declarations into fixed blocks grew their compressed size by 63%.
A full load takes about 108 MiB RSS. The cost now lies in evolution and loading, not in encoding.

Probe	Reported	Where	Rule
unknown section kind	the whole document is refused	`storage/sectioned_document.py` (single-file sectioned document)	D14
unknown field in a function record	accepted, then dropped on re-save	DTO decode/encode	D14
only `created_at` differs	digest changes	`snapshot_content_digest` (`storage/snapshot_encode.py`)	D16: by design
functions reordered, comparison `NO_CHANGE`	digest changes	`snapshot_content_digest`	D16
republish whose object check fails	`manifest.json` unchanged, previous package unreadable	`write_project_manifest`	D17

The last row is a failure mode beyond the deferred gap that write_project_manifest's docstring records. That gap covers concurrent readers and interruption. This one is a plain refused publish. Refs are written before the object check raises, so the old manifest ends up naming overwritten refs.

D14 — Optional and must-understand extensions¶

Carrier. Each section header (SectionDTO) and each package manifest gains a must_understand list. It names the extensions, sections or fields, that a reader must recognize before using that section for comparison. Anything not listed is optional.

Optional data is preserved. Each DTO record and each section header gains an extensions map from names to canonical JSON values (D5). It is never raw bytes and never a domain-object field (ADR-063 D8). A reader keeps an unknown optional field or section there and writes it back unchanged on re-save. Dropping it is a defect. extensions is inside the evidence and publication digests (D16).

Unknown must-understand data blocks only its dependents. An unknown name in must_understand blocks only the analyses that read that section. Those analyses resolve to NOT_COMPARABLE, with a diagnostic that names the extension. Unrelated sections are still compared. An unknown section kind not listed in must_understand is kept in extensions instead of refusing the document. This replaces the current blanket refusal in sectioned_document.py.

A newer section_schema_version of a known kind. The reader treats the section as must-understand and blocks its dependents. It does not refuse the whole document. This brings the code into line with D2, which calls the axis informational at the package level.

Which mechanism to use:

comparison_contract_version (D2, fail-closed for the whole package) is bumped only when a change can alter the verdict of analyses that do not read the new data. Examples are a change to matching or identity rules.
Any other semantic addition uses must_understand.
Display or provenance additions use neither.

Implementation decisions (2026-10-30).

Scope of a name. A section header's must_understand names extensions or section kinds; an unknown one blocks that section. A document- or manifest-level must_understand applies to the whole document or package, so an unknown name there refuses it (no section to narrow it to). The provenance section holds the identity every analysis reads, so blocking it refuses the document too.
Which analyses. compare/section_readers.py declares the stored sections each detector (and the surface-metric roll-up) reads, checked against an AST scan of the detector's code. A blocked detector is not_evaluated with a coverage_gap starting not comparable: and naming the section and extension; analysis_assurance reads partial. No exit code changes: a whole-run not_comparable (exit 16) would also discard the analyses that did compare.
Directory packages (2026-10-31, revised). A section object read from a directory package with an unknown must-understand name blocks that section's readers and is kept verbatim for a re-save, exactly as in a single-file document; only the provenance section refuses. (It was refused outright before.)
Newer section versions in directory packages (2026-11-02). A section object stored at a section_schema_version newer than the reader's, and stated so in the manifest's section_schema_versions, blocks only that section's readers with the same cause text as a single-file document and is kept verbatim for a re-save; the manifest-level version check no longer refuses the package for it. A newer provenance still refuses.
Directory packages carry the carriers (2026-10-31, revised). write_legacy_snapshot_package, given the AbiSnapshot, stores its carriers (WireExtensions.to_wire: kept sections and their versions, header extras, record extras, non-present availability entries and the document-level must-understand list) as one content-addressed object named by the new optional ArtifactRef.carriers. Every section the carriers hold undecoded (failed, not collected, blocked, kept verbatim) is left out of ArtifactRef.sections and of the manifest's section_schema_versions, so no default-filled payload is stored as evidence. On read, wire_extensions.carriers_for_reader re-decides what blocks for the reading build (what blocks depends on the reader, so unreadable is not stored), including a kept section of a kind the reader knows but the writer did not decode. The evidence digest stored on the ref hashes the carriers in, so it equals the digest of the snapshot the package was written from. Versioning: a ref that sets carriers is written at ref_schema_version 2; every other ref stays at 1 with unchanged bytes and ref_digest. A reader of version 1 refuses a version-2 ref through D17's existing fail-closed check, which is required: it would otherwise read the left-out sections as empty. No package_format_version bump: the layout is unchanged, and the per-ref axis already fails closed (D2: the narrowest axis that makes an older reader refuse). A ref stating version 1 while carrying carriers is refused as corrupt. A flat mapping handed to write_legacy_snapshot_package cannot say what it lost and is written as what it says (known gap).
Bundles carry the carriers (2026-10-31). A bundle package stores each member's carriers on its ArtifactRef.carriers, exactly as a directory package. A G38 bundle document or G40 bundle archive has no ref to hang them on, so a member whose flat document would lose state (wire_extensions.flattening_loses) is stored as its single-file sectioned document, which carries every carrier and which snapshot_from_dict already reads (storage/bundle_member_carriers.py); every other member keeps its flat bytes. Versioning (D2): the narrowest axis an older reader fails closed on is the bundle's own schema_version, so such a document or archive declares 5 (SECTIONED_MEMBERS_SCHEMA_VERSION); one without a sectioned member keeps declaring what it did (2-4) and its bytes. A reader at 4 refuses a 5, instead of decoding a sectioned member as a flat snapshot with every field defaulted; a sectioned member under a declared version below 5 is refused as corrupted. No D13 adapter: no stored document changes meaning.
Nested record fields (2026-10-31). Unknown keys are kept at any depth inside an identity-matched record (parameters, record fields, enum members) and on the binary section's ELF/PE/Mach-O metadata objects (including ELF symbols and imports, PE and Mach-O exports), keyed by the top-level record's identity and ordinal and the nested record's list position. The nested record classes are the list-of-dataclass fields of each model class's type hints (wire_record_extensions.nested_record_classes), so they cannot drift from the model.
Presence when functions is blocked (2026-10-31). The functions detector reads types (and debug, graph, layout, semantic_ir), so a blocked types section made a plain function removal unreportable. A separate function_presence detector, reading only binary and declarations, reports removals classified exactly as functions does (diff_symbols._check_removed_function), and runs only while functions is blocked -- dedupe by construction, no post-pass. Pairing is by mangled name only (the extern-C and ctor/dtor reconciliations compare signatures).
Runtime carrier. The decoded snapshot holds the carriers on the runtime-only, compare=False field AbiSnapshot.wire_extensions, beside unreadable_sections, so dataclasses.replace, copy and pickle keep them together; a sectioned write of a snapshot naming unreadable sections but holding no carriers is refused.
No version bump. Every carrier is written only when non-empty, so existing documents keep their bytes. A pre-D14 reader still never misreads the new states: it refuses an unknown section kind and a newer section version on its own. It would ignore a must_understand name on a known section's header, so a writer adding one bumps that section's section_schema_version.

D15 — The wire contract is independent of runtime classes¶

ADR-063 D8 owns the Domain → DTO vN → wire layering. This amendment adds three requirements to it:

A field inventory. A checked-in per-section field inventory, generated from the SectionDTO encoders, is diffed in CI. A wire change without a section_schema_version bump and a D13 adapter fails.
Byte identity. A fixed corpus serializes to identical bytes before and after a runtime-only dataclass refactor.
Section availability. At section level, the states use D3's FactAvailability statuses. A missing section means not_collected. A present but empty section means present. An extractor failure is stored as failed, with diagnostics. A missing section never reads as present. The carrier is the section-header table in the document or manifest, not the payload. Every section kind the producer attempted has a header entry with its FactAvailability, even when no payload object exists. So failed is a header entry with no payload, and not_collected is a header entry with that status, or no entry at all for producers that predate this rule. The field inventory covers the header entry.

Implementation decisions (2026-10-30). The inventory is generated by tests/_wire_inventory.field_inventory from the document the encoders write for a fixed corpus; tests/unit/storage/test_wire_contract.py diffs it against tests/fixtures/wire_contract/field_inventory.json and accepts a moved section only with a section_schema_version bump and a registered migration from the pinned version. The availability carrier is a document-level section_availability table beside section_schema_versions: a key a pre-D15 reader ignores, so no version bump. A failed entry stays listed in section_schema_versions without a payload, so such a reader refuses the document rather than read the section as empty; this build reads it as unusable and blocks its readers like an unknown must-understand name. Producers record a failure with storage.wire_extensions.record_section_failure, which also blocks the readers of the live snapshot, so an in-process comparison and one of the stored dump agree. Its production caller (2026-10-31) is the PE dump: a PDB that is located but fails to parse records debug as failed with the parse error on the export-table path, where debug holds nothing but the PDB's facts. A missing PDB is not_collected (optional inputs stay optional). Other sections aggregate several extractors whose failures are fact-family FactAvailability inside the payload, so no other extractor fails a whole section; a header-scoped PE dump's debug section also holds the header parse's provenance and is not failed wholesale (known gap). Re-checked 2026-10-31 for castxml and DWARF: a castxml failure aborts the dump (requested headers are not optional), and a DWARF open/parse failure covers neither debug (which also holds the header AST's provenance and reliability facts) nor layout (scope and contract facts); DWARF is also an optional input on the binary path, so failing a section for it would let an unrequested input lower a passing run's assurance. Neither gets a producer. A not_collected entry is written back absent on re-save (it was re-saved as present, which made the document unreadable). A not_collected section adds no assurance note when a detector reads it, by decision: every pre-D15 document and every optional section a run did not request (build, source) is not_collected, so a note would fire on nearly every comparison, and the missing evidence is already reported where the detector reads it, through the fact-family FactAvailability defaults the decoder fills. Only failed and blocked sections take the unreadable_section_gap path.

Amendment (2026-11-03): a not_collected section is read per section. The decision above is superseded. Its premise was that not_collected would cover every unrequested section; in practice the writer stamps every section it writes present (an unrequested build is written, empty), a pre-D15 document has no table, and no producer writes a not_collected entry. So an explicit entry is rare, and where one appears the decoder did not report the gap: it filled the section's fields with defaults, and the comparison reported every entity the other side held as removed or added (a not_collected types section read as type_removed with assurance complete). Now, for the kinds whose decoded fields cannot say "absent" (binary, build, debug, declarations, layout, types), a not_collected entry blocks its readers exactly as failed does; the identity section cannot be not collected; graph and semantic_ir decode absence as None and are unaffected. A section absent from a document with no entry is unrecorded and reads as before. The decoded snapshot carries every non-present section in AbiSnapshot.section_availability, and the compare report lists them in analysis_assurance.section_availability (report schema 7.1). The producer API is storage.section_states.record_section_availability (failed or not_collected); record_section_failure moved there. The inventory is now generated from the encoders' declarations (tests/_wire_inventory.declared_field_inventory), and a written corpus must use only declared keys. No snapshot or section version bump: a pre-D15 reader ignores the table, and the reader already accepted not_collected entries; only its reading of them changed.

D16 — One digest per question, none silently redefined¶

Role	Covers	Today
`transport`	the exact stored file bytes, after compression	`storage/transport_digest.py`: SHA-256 of each package-archive member's stored bytes, in its zip extra field
`publication`	the exact published snapshot, including capture metadata such as `created_at`	`snapshot_content_digest`, unchanged
`evidence`	D5-normalized facts, excluding volatile metadata and the order of unordered collections	`storage/evidence_digest.py`; persisted as the optional `ArtifactRef.evidence_digest`
`comparison_input`	the `evidence` digests plus the contract and policy inputs one comparison read	not implemented: no comparison-result cache exists to consume it

Each role answers one question:

transport: is this stored file intact, before decompression?
publication: is this the exact artifact that a release recorded?
evidence: is this the same evidence? Used for dedup and caches.
comparison_input: is a cached comparison result still valid?

The object store's address is a separate thing and keeps its meaning. JSON objects are addressed by semantic_digest, and raw payloads by raw_digest over the uncompressed payload (project_snapshot_store.py). That is payload integrity, not transport.

snapshot_content_digest keeps its meaning. The created_at probe is that digest working as designed, not a defect. The reorder probe is a defect only if it reproduces on the evidence digest. D5 already requires stable sort keys there, and A1.9 adds the test. evidence and comparison_input are added as new fields. No stored digest is reinterpreted.

Implementation decisions (2026-10-31). transport. The one place stored bytes travel as a unit is the package archive (storage/project_package_archive.py), so the packer writes the SHA-256 of each member's stored (zstd-compressed) bytes into that member's zip extra field (header id 0x6461); unpack_project_package verifies every member before writing anything and refuses a mismatch by member name, or an archive in which only some members carry the field. An extra field keeps the layout, so an older reader still unpacks a new archive and an archive without any field (packed before this) is still read: no format bump. The directory store keeps verifying each object's content address on read; hashing its compressed files a second time would answer no new question. comparison_input stays unimplemented: no comparison-result cache exists, and a digest with no consumer would be an unverified claim.

D17 — Immutable refs and atomic publication (amends D6)¶

D6's ref layout changes to versioned, never-overwritten paths:

refs/variants/<variant-id>/<ref-digest>.json
refs/artifacts/<artifact-id>/<ref-digest>.json

manifest.json records {id, ref_digest} per ref, not bare ids.

ref_digest is storage.canonical.semantic_digest (sha256, sha256:<hex>) over the complete canonical ref document. The document includes its id, its schema version, and every object digest it names. A ref carries no volatile capture metadata; any it gains belongs in the manifest. A reader recomputes ref_digest from the file it loads. A mismatch with either the path or the manifest entry fails closed, as a corrupted package, the same way an object digest mismatch does today. It stays the root and the only commit point.

Objects and refs are written and verified.
manifest.json is replaced by write-to-temp plus atomic rename.
A failed or interrupted publish leaves the previous manifest and every ref it names intact.

Version and migration. This is a layout change. It bumps package_format_version, so older readers refuse new packages, as D2 requires. A D13 adapter reads the id-path layout. Regression tests cover:

a failed object check;
an interruption between the ref writes and the manifest write;
a changed-content republish under the same ids.

Concurrent writers to one root are serialized. They take storage/cross_process_lock.py's lock for the whole publish.

Implementation decisions (2026-10-31). Windows. The lock uses msvcrt.locking over the lock file's first byte where fcntl is absent; both are released by the OS when the holder dies. Garbage collection. workflows/package_gc.collect_package_garbage(root, min_age_seconds=3600, dry_run=False) removes the files under refs/ and objects/ that the live manifest does not name, directly or through its refs. It holds the publish lock, so no publish is mid-commit; readers take no lock, so only files older than the grace period go, and a reader holding a manifest older than that fails closed (every read is digest-verified). Writers stage objects before taking the lock, so DirectoryObjectStore.put refreshes the mtime of an object it reuses, an in-flight .tmp file is never collected inside the default grace period, directories are never removed, and a writer whose staged object was collected anyway is refused at its locked re-verification rather than publishing a dangling reference. It is a Python API only: ADR-054 admits a new root command only past its bar, and no project subcommand exposes package maintenance yet. This narrows ADR-066 D1's "content digest" for release history to the publication digest.

D18 — Selective load before more compression (amends D8)¶

D8's size-bounded chunking target, 1-8 MB decoded, is withdrawn.

Sections are not pre-chunked. The 63% measurement argues against it.
A section is split only when a measured access pattern needs part of it.
The performance lever is D8's lazy, per-section loading. D8's preload of the small L0 binary sections stays, because pairing needs them before any analysis is chosen. Every other section is opened only when a requested analysis reads it.
Multi-library runs share an immutable, digest-keyed (evidence) decoded-section cache (D11).

Implementation decisions (2026-10-31). Step 1, the per-section decode split, is implemented; lazy loading of any further section is not.

Each section kind has its own decoder (storage/snapshot_section_decode.py); every derivation spanning sections is a named join with declared reads (raw sections) and after (decoded steps), listed in DECODE_STEPS. A step sees the document only through a view of its declared sections, which raises on an undeclared read, so the declarations are enforced, not documented.
The eager path composes the steps and must stay byte-identical (saved bytes, publication and evidence digests); a test pins this against the same steps fed the whole document, and each section decoder against its section alone.
The joins that block laziness are kept as joins rather than redesigned: from_headers inference, the reliability flags and the ADR-063 legacy fact backfill (which rewrites decoded declarations/types), the generation spelling migration, semantic_ir assembly and closure-identity renumbering. A section becomes lazy only for a document on which every join touching it is a no-op; semantic_ir assembly is deferred into a cell that forces its input sections. The plan's A1.9 D18 entry holds the dependency table and the next step.

Implementation decisions (2026-10-31, step 2). The declarations, types and debug-layout sections decode into one attribute, AbiSnapshot.semantic_ir (the IR owns the declaration store), so they are deferred together, with the semantic_ir section and the per-entity half of extraction_scope:

AbiSnapshot.semantic_ir is a LazyGraphField; a lazily loaded snapshot holds a PendingGraph whose decoder (model/lazy_declarations. DeferredSemanticIR) rebuilds the snapshot the eager decode would build and yields its IR. semantic_ir assembly (finalize_snapshot_semantic_ir) runs inside that cell.
Eligibility (storage/snapshot_lazy_decode.lazy_ir_eligible): recorded from_headers; schema_version at or above every legacy fact-backfill rule (so the backfill is a no-op); the document text scanned and free of closure markers, which proves the closure-identity renumbering a no-op. Any other document decodes eagerly. The generation spelling migration runs on the whole document first in both modes.
The ELF-binding backfill and the stored export-absence reconciliation run on every load and rewrite declarations, so they are queued into the cell (defer_declaration_step) and run in their eager order when it decodes.
A decode error in a deferred section is raised at every access, the error the eager load raises. A malformed deferred section is therefore reported at first use, not at load; eager_section_decode() restores load-time reporting.
Not built: per-pair release of decoded sections and the shared decoded-section cache. Nearly every detector reads declarations/types, so a full compare decodes the cell anyway, and the peak is set by the whole-document json.loads, which lazy decoding does not remove. No real consumer path would read the cache, so per D11 it is not built. Measurements are in the plan's A1.9 D18 entry.
2026-11-02: a per-section parse (each section's JSON parsed from its own text span) was measured and not adopted. D14's record-extras split and the generation migration still parse every section at load, and the best case is 20% below today's peak. The element-level declarations decode that would bound the per-member peak is deferred as a separate future item (maintainer decision); release fan-out peak does not grow with member count. Figures are in the plan's A2.5 entry.

Exit criteria and non-goals¶

A1.9 is complete when these hold:

each probe has a passing regression test;
validation passes on the oneDAL/MKL corpus and on Windows and macOS;
a test shows a second concurrent publisher blocked or refused.

This amendment introduces no binary codec and no user-facing compression knob (D1, D12).

Amendment (2026-11-03): an optional fact is an extension, not a global bump¶

Status of this amendment: Accepted — implemented (D19). Plan item A1.10 in the storage format v2 plan.

Context. Almost every new persisted fact so far raised the global SCHEMA_VERSION (storage/snapshot_schema_versions.py, 69 at this amendment). A reader hard-rejects a document newer than its own version (_MIN_SCHEMA_VERSION_REQUIRING_HARD_REJECTION), so one optional new fact made every new dump unreadable for every older build, including the analyses that never read the fact. D14 already gives each section header an extensions map that an older reader keeps verbatim, and a must_understand list that blocks only that section (or, at document level, the document). D2 says to use the narrowest axis that makes an older reader fail closed. The global integer was the widest one, used by default.

D19 — An optional new fact is a declared extension fact¶

Rule.

A new fact whose absence an older reader already reads correctly as "not collected" is an extension fact. Its value is stored in its section header's extensions map. It does not bump SCHEMA_VERSION, any section_schema_version, or any package axis.
must_understand is set only when ignoring the fact would make an older reader's result wrong, not merely less informed. A SECTION_MUST_UNDERSTAND fact blocks exactly the analyses that read its section (D14). A DOCUMENT_MUST_UNDERSTAND fact refuses the document. must_understand_when narrows the requirement to the values that need it, so a common value costs an older reader nothing.
The global SCHEMA_VERSION is reserved for envelope changes (the document's own shape: top-level keys, section set or layout, framing) and wire changes (an existing field's encoding or meaning, so an older reader would misread a value it believes it understands). Each bump after v69 records its kind and why it cannot be an extension fact.

Mechanism.

storage/extension_fact_catalog.py: ExtensionFact declares a fact's dotted name, its section, its ReaderSemantics (OPTIONAL/SECTION_MUST_UNDERSTAND/DOCUMENT_MUST_UNDERSTAND), what its absence reads as, its owning ADR, its encode/decode pair and an optional must_understand_when. EXTENSION_FACTS is this build's catalog, a static tuple in that module, never filled by registration calls from elsewhere, so what a build understands cannot depend on import order. wire_extensions.UNDERSTOOD_EXTENSIONS is derived from it. A fact name is <namespace>.<name> with at least one dot, so it can never be read as a section kind (the two share must_understand).
storage/extension_facts.py: write_extension_fact records a declared fact on a snapshot's D14 carriers and maintains its must_understand entry; read_extension_fact returns a Fact: PRESENT, NOT_COLLECTED when absent, or FAILED when the section is blocked for this reader or the stored value does not decode. The value is never a domain-model field (ADR-063 D8). It is in the evidence digest, survives every re-save and in-process copy, and travels into packages and bundles with the other carriers.
A write refuses header extensions recorded for a section the document does not write (for example a graph fact on a snapshot with no graph): D14 makes dropping them a defect (wire_extensions.refuse_orphaned_header_extras, on both the materialized and the streamed write path).
snapshot_schema_versions.SCHEMA_BUMP_REASONS maps each version after SCHEMA_BUMP_RULE_BASELINE (69) to SchemaBump(kind, why_not_extension), kind envelope or wire. The schema-bump-reason ai-readiness check (scripts/schema_bump_gate.py) fails a bump without a row, a placeholder reason, an unknown kind, a row for a version not allocated, and a moved baseline (the baseline is pinned in the gate as well).

Scope. Section-header facts. A per-record extension fact (one value per function or type) is not implemented; record-level unknown fields are already preserved (D14), but a typed per-record API is future work. The flat snapshot_to_dict document has no header to carry a fact, so a member or package written from it follows wire_extensions.flattening_loses (stored sectioned), as for every other carrier. Existing v1-v69 fields are not migrated: the rule applies going forward.

Older readers. Every build that accepts a v69 document reads the sectioned extensions map, except main commit 9bc29954f (2026-10-29), which allocated v69 one day before D14 landed and was never released; the newest tag, v0.7.0 (2026-10-13), reads up to v60 and refuses v69 outright. D14 said a writer adding a must_understand name to a known section also bumps that section's section_schema_version, to protect a pre-D14 reader. For an extension fact that protection is not required: no released reader can accept the document and ignore the name. A D14 reader blocks the section either way, so the bump would change only the diagnostic text.

Applied to ADR-079. ADR-079's export_read (complete/partial/ failed per platform) is the first candidate. An older reader that ignores a partial value reads a cut-short export table as complete, which is wrong, so the fact is SECTION_MUST_UNDERSTAND on the binary section with must_understand_when true for any non-complete value. A dump with a complete read is readable by every older reader at no cost, and one with a partial read blocks only the analyses that read binary there, instead of refusing the whole document. ADR-079's D5 global bump is superseded by this mapping; its D1-D4 stand.

Consequences. A dump that adds an optional fact stays readable by the previous release. A must-understand fact narrows an older reader's result instead of refusing it. A reviewer sees every global bump's justification in the diff that makes it.