Skip to content

Project Snapshot Format (ProjectSnapshot, v2)

This page is not the format dump writes. For that — the sectioned single-file envelope every dump produces by default — see Snapshot Format. This page describes the directory-backed, content-addressed package, an advanced storage shape with no dump CLI flag behind it today.

Redesigned as a single file (Phase 8). dump's real -o/--output/ stdout output is now this format's D8 section split (independently versioned binary/declarations/types/layout/debug/build/ graph/provenance sections, structurally validated on read) packaged as one JSON document (storage.sectioned_document), not a directory — every dump invocation gets this by default, no flag needed. serialization.snapshot_from_dict reads either shape transparently, so an older flat .abi.json a prior build wrote stays fully readable.

The directory-backed package this page otherwise describes (the manifest.json/refs//objects/sha256/... layout) still exists as a typed-API primitive (project_snapshot_legacy.write_legacy_snapshot_package) and compare still accepts one as an input path — but no dump CLI flag writes one today. The directory shape's real value (content dedup, independent per-section objects) only pays off once a project shares content across multiple artifacts, which nothing produces yet; for the single-artifact case every dump performs today, the single-file shape gets the same structural benefits without the directory's storage-UX cost (many small files instead of one, awkward to scp/commit/upload as a CI artifact). See the ADR's own Status for the full picture.

ProjectSnapshot is the replacement for four separate persistence shapes (per-library .abi.json snapshots, baseline sets, BundleFacts, and embedded BuildSourcePack data) — one content-addressed package per project, addressed by digest rather than embedded whole. See the storage-format-v2 plan for phasing and acceptance criteria; this page documents the on-disk shape itself.

Directory layout

A package is a directory (D6):

project.abicheck/
  manifest.json            # small; loads immediately
  refs/variants/<variant-id>/<ref-digest>.json
  refs/artifacts/<artifact-id>/<ref-digest>.json
  objects/sha256/<aa>/<digest>.json.zst

manifest.json carries only the package's version axes (see below) and one {id, ref_digest} entry per variant and per artifact (variant_refs/ artifact_refs) — never a full record. Each record lives in its own ref document, loaded lazily. The document carries a ref_schema_version, and its ref_digest is the semantic digest of the whole document; a reader recomputes it and refuses a document that does not match. Section content (declarations, types, layout, ...) is never embedded in a ref — it is a separate, content-addressed object under objects/, referenced by digest. An artifact ref may carry evidence_digest: the evidence digest of the artifact's snapshot document (capture metadata such as created_at excluded, identity-matched record lists in a stable order). It is optional; a reader that does not know it ignores it, and it is covered by ref_digest like every other field. An artifact ref may also carry carriers: an object reference (kind: "wire_carriers") to what the artifact's snapshot holds beyond its decoded sections -- a failed or not_collected section and its diagnostics, a section kept verbatim (an unknown kind, an unknown must-understand name, a newer section version), section-header extensions and must_understand names, unknown record fields at any depth, and a document-level must_understand list (see snapshot format). The sections it holds undecoded are absent from the ref's sections and from the manifest's section_schema_versions, and the evidence_digest covers it. A ref with carriers is written at ref_schema_version 2, which a reader of version 1 refuses; every other ref is written at version 1. A package format v1 package kept each ref at a mutable refs/variants/<id>.json/refs/artifacts/<id>.json path and listed bare ids; it is still read.

Publication is atomic. A writer verifies every section object, writes each ref document once to its digest-named path, and only then replaces manifest.json by temp file plus atomic rename — the commit point. A ref document is never overwritten, so a publish that fails or is interrupted, including a republish of changed content under the same ids, leaves the previous package fully readable. Writers to one package are serialized by a cross-process lock (flock, or msvcrt.locking on Windows).

Superseded files stay until collected. A republish leaves the previous refs and objects behind, unnamed. abicheck.workflows.package_gc.collect_package_garbage(root) removes them under the same lock, only once older than a grace period (one hour by default), so a reader still using the previous manifest is not cut off; a reader older than that fails closed, never reads wrong data. There is no CLI for it.

The one-file transport form is a zip archive (see One-file transport).

Content addressing

Every object under objects/ is addressed by the SHA-256 digest of its own canonical form: keys sorted, floats normalized, unordered collections given an explicit sort key, and a reserved capture metadata subtree excluded from the hash domain (D5). Two objects with identical content — however they were produced, in whatever order their producer traversed its own data — always address the same file; storing the same content twice is a no-op. A JSON-shaped object is stored at objects/sha256/<digest[:2]>/<digest>.json.zst; a raw binary payload (one ObjectStore.put() cannot represent as JSON) is stored the same way at .bin.zst instead.

Version axes

A package's manifest.json states seven core version axes plus two legacy-import provenance axes (D2) — none of them the single overloaded schema_version integer .abi.json uses:

Axis Meaning
package_format_version container/manifest layout
section_schema_versions per-section field layout, keyed by section kind
normalization_recipe how spellings/paths/identities were normalized
producer what emitted the facts (name, version, binary digest)
extractor_generation extraction semantics epoch
resolver_generation derived-graph/resolution semantics epoch
comparison_contract_version what a reader must understand to compare safely
source_schema_version / source_producer_generation import provenance for a package adapted from a legacy .abi.json document

package_format_version and comparison_contract_version fail closed: an unstated or unrecognized value refuses to load rather than being treated as "this build's own version". The rest are informational.

manifest.json may also carry an optional must_understand list, written only when non-empty: an unknown name refuses the package. A section object's own header may carry must_understand and extensions; an unknown must-understand name on a section blocks only that section's analyses (they are reported not comparable), in a directory package as in a single-file .abi.json (see snapshot format). On the provenance section, which holds the snapshot's identity, it refuses the package.

Sections

An artifact's content is split into independently-addressable sections (D8) — binary, declarations, types, layout, debug, build, source_abi, graph, provenance, diagnostics, raw_refs are the named vocabulary, though ArtifactRef.sections accepts any section kind string.

SemanticIR (the cross-backend declaration/type representation) is the one domain type actually promoted onto a typed, versioned section built from a real domain object, under section kind "semantic_ir", via abicheck/storage/dto.py's semantic_ir_to_dto/semantic_ir_from_dto. Every other field a legacy .abi.json document carries is split across the rest of D8's named sections too (abicheck/storage/legacy_sections.py's split_legacy_document/join_legacy_document): binary (ELF/PE/Mach-O container facts, build_id, source path/size), declarations (functions, variables, enums, typedefs, constants, Python/SYCL extension surfaces), types (record/class/union types), layout (DWARF-vs-header coherence, conditional fields, extraction contract), debug (DWARF/AST toolchain facts and their reliability flags), build (embedded BuildSourcePack data), graph (the surface reachability graph), and provenance (library/ version identity, git/build metadata, dump's own dump_provenance block). Each section is independently versioned (storage.dto.SECTION_SCHEMA_VERSIONS) and carries an explicit, reviewed field allowlist — a document field with no assigned section is a hard error at import time, not a silent drop. What this split does not do yet is decode a section's own internal shape into a typed domain object the way semantic_ir is: elf/dwarf/build_source/... inside their own section still carry exactly the JSON serialization.snapshot_to_dict() already produced for them. That deeper per-field typing is real, separately-scoped future work.

Every SectionDTO is a small, explicit envelope:

{
  "section_kind": "semantic_ir",
  "section_schema_version": 1,
  "payload": { "...": "the section's own explicit, hand-encoded content" }
}

section_schema_version is the DTO's own version — independent of every other axis above — and a per-section-kind migration chain (abicheck/storage/dto.py's migrate_section_dto) advances an older payload to the current version one registered step at a time. No section's payload is ever built via dataclasses.asdict(): each is written by an explicit, hand-authored encoder (storage/semantic_ir_codec.py's semantic_ir_to_document, for the one section that exists today), enforced mechanically by scripts/check_ai_readiness.py's project-snapshot-dto-no-asdict check.

Importing a legacy snapshot

abicheck/storage/import_v1.py's import_legacy_snapshot takes one already-serialized .abi.json-shaped document (any schema version this build can still read) and produces a one-artifact, one-variant PackageManifest — a single-library dump represented as a minimal project. No existing baseline is rewritten: the adapter only ever reads a document and builds new, additional structures from it. export_legacy_snapshot is the exact inverse: given an ArtifactRef and the ObjectStore it was written into, it reads every section back, migrates each to its current version, and reassembles the original document.

CLI wiring

storage.sectioned_document's to_sectioned_document/ from_sectioned_document package the same D8 split as one JSON document, reused by serialization.snapshot_to_json/snapshot_from_dict (the real -o/--output/stdout write and read path) so every dump/compare invocation gets it by default. abicheck/project_snapshot_legacy.py's write_legacy_snapshot_package/read_legacy_snapshot_document remain the real, directory-backed round trip built on the primitives above (DirectoryObjectStore + import_legacy_snapshot/export_legacy_snapshot + write_project_manifest), reached through abicheck.workflows.storage's facade re-export (frontends -> workflows -> storage layering):

  • No dump CLI flag writes a directory package today — every dump invocation writes the single-file sectioned shape instead (see above). The directory writer stays available as a typed-API primitive for a caller that wants it directly.
  • compare still accepts a ProjectSnapshot package directory as an input path — detected by a real, validated manifest.json read (is_project_snapshot_package_dir, which distinguishes it from a BuildSourcePack's own identically-named manifest.json, and from a plain directory-of-libraries compare operand, which stays routed to the release/bundle fan-out). Resolved by workflows.input_resolution.resolve_input's directory branch into the identical in-memory AbiSnapshot a .abi.json/sectioned file resolves to, so every downstream detector, report, and exit code behaves the same regardless of which of the three shapes the input actually is.
  • compare's release fan-out (a directory/package operand) also accepts a multi-artifact ProjectSnapshot package directory as either side, alongside its original loose-directory-of-.so- files operand — cli_resolve.classify_compare_operand distinguishes a multi-artifact package (routed to the fan-out) from the single-artifact shape above (still read directly as one snapshot) via workflows.release_package.is_multi_artifact_package. workflows.release_package.resolve_release_package_map unpacks the selected variant into the same {canonical_library_name: Path} shape the fan-out already builds from a live directory — one real, independently-readable single-artifact sub-package per library (project_snapshot_legacy.materialize_release_variant_artifacts, the storage-layer half; kept in workflows/ rather than project_snapshot_legacy.py itself only because matching a stored artifact's real filename to a live directory's own filename needs binary_utils._canonical_library_key, an extract-layer primitive storage-classified code may not import). The three sides of a compare/compare-release pair may be mixed freely — stored/live, live/stored, and stored/stored all produce the identical DiffResult set a live/live run over the equivalent loose directories would, per-library bundle analysis (bundle.build_bundle_snapshot_mixed) included.

--variant (cli_options.variant_options) selects which of a multi-variant package's VariantRefs to compare when a package declares more than one — side-scoped like every other two-sided compare input (--variant old=v1 --variant new=v2; a bare value applies to both); omitted, a package must declare exactly one variant or the CLI raises a usage error (exit 64) naming the ambiguity, the same "no silent first match" discipline SymbolIdentityIndex.unique_alias_match already establishes elsewhere in this codebase. Declared once on compare itself (expose_value=False, stashed on click.Context.meta via cli_options. _stash_variant_in_context) rather than threaded through every intermediate dispatch function's own parameter list, and read back (cli_options.variant_kwargs_from_context) from the still-current Click context by cli_compare_release.compare_release_cmd's own body — the one function that actually needs them, reached whether the operand was classified as a loose directory or a package.

One-file transport (zip)

A package directory can be moved as a single file: workflows.storage.pack_project_package(package_dir, out_path) writes it as a zip archive, and unpack_project_package(archive, dest_dir) restores it file-for-file (abicheck/storage/project_package_archive.py, storage-format-v2 A1.1).

  • Layout. A stored mimetype member comes first (application/vnd.abicheck.project-snapshot+zip), so the file identifies itself without a suffix. The package's own paths follow in sorted order: manifest.json, refs/**, objects/**. Every member is stored uncompressed, because objects are already zstd-compressed.
  • Deterministic. Every zip header field is pinned (storage/zip_member.py, shared with the G40 bundle archive), so the same package always packs to the same bytes.
  • Untrusted input. Unpacking refuses, before writing anything:
  • any member that is not exactly a package path (no .., absolute path, backslash, directory entry or stray file);
  • a symlink, a compressed member or a duplicate name;
  • an archive over the member-count or size budgets;
  • a member whose stored bytes do not match the transport digest (SHA-256) the packer wrote into its zip extra field, or an archive in which only some members carry one. An archive packed before this field existed carries none and is still read.

Object digests are verified when each object is read, as for a directory. - Accepted as a compare operand anywhere a package directory is. - The CLI unpacks an archive operand into a private temporary directory for the run (frontends/cli/options/operand_path.py). A multi-artifact archive therefore reaches the release fan-out exactly as its directory would. - The typed-API input path reads a single-artifact archive directly (workflows.project_package_input.resolve_project_package). - Zip, not .tar.zst, for the reason the G40 bundle archive records: the central directory allows opening one member without reading the rest.

No dump flag writes a package, so an archive is produced through the typed API, the same way a package directory is.