Skip to content

Snapshot write still builds the largest section's tree whole

Index: Known gaps

Status (2026-10-31): NARROWED. Storage plan A2.2's section step is done: a write no longer builds the whole encoded document tree, only one section's at a time. The largest section is still built whole.

What changed. storage/snapshot_section_stream.streamed_sectioned_document hands the writer a json_stream.LazyItems document whose sections member encodes each section when the serializer reaches it. The flat keys a section owns (legacy_sections.section_fields) go through snapshot_encode._snapshot_to_dict(..., keys=...), which walks only those fields and skips every post-pass none of whose outputs is wanted (encode_surface_graph, encode_semantic_ir, encode_extraction_scope, the sidecar entity ids, the embedded build source). The section is packaged by import_v1.legacy_section_dto_dict, the per-section step the whole-document split also uses, gets its D14/D15 carriers back, is written and dropped. Bytes are unchanged: tests/test_snapshot_section_stream.py checks indented and compact output against the whole-document encoder (tests/_whole_document_encoder.py) over Hypothesis-generated snapshots with record/header extensions, unknown, blocked and failed sections, and over 51 fixture-corpus snapshots.

Measured (tracemalloc peak of encode plus the fragment stream; RSS growth without tracing; same sha256 both ways):

Input Whole document Section by section Time
Synthetic, 40k functions + 40k records (two equal sections) 550.5 MiB, RSS +584 MiB 286.2 MiB, RSS +316 MiB 10.5 s -> 10.1 s
libstdc++.so.6 binary-only dump (23.9 MB JSON) 39.8 MiB 37.1 MiB 0.76 s -> 0.75 s

What remains. The peak is now the largest section. On a binary-only dump nearly everything is the binary section (the ELF symbol list), so the libstdc++ figure barely moves. Going below one section needs the record-level step: the section payload (canonical_form_shared, the section DTO codecs, restore_record_extras) would have to accept a lazy record list, and the per-record passes (_compact_elf_symbols, encode_fact_fields, encode_entity_ids, ...) would run per record as it is emitted.

Do not stream by emitting sections out of their current key order. The publication digest and existing baselines depend on today's bytes.