Validate recovery snapshot endorsement chains in memory

[achamayou]

Summary

Implements the recovery-snapshot endorsement-chain design from Discussion #8080, simplified in response to Eddy Ashton’s review: the chain is retained only in memory for the current recovering node. No endorsement sidecar is created, read, cached, or reused.

• Applies only to StartType::Recover with a local startup snapshot.
• Before snapshot KV deserialisation and before PartOfPublicNetwork, scans configured ledger files directly, without ringbuffer requests or a temporary KV store.
• Extracts bounded previous-service-identity endorsement candidates from complete public-domain entries after snapshot seqno S.
• Accepts a snapshot directly valid under operator-provided recover.previous_service_identity; otherwise requires a non-empty complete newest-to-oldest COSE chain from that trusted identity to the snapshot signer.
• Keeps snapshot installation outside verification catches. Verification failures fall back before Store mutation; install/deserialisation failures mark Store failed and propagate.
• Leaves snapshot and ledger files unchanged.

Trust model

config.recover.previous_service_identity is the sole target trust anchor In.

The ledger suffix is only untrusted transport for candidate CoseEndorsement records. Acceptance does not rely on suffix service state, signature transactions, ledger inclusion, or serialized Merkle trees. The complete candidate list must prove:

• all candidate writes are after S;
• previous_version ordering is complete and exact;
• signed COSE epoch headers match record metadata;
• adjacent epoch ranges are contiguous;
• every signature endorses the preceding service key;
• the newest signing key equals In; and
• the oldest range covers S.

The derived oldest key verifies the snapshot COSE receipt claims digest, inclusion proof/root, and detached signature. Injected, missing, reordered, or tampered candidates break the complete anchored chain and force full replay.

JSON receipt augmentation remains intentionally out of scope. JSON snapshots are accepted only when directly valid under In.

Untrusted candidate boundaries

For Recover, snapshot discovery/name/seqno parsing, file reads, transaction/segment headers, and pre-install receipt structure/claims are candidate verification. Malformed discovery metadata selects normal no-snapshot recovery; malformed individual files remain untouched while the next candidate is considered.

Recovery uses a throwing exact-length reader rather than process-terminating files::slurp, so missing, non-regular, raced, unreadable, oversized, short-read, and seek failures fall back before Store mutation. Join retains its previous fatal structural behavior and corrupt-receipt ignore/rename behavior.

The shared raw KV reader validates fixed-width reads, size prefixes, payload lengths, vector element counts, cursor advances, and public-domain slices before pointer arithmetic, allocation, or copying.

Receipt proof hardening

All fixed SHA-256 fields in decoded COSE Merkle proofs are checked before any fixed-extent span, copy, or hash construction:

• leaf write-set digest;
• leaf claims digest; and
• every path sibling.

Each must be exactly Sha256Hash::SIZE. Decode-time and defensive root-recomputation checks reject short or oversized fields with COSEDecodeError. Valid 32-byte fields and existing receipt root/signature verification are unchanged.

Tests

• Chain tests cover ordering, tampering, missing links, wrong In, back-pointers, signed epoch metadata, injected candidates, adjacency, and snapshot coverage.
• Scanner tests cover direct candidate collection without signature transactions, count/item/aggregate/raw-entry limits, and malformed evidence before install.
• KV tests cover truncated fixed fields, short/prefix-only values, the 9-byte/declared-length-2 regression, impossible lengths, non-divisible vectors, invalid null input, invalid public-domain lengths, and truncated claims digests.
• COSE tests mutate a valid receipt sibling to lengths 0, 1, 31, 32, and 33; only 32 succeeds. Adjacent write-set and claims digest length mutations are also rejected.
• Snapshot boundary tests cover zeroed/truncated/oversized-body snapshots, malformed names, missing/non-regular files, unchanged Store state before install, and install-exception propagation.
• recovery_snapshot_endorsements_test covers real two-DR in-memory recovery, incomplete suffix fallback, actual node-local byte preservation, malformed discovery/read/header candidates, and a real snapshot receipt with a one-byte proof sibling. Every malformed case enters full-ledger recovery without snapshot installation.
• Focused builds/units, production clang-tidy, Python formatting, shellcheck, and all non-Ruff repository checks pass.

Lint tooling is unchanged from current main; this PR contains no Ruff/Black/mypy/pyproject/setup/tox/pre-commit/workflow configuration changes or lint-only Python cleanup. The only changed Python file is the functional recovery e2e above.

With restored current-main behavior, unpinned Ruff 0.16.0 fails on both a clean current-main checkout and the exact PR merge. Local normalized JSON comparison finds 505 diagnostics in each, with identical (file, rule, message, location) sets. GitHub VMSS Virtual A job 89429343314, step 6, reports those findings plus one runner-only EXE002 on executable tests/infra/platform_detection.py; that file has identical mode 100755 and blob 5b7e89711d38c191d70a26b8839f07749d2961f8 in current main and the PR merge. No lint pin, configuration workaround, or mass cleanup is included.

Limitations

• JSON receipt augmentation is not implemented.
• The pre-pass extracts only public endorsement candidates; it does not replay application state or establish ledger commitment for those candidates. Trust comes solely from the complete COSE chain anchored at In.

[Copilot]

Pull request overview

This PR implements recovery snapshot endorsement “sidecars” to allow recovery to start from a snapshot whose COSE receipt was signed by an older service identity, by deriving/verifying a contiguous endorsement chain anchored at the operator-provided recover.previous_service_identity and caching it alongside the snapshot.

Changes:

• Adds sidecar path derivation and lifecycle support (including cleanup when snapshots are deleted).
• Implements synchronous scanning of the public ledger suffix (without building/applying a KV) to collect and validate previous-service-identity COSE endorsements and persist them as a CBOR sidecar.
• Extends Python snapshot/COSE tooling and adds focused C++/Python/e2e tests, plus CI wiring for the new e2e test.

Custom instructions used:

• .github/copilot-instructions.md
• .github/instructions/reviewing.instructions.md

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/recovery_snapshot_endorsements.py	New e2e coverage for sidecar derivation, reuse, tamper handling, and fallback.
tests/ci-buckets.txt	Adds the new e2e test to bucket_b.
src/snapshots/filenames.h	Adds helper to derive snapshot endorsement sidecar path.
src/node/test/snapshotter.cpp	Unit tests for sidecar lifecycle and direct ledger-file scanning behavior.
src/node/test/network_identity_subsystem.cpp	Unit tests for newest-to-oldest endorsement validation and fail-closed behavior.
src/node/snapshot_serdes.h	Adds CBOR (de)serialisation for endorsements, sidecar IO, and recovery snapshot verification using endorsement chains.
src/node/rpc/network_identity_subsystem.h	Refactors endorsement signature verification to use shared helper.
src/node/rpc/network_identity_chain_helpers.h	Introduces shared endorsement parsing/validation helpers used by recovery and identity subsystems.
src/node/recovery_snapshot_ledger.h	Adds direct ledger-file scanner for service info + endorsement collection after a snapshot.
src/node/node_state.h	Integrates sidecar verification/derivation into recovery startup flow and adds sidecar cleanup/fallback logic.
src/host/test/files_cleanup_test.cpp	Verifies snapshot cleanup also removes adjacent endorsement sidecars.
src/host/files_cleanup_timer.h	Deletes endorsement sidecars when deleting old snapshots.
python/tests/test_cose_sign.py	Adds Python tests for endorsement sidecar chain verification and strict CBOR decoding.
python/src/ccf/read_ledger.py	Adds CLI options to verify a snapshot receipt against a target cert + optional endorsements sidecar.
python/src/ccf/ledger.py	Adds sidecar path helper and reader; snapshot API caches receipt/digest for verification.
python/src/ccf/cose.py	Adds CBOR encoding/decoding for endorsements and endorsement-chain verification utilities.
CMakeLists.txt	Registers the new e2e test target.

[eddyashton]

I don't think this should be stored in a sidecar file. The endorsement chain can be retained in memory by the current node, and used by that node.

I accept there's significant work in creating this endorsement chain, that we could save if a future recovery attempt is able to use this file, but believe that case will be exceedingly rare. We eagerly create snapshots on successful recovery, so falling into this in the first place is unusual. If we do serial recoveries with no fresher snapshots, its likely that each attempt is actually using the next identity, so this off-by-one endorsement chain doesn't help us (an endorsement chain to Identity N won't help us at N +1, its equivalent to junk because we jump to a crypto check, there's no attempt to augment an existing partial endorsement sidecar with newer certs. And good, that's even more complex!).

Fundamentally, this is a rare-enough path that we should prefer it being as branchless (simple) as possible. The real goal is to read the ledger and recover from the valid snapshot+suffix, but much of the code in this PR is concerned with managing sidecar files and testing that we can correctly ignore corrupt ones - totally unnecessary work.

Discussing this with the robot, it's raised the point that the sidecar file can be reused by other tools, eg read_ledger.py. But I think that's slightly artificial - when the file is there, it looks like good coverage for all of our tooling to use it (including duplicating path construction logic etc). Before the file existed, the tooling took no snapshot signer arg, and simply didn't care about this. I think that's better! There's already a big difference between what the Python tooling does and what the node does (snapshot discovery!), I don't think this is a useful extra bit of verification.

Big rant! TL;DR: Dump the file, store the chain in-memory for the current node, delete half of this PR.