bottomless: fix snapshot upload pipeline silently dropping generation snapshots

[LeMikaelF]

Summary

The bottomless continuous-backup pipeline could run for days while every generation it created was missing its base snapshot in object storage — degrading or breaking restore/fork/point-in-time recovery — and, in one failure mode, blocking checkpoints entirely so the WAL grew without bound. The production signature was recurring DispatchFailure(... BodyWriteAborted, NotEof(_)) snapshot upload errors, 20+ consecutive generations with no db.* snapshot object while checkpoints kept "succeeding" on the configured interval, and post-restart generations with no .dep parent link.

This PR fixes the five interacting defects behind that signature.

Bugs and fixes

1. A failed snapshot upload was never retried and wedged checkpoints until process restart. The snapshot task attempted its PutObject exactly once; on failure the checkpoint gate refused every subsequent checkpoint with SQLITE_BUSY, and with autocheckpoint disabled the WAL grew without bound. The upload is now retried with backoff inside the snapshot task (rebuilding the ByteStream per attempt so a retry never sends a stale Content-Length), and the checkpoint gate re-triggers the snapshot of the current generation when it failed or never ran. Re-uploading at any later point is correct because the main database file only changes on a TRUNCATE checkpoint, and checkpoints stay refused until the upload succeeds.

2. The empty-WAL checkpoint path falsely marked generations as snapshotted. skip_snapshot_for_current_generation() unconditionally stamped the snapshot waiter with success, laundering failed or still-in-flight uploads — this is why production showed days of on-schedule checkpoints with zero snapshots. The function is removed; the empty-WAL branch now re-triggers a missing snapshot instead of overwriting the failure state.

3. Concurrent snapshot tasks truncated each other's shared temp file. Compression wrote to a fixed db.gz/db.zstd path with truncate(true), shrinking the file under another task's in-flight upload stream (hyper's BodyWriteAborted/NotEof) or silently uploading a torn object. Compression now targets a per-generation artifact path, an in-flight flag (cleared by a panic-safe drop guard) guarantees at most one snapshot task at a time, and stale artifacts — including legacy fixed-name ones — are swept only while no upload runs.

4. The AWS SDK's stalled-stream protection killed slow snapshot uploads. A whole-database PutObject can run for minutes and legitimately stalls beyond the ~5s grace period whenever the process is busy; the watchdog abort surfaces as the same BodyWriteAborted signature and the SDK cannot retry a partially consumed stream. Stalled-stream protection is now disabled on the bottomless S3 client; bottomless owns its retry policy.

5. Restarts created orphan generations with no .dep link. compare_with_local returned early for any database with change counter > 1 — before storing the generation it compared against — so new_generation() saw no parent and wrote no .dep, severing the chain restore walks when a generation has no snapshot. The store now happens before the early return.

Snapshot upload log lines now include the database name and generation to make multi-tenant incidents debuggable.

Compatibility

S3 object keys are unchanged ({db}-{generation}/db.{gz,zstd,raw}, .changecounter, .dep, .meta) — only the local temp artifact path changed — so existing buckets restore unchanged, and old/new versions interoperate.

Testing

• New snapshot_pipeline regression tests in libsql-server/src/test/bottomless.rs drive the real bottomless WAL wrapper through a raw libsql connection against a per-test mock S3 server with failure injection, covering: upload retry after a transient failure, checkpoint recovery without a process restart (including the empty-WAL laundering case, which fails on the old code), the snapshot overlap guard, and the .dep link on restart.
• cargo test -p libsql-server bottomless passes, including the #[ignore]d backup_restore test (restore across generations missing snapshots).
• cargo test -p bottomless, cargo fmt --check, and cargo check on bottomless, bottomless-cli, and libsql-server are clean.

---

🤖 Generated with Claude Code

https://claude.ai/code/session_013Q5tV6oA1zXfrWhchz2Cqk