Skip to content

feat(acp): retry initial relay connection with terminal/transient error classification#1922

Merged
wpfleger96 merged 6 commits into
mainfrom
duncan/acp-startup-connect-retry
Jul 16, 2026
Merged

feat(acp): retry initial relay connection with terminal/transient error classification#1922
wpfleger96 merged 6 commits into
mainfrom
duncan/acp-startup-connect-retry

Conversation

@wpfleger96

@wpfleger96 wpfleger96 commented Jul 15, 2026

Copy link
Copy Markdown
Member

Summary

ACP agent startup previously made exactly one relay connection attempt — any dropped WebSocket handshake on a spotty link failed the agent outright, requiring a manual respawn.

retry_initial_connect() retries transient failures with jittered backoff (1 immediate + 5 delayed attempts at 1/2/4/8/16s base intervals), sharing the delay schedule already used by the post-start reconnect loop via STARTUP_CONNECT_BACKOFFS. Terminal errors fail immediately since retrying them is pointless.

Classification contract

is_terminal_connect_error() classifies every RelayError variant, and is_terminal_ws_error() exhaustively classifies every tungstenite::Error inner variant (no wildcard — a tungstenite upgrade forces reclassification at compile time).

Terminal (fail fast):

  • RelayError::Http — HTTP-level rejection
  • RelayError::Json / RelayError::UnexpectedMessage — protocol mismatch
  • WebSocket(Url(_)) — deterministic URL/scheme misconfiguration (never reaches the network)
  • WebSocket(Capacity(_)) / WebSocket(Utf8(_)) / WebSocket(HttpFormat(_)) / WebSocket(AttackAttempt) — deterministic handshake-shape failures
  • WebSocket(Http(resp)) — non-101 HTTP response, unless status is 408, 429, or 5xx (server-side transient conditions that can recover)
  • WebSocket(Protocol(_)) — all variants (wrong method/version, missing/invalid upgrade headers, accept-key mismatch, junk-after-request, etc.) except HandshakeIncomplete and ResetWithoutClosingHandshake (connection dropped mid-handshake or abrupt reset — link-level failures)
  • WebSocket(Tls(_)) — deterministic TLS config failures. On our rustls build the only connect-time Tls variant is InvalidDnsName
  • WebSocket(Io(_)) with a deterministic rustls::Error in the source chain — tokio-rustls wraps rustls handshake failures as io::Error with the rustls::Error as source; tokio-tungstenite surfaces these as Error::Io. Classifier walks get_ref() + source() chain to find the rustls::Error, then matches a terminal allowlist: InvalidCertificate(_), InvalidCertRevocationList(_), NoCertificatesPresented, UnsupportedNameType, PeerIncompatible(_) — deterministic cert/config/incompatibility failures that retry cannot fix. Relies on a single rustls version in the dep tree (0.23.40)
  • AuthFailed with invalid: / auth-required: / restricted: / blocked: prefix, or any unrecognized prefix (fail-safe)

Transient (retry):

  • WebSocket(Io(_)) without a rustls source, or with an ambiguous rustls error (AlertReceived(_), InappropriateMessage/InappropriateHandshakeMessage, InvalidMessage(_), DecryptError, PeerMisbehaved(_), General(_), and any other non-allowlisted variant — rustls::Error is #[non_exhaustive], so unknowns default transient) — plain transport failures (reset, EOF, timeout, refused, mid-handshake TLS transport loss) and ambiguous protocol/decrypt/server-alert shapes stay retryable under the bounded budget
  • WebSocket(ConnectionClosed) — link-level closure
  • WebSocket(AlreadyClosed) / WebSocket(WriteBufferFull(_)) — unreachable during connect_async; kept fail-safe transient
  • WebSocket(Http(resp)) where status is 408, 429, or 5xx
  • WebSocket(Protocol::HandshakeIncomplete) / WebSocket(Protocol::ResetWithoutClosingHandshake)
  • RelayError::ConnectionClosed / Timeout / NoAuthChallenge
  • AuthFailed with error: prefix — relay dependency fault (e.g., ban-state DB lookup failure), distinct from an explicit credential rejection

Changes

  • Add retry_initial_connect() with the jittered backoff schedule
  • Add is_terminal_connect_error() that classifies every RelayError variant
  • Add is_terminal_ws_error() with exhaustive match on every tungstenite::Error variant including Http status split and per-Protocol-variant classification
  • Add is_terminal_rustls_io_error() — walks io::Error inner error + source chain to find a rustls::Error wrapped as Io by tokio-rustls/tokio-tungstenite, then matches a deterministic cert/config/incompatibility allowlist (InvalidCertificate, InvalidCertRevocationList, NoCertificatesPresented, UnsupportedNameType, PeerIncompatible); ambiguous protocol/decrypt/server-alert shapes (AlertReceived, DecryptError, InappropriateMessage, General, etc.) default transient under the bounded retry budget
  • Add is_terminal_auth_failure() that classifies AUTH rejection text by NIP-01 prefix, cross-referenced against crates/buzz-relay/src/handlers/auth.rs
  • Extract STARTUP_CONNECT_BACKOFFS shared between startup and reconnect loops
  • Table-driven test covering every RelayError variant and every tungstenite::Error inner variant, including production-shaped Io-wrapped rustls errors terminal on the allowlist (InvalidCertificate(Expired), InvalidCertificate(NotValidForName), NoCertificatesPresented), transient on ambiguous variants (General, AlertReceived(InternalError), DecryptError), explicit transport-transient rows (ConnectionReset, UnexpectedEof, TimedOut), and Tls arm-pinning rows (InvalidDnsName, CertificateError::Expired)
  • Async do_connect_wrong_scheme_is_terminal test driving a literal https:// URL through production do_connect()
  • Tests for error:-prefixed AuthFailed retry/recovery, terminal error fail-fast, exhaustion, and backoff sleep behavior

@wpfleger96 wpfleger96 requested a review from a team as a code owner July 15, 2026 22:30
@wpfleger96 wpfleger96 marked this pull request as draft July 15, 2026 22:31
@wpfleger96 wpfleger96 marked this pull request as ready for review July 16, 2026 15:39
Comment thread crates/buzz-acp/src/relay.rs Outdated

@wesbillman wesbillman left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One classification issue remains: deterministic TLS certificate/configuration failures are retried as transient. I left an inline comment requesting a split and regression coverage.

The retry loop and remaining exhaustive WebSocket classification otherwise look solid, and current CI is green. The branch is 22 commits behind main; rebase and rerun CI after the code fix before approval.

npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 and others added 4 commits July 16, 2026 12:00
HarnessRelay::connect() previously made exactly one do_connect() attempt;
any dropped handshake on a spotty link failed agent startup outright,
requiring a manual respawn. Add retry_initial_connect(), which retries
transient failures (ConnectionClosed, Timeout, NoAuthChallenge, etc.)
with the same 1/2/4/8/16s jittered backoff already used by the post-start
reconnect loop, now shared via STARTUP_CONNECT_BACKOFFS. Terminal errors
(RelayError::Http, RelayError::AuthFailed) fail immediately since they're
deterministic — retrying bad credentials or a malformed URL just delays
surfacing a real problem.

The startup_watermark capture point and post-start reconnect/replay logic
are untouched; the retry loop is entirely inside connect().

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
…errors

is_terminal_connect_error() classified every WebSocket, Json, and
UnexpectedMessage error as transient, including deterministic ones: a
malformed relay_url (tungstenite Error::Url — bad scheme, missing host)
never touches the network, and a Json/UnexpectedMessage parse failure
reflects a protocol mismatch, not link noise. Both were burning the full
retry budget on failures a later attempt can't fix. Classify Http/Json/
UnexpectedMessage as terminal and split WebSocket on Error::Url (terminal)
vs everything else (transient, e.g. Io).

AuthFailed was uniformly terminal, but the relay emits a NIP-01 error:
prefix (crates/buzz-relay/src/handlers/auth.rs) for its own dependency
faults (e.g. a ban-state DB lookup that couldn't run) — a real transient
failure worth retrying, distinct from an explicit invalid:/auth-required:/
restricted:/blocked: rejection. Add is_terminal_auth_failure(), which
treats only the error: prefix as transient and fails safe (terminal) on
any unrecognized prefix.

Also corrects STARTUP_CONNECT_BACKOFFS's doc comment: the initial-connect
and reconnect loops share backoff *values*, not an identical schedule —
the reconnect loop skips the sleep after its final attempt.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
…retry

The Url-only terminal split inside is_terminal_connect_error() let
deterministic handshake failures (non-101 HTTP, missing upgrade headers,
accept-key mismatch) burn all six retry attempts. Replace with an
exhaustive match on tungstenite::Error — no wildcard, so a tungstenite
upgrade forces reclassification at compile time.

Terminal: Url, Capacity, Utf8, HttpFormat, AttackAttempt, Http (non-408/
429/5xx), and all Protocol variants except HandshakeIncomplete and
ResetWithoutClosingHandshake.

Transient: Io, ConnectionClosed, Tls, AlreadyClosed, WriteBufferFull,
Protocol::HandshakeIncomplete, Protocol::ResetWithoutClosingHandshake,
and Http 408/429/5xx.
WsError::Tls(_) was transient, burning all retry attempts on deterministic
TLS failures (InvalidDnsName, expired/invalid certificates). On our rustls
build, the only connect-time Tls errors are deterministic validation
failures — transport-level TLS loss surfaces as Io (already transient).
Flip Tls to terminal per Wes's review.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@wpfleger96 wpfleger96 force-pushed the duncan/acp-startup-connect-retry branch from b5422f8 to 363be1e Compare July 16, 2026 17:01
tokio-rustls wraps all rustls handshake failures (invalid/expired
certificate, hostname mismatch, protocol errors) as io::Error with the
rustls::Error as source. tokio-tungstenite then surfaces these as
Error::Io, not Error::Tls — so the prior Tls-only classification missed
them, letting deterministic cert validation failures burn all retry
attempts.

Split the Io arm: walk the error source chain via get_ref() + source();
if any link downcasts to rustls::Error, classify terminal. Plain
transport Io (reset, EOF, timeout) stays transient. Add
production-shaped test rows using the real Io-wrapped rustls error
shape, plus explicit transient rows for transport errors. Relabel
existing Tls rows as arm-pinning fixtures rather than production
reachability claims.
…ation

The previous has_rustls_source() treated any rustls::Error in the Io
source chain as terminal, but rustls includes ambiguous protocol,
decrypt, and server-alert failures that may be transient. Refactor to
is_terminal_rustls_io_error(): find the rustls::Error via the existing
get_ref-then-source walk, then match on a deterministic allowlist
(InvalidCertificate, InvalidCertRevocationList, NoCertificatesPresented,
UnsupportedNameType, PeerIncompatible). AlertReceived, DecryptError,
InappropriateMessage, and other ambiguous shapes stay transient under
the bounded retry budget.
@wpfleger96 wpfleger96 requested a review from wesbillman July 16, 2026 18:28
@wpfleger96 wpfleger96 merged commit 44359f6 into main Jul 16, 2026
32 checks passed
@wpfleger96 wpfleger96 deleted the duncan/acp-startup-connect-retry branch July 16, 2026 18:40
tlongwell-block pushed a commit that referenced this pull request Jul 16, 2026
Co-authored-by: npub12gtutshhh76rx0jx697f32f9tffd4hhp3hx58fp4x6u4uemkm7sqf8f757 <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@sprout-oss.stage.blox.sqprod.co>

Signed-off-by: npub12gtutshhh76rx0jx697f32f9tffd4hhp3hx58fp4x6u4uemkm7sqf8f757 <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@sprout-oss.stage.blox.sqprod.co>

* origin/main:
  feat(acp): retry initial relay connection with terminal/transient error classification (#1922)
  feat(mobile): add external release signing mode for central APK Signer pipelines (#1972)
  Guide CLI installation and subscription sign-in (#1980)
  Scope relay runtime state by community (#1658)
atishpatel added a commit that referenced this pull request Jul 16, 2026
…adata

* origin/main:
  fix(join-policy): require legal consent on hosted invites (#1987)
  [codex] Prevent actor-tag UI impersonation (#1931)
  chore(release): release Buzz Desktop version 0.4.9 (#1986)
  Restyle onboarding: branded landing screen, yellow/gradient backgrounds, new starter avatars (#1982)
  feat(acp): retry initial relay connection with terminal/transient error classification (#1922)
  feat(mobile): add external release signing mode for central APK Signer pipelines (#1972)
  Guide CLI installation and subscription sign-in (#1980)
  Scope relay runtime state by community (#1658)
  unify channel add + search into one entry point (#1964)
  Apply optional relay join policy across join flows (#1894)
  fix(desktop): preserve relaunch through mesh shutdown (#1966)
  Persist agent audiences with native inline mentions (#1949)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants