feat(acp): retry initial relay connection with terminal/transient error classification#1922
Merged
Merged
Conversation
wesbillman
reviewed
Jul 16, 2026
wesbillman
requested changes
Jul 16, 2026
wesbillman
left a comment
Collaborator
There was a problem hiding this comment.
One classification issue remains: deterministic TLS certificate/configuration failures are retried as transient. I left an inline comment requesting a split and regression coverage.
The retry loop and remaining exhaustive WebSocket classification otherwise look solid, and current CI is green. The branch is 22 commits behind main; rebase and rerun CI after the code fix before approval.
HarnessRelay::connect() previously made exactly one do_connect() attempt; any dropped handshake on a spotty link failed agent startup outright, requiring a manual respawn. Add retry_initial_connect(), which retries transient failures (ConnectionClosed, Timeout, NoAuthChallenge, etc.) with the same 1/2/4/8/16s jittered backoff already used by the post-start reconnect loop, now shared via STARTUP_CONNECT_BACKOFFS. Terminal errors (RelayError::Http, RelayError::AuthFailed) fail immediately since they're deterministic — retrying bad credentials or a malformed URL just delays surfacing a real problem. The startup_watermark capture point and post-start reconnect/replay logic are untouched; the retry loop is entirely inside connect(). Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
…errors is_terminal_connect_error() classified every WebSocket, Json, and UnexpectedMessage error as transient, including deterministic ones: a malformed relay_url (tungstenite Error::Url — bad scheme, missing host) never touches the network, and a Json/UnexpectedMessage parse failure reflects a protocol mismatch, not link noise. Both were burning the full retry budget on failures a later attempt can't fix. Classify Http/Json/ UnexpectedMessage as terminal and split WebSocket on Error::Url (terminal) vs everything else (transient, e.g. Io). AuthFailed was uniformly terminal, but the relay emits a NIP-01 error: prefix (crates/buzz-relay/src/handlers/auth.rs) for its own dependency faults (e.g. a ban-state DB lookup that couldn't run) — a real transient failure worth retrying, distinct from an explicit invalid:/auth-required:/ restricted:/blocked: rejection. Add is_terminal_auth_failure(), which treats only the error: prefix as transient and fails safe (terminal) on any unrecognized prefix. Also corrects STARTUP_CONNECT_BACKOFFS's doc comment: the initial-connect and reconnect loops share backoff *values*, not an identical schedule — the reconnect loop skips the sleep after its final attempt. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
…retry The Url-only terminal split inside is_terminal_connect_error() let deterministic handshake failures (non-101 HTTP, missing upgrade headers, accept-key mismatch) burn all six retry attempts. Replace with an exhaustive match on tungstenite::Error — no wildcard, so a tungstenite upgrade forces reclassification at compile time. Terminal: Url, Capacity, Utf8, HttpFormat, AttackAttempt, Http (non-408/ 429/5xx), and all Protocol variants except HandshakeIncomplete and ResetWithoutClosingHandshake. Transient: Io, ConnectionClosed, Tls, AlreadyClosed, WriteBufferFull, Protocol::HandshakeIncomplete, Protocol::ResetWithoutClosingHandshake, and Http 408/429/5xx.
WsError::Tls(_) was transient, burning all retry attempts on deterministic TLS failures (InvalidDnsName, expired/invalid certificates). On our rustls build, the only connect-time Tls errors are deterministic validation failures — transport-level TLS loss surfaces as Io (already transient). Flip Tls to terminal per Wes's review. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
b5422f8 to
363be1e
Compare
tokio-rustls wraps all rustls handshake failures (invalid/expired certificate, hostname mismatch, protocol errors) as io::Error with the rustls::Error as source. tokio-tungstenite then surfaces these as Error::Io, not Error::Tls — so the prior Tls-only classification missed them, letting deterministic cert validation failures burn all retry attempts. Split the Io arm: walk the error source chain via get_ref() + source(); if any link downcasts to rustls::Error, classify terminal. Plain transport Io (reset, EOF, timeout) stays transient. Add production-shaped test rows using the real Io-wrapped rustls error shape, plus explicit transient rows for transport errors. Relabel existing Tls rows as arm-pinning fixtures rather than production reachability claims.
…ation The previous has_rustls_source() treated any rustls::Error in the Io source chain as terminal, but rustls includes ambiguous protocol, decrypt, and server-alert failures that may be transient. Refactor to is_terminal_rustls_io_error(): find the rustls::Error via the existing get_ref-then-source walk, then match on a deterministic allowlist (InvalidCertificate, InvalidCertRevocationList, NoCertificatesPresented, UnsupportedNameType, PeerIncompatible). AlertReceived, DecryptError, InappropriateMessage, and other ambiguous shapes stay transient under the bounded retry budget.
tlongwell-block
pushed a commit
that referenced
this pull request
Jul 16, 2026
Co-authored-by: npub12gtutshhh76rx0jx697f32f9tffd4hhp3hx58fp4x6u4uemkm7sqf8f757 <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@sprout-oss.stage.blox.sqprod.co> Signed-off-by: npub12gtutshhh76rx0jx697f32f9tffd4hhp3hx58fp4x6u4uemkm7sqf8f757 <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@sprout-oss.stage.blox.sqprod.co> * origin/main: feat(acp): retry initial relay connection with terminal/transient error classification (#1922) feat(mobile): add external release signing mode for central APK Signer pipelines (#1972) Guide CLI installation and subscription sign-in (#1980) Scope relay runtime state by community (#1658)
atishpatel
added a commit
that referenced
this pull request
Jul 16, 2026
…adata * origin/main: fix(join-policy): require legal consent on hosted invites (#1987) [codex] Prevent actor-tag UI impersonation (#1931) chore(release): release Buzz Desktop version 0.4.9 (#1986) Restyle onboarding: branded landing screen, yellow/gradient backgrounds, new starter avatars (#1982) feat(acp): retry initial relay connection with terminal/transient error classification (#1922) feat(mobile): add external release signing mode for central APK Signer pipelines (#1972) Guide CLI installation and subscription sign-in (#1980) Scope relay runtime state by community (#1658) unify channel add + search into one entry point (#1964) Apply optional relay join policy across join flows (#1894) fix(desktop): preserve relaunch through mesh shutdown (#1966) Persist agent audiences with native inline mentions (#1949)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ACP agent startup previously made exactly one relay connection attempt — any dropped WebSocket handshake on a spotty link failed the agent outright, requiring a manual respawn.
retry_initial_connect()retries transient failures with jittered backoff (1 immediate + 5 delayed attempts at 1/2/4/8/16s base intervals), sharing the delay schedule already used by the post-start reconnect loop viaSTARTUP_CONNECT_BACKOFFS. Terminal errors fail immediately since retrying them is pointless.Classification contract
is_terminal_connect_error()classifies everyRelayErrorvariant, andis_terminal_ws_error()exhaustively classifies everytungstenite::Errorinner variant (no wildcard — a tungstenite upgrade forces reclassification at compile time).Terminal (fail fast):
RelayError::Http— HTTP-level rejectionRelayError::Json/RelayError::UnexpectedMessage— protocol mismatchWebSocket(Url(_))— deterministic URL/scheme misconfiguration (never reaches the network)WebSocket(Capacity(_))/WebSocket(Utf8(_))/WebSocket(HttpFormat(_))/WebSocket(AttackAttempt)— deterministic handshake-shape failuresWebSocket(Http(resp))— non-101 HTTP response, unless status is408,429, or5xx(server-side transient conditions that can recover)WebSocket(Protocol(_))— all variants (wrong method/version, missing/invalid upgrade headers, accept-key mismatch, junk-after-request, etc.) exceptHandshakeIncompleteandResetWithoutClosingHandshake(connection dropped mid-handshake or abrupt reset — link-level failures)WebSocket(Tls(_))— deterministic TLS config failures. On our rustls build the only connect-timeTlsvariant isInvalidDnsNameWebSocket(Io(_))with a deterministicrustls::Errorin the source chain —tokio-rustlswraps rustls handshake failures asio::Errorwith therustls::Erroras source;tokio-tungstenitesurfaces these asError::Io. Classifier walksget_ref()+source()chain to find therustls::Error, then matches a terminal allowlist:InvalidCertificate(_),InvalidCertRevocationList(_),NoCertificatesPresented,UnsupportedNameType,PeerIncompatible(_)— deterministic cert/config/incompatibility failures that retry cannot fix. Relies on a single rustls version in the dep tree (0.23.40)AuthFailedwithinvalid:/auth-required:/restricted:/blocked:prefix, or any unrecognized prefix (fail-safe)Transient (retry):
WebSocket(Io(_))without a rustls source, or with an ambiguous rustls error (AlertReceived(_),InappropriateMessage/InappropriateHandshakeMessage,InvalidMessage(_),DecryptError,PeerMisbehaved(_),General(_), and any other non-allowlisted variant —rustls::Erroris#[non_exhaustive], so unknowns default transient) — plain transport failures (reset, EOF, timeout, refused, mid-handshake TLS transport loss) and ambiguous protocol/decrypt/server-alert shapes stay retryable under the bounded budgetWebSocket(ConnectionClosed)— link-level closureWebSocket(AlreadyClosed)/WebSocket(WriteBufferFull(_))— unreachable duringconnect_async; kept fail-safe transientWebSocket(Http(resp))where status is408,429, or5xxWebSocket(Protocol::HandshakeIncomplete)/WebSocket(Protocol::ResetWithoutClosingHandshake)RelayError::ConnectionClosed/Timeout/NoAuthChallengeAuthFailedwitherror:prefix — relay dependency fault (e.g., ban-state DB lookup failure), distinct from an explicit credential rejectionChanges
retry_initial_connect()with the jittered backoff scheduleis_terminal_connect_error()that classifies everyRelayErrorvariantis_terminal_ws_error()with exhaustive match on everytungstenite::Errorvariant includingHttpstatus split and per-Protocol-variant classificationis_terminal_rustls_io_error()— walksio::Errorinner error + source chain to find arustls::Errorwrapped asIobytokio-rustls/tokio-tungstenite, then matches a deterministic cert/config/incompatibility allowlist (InvalidCertificate,InvalidCertRevocationList,NoCertificatesPresented,UnsupportedNameType,PeerIncompatible); ambiguous protocol/decrypt/server-alert shapes (AlertReceived,DecryptError,InappropriateMessage,General, etc.) default transient under the bounded retry budgetis_terminal_auth_failure()that classifies AUTH rejection text by NIP-01 prefix, cross-referenced againstcrates/buzz-relay/src/handlers/auth.rsSTARTUP_CONNECT_BACKOFFSshared between startup and reconnect loopsRelayErrorvariant and everytungstenite::Errorinner variant, including production-shapedIo-wrapped rustls errors terminal on the allowlist (InvalidCertificate(Expired),InvalidCertificate(NotValidForName),NoCertificatesPresented), transient on ambiguous variants (General,AlertReceived(InternalError),DecryptError), explicit transport-transient rows (ConnectionReset,UnexpectedEof,TimedOut), andTlsarm-pinning rows (InvalidDnsName,CertificateError::Expired)do_connect_wrong_scheme_is_terminaltest driving a literalhttps://URL through productiondo_connect()error:-prefixedAuthFailedretry/recovery, terminal error fail-fast, exhaustion, and backoff sleep behavior