fix(filler): crash instead of idling when a rejection or worker death kills the reader#95
Open
robrigo wants to merge 1 commit into
Open
fix(filler): crash instead of idling when a rejection or worker death kills the reader#95robrigo wants to merge 1 commit into
robrigo wants to merge 1 commit into
Conversation
… kills the reader The process-level unhandledRejection/uncaughtException handlers logged and swallowed, so a startup rejection (a statement timeout in AtomicAssetsHandler.init on a cold cache) left the process alive with the reader never started - Kubernetes saw nothing to restart while the filler fell behind at chain rate. The primary had the same hole one level up: it only exits on a worker's explicit failure message, so a worker killed by its own handler left the primary running with zero readers and a green health check. Both now log and exit non-zero so the pod restarts; a SIGTERM-initiated shutdown is exempted from the worker-exit escalation. The API server keeps its swallowing handlers - a stray rejection there should not take the read API down.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A statement timeout during AtomicAssetsHandler.init on a cold cache surfaced as an unhandled rejection, and the filler's process-level handlers logged it and stayed up. The process was alive, the reader never started, and Kubernetes had no crashed process to restart, so the filler fell behind at chain rate until a manual pod bounce. The reader stall watchdog never fires for a dead init, and the primary only exits on a worker's explicit failure message, so the same zombie shape existed one level up: a worker dying without that message left the primary running with zero readers and a green /healthc.
Both levels now log and exit non-zero so Kubernetes restarts the pod. The cluster exit listener exempts SIGTERM-initiated shutdowns, and the API server keeps its separate swallowing handlers since a stray rejection there should not take the whole read API down.
Validation
check-types, lint, and the suite are clean: 262 passing, 37 pending, identical to the unmodified baseline (no test exercises bin/filler.ts; the repo has no scaffolding for process-level handler tests and none was invented). The same fix is cherry-picked onto release/1.7 in the 1.7.19 hotfix PR.