Skip to content

fix(worker): fail open when timeseries DB connection fails - #1869

Open
sentry[bot] wants to merge 1 commit into
mainfrom
seer/fix/worker-timeseries-db-resilience
Open

fix(worker): fail open when timeseries DB connection fails#1869
sentry[bot] wants to merge 1 commit into
mainfrom
seer/fix/worker-timeseries-db-resilience

Conversation

@sentry

@sentry sentry Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

This PR addresses OperationalError: (psycopg2.OperationalError) SSL SYSCALL error: EOF detected occurring in app.tasks.upload.UploadFinisher and other timeseries-related tasks.

Root Cause:
The error was traced to transient infrastructure disruptions (e.g., restarts, failovers) of the TimescaleDB instance (prod-timescaledb.codecov.dev). When repository_datasets_query was called, it attempted to establish a new database connection. During this connection setup, SQLAlchemy's on_connect hook would execute a pg_type query, but the SSL connection was abruptly terminated by the server (EOF) before completion.

Problem:
Since the timeseries data collection is a non-critical, optional step, a transient TimescaleDB unavailability should not cause the entire UploadFinisher task (or other related tasks) to fail.

Solution:
Implemented try/except blocks around calls to repository_datasets_query in the following files:

  • apps/worker/tasks/upload_finisher.py
  • apps/worker/tasks/save_commit_measurements.py
  • apps/worker/services/bundle_analysis/report.py

If an OperationalError (or any Exception) occurs during the timeseries dataset query, a warning is logged, and the task proceeds without attempting to save timeseries measurements. This ensures that critical worker tasks can complete successfully even if the timeseries database experiences momentary unavailability, making the system more resilient.

Legal Boilerplate

Look, I get it. The entity doing business as "Codecov" is owned by Harness, Inc. In 2026 Harness acquired Codecov and as a result Harness is going to need some rights from me in order to utilize my contributions in this PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Harness can use, modify, copy, and redistribute my contributions, under Harness's choice of terms.

Fixes WORKER-Z5F

@codecov-notifications

codecov-notifications Bot commented Jul 31, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 26.66667% with 11 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
apps/worker/tasks/save_commit_measurements.py 0.00% 5 Missing ⚠️
apps/worker/services/bundle_analysis/report.py 40.00% 3 Missing ⚠️
apps/worker/tasks/upload_finisher.py 40.00% 3 Missing ⚠️

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@codecov

codecov Bot commented Jul 31, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 26.66667% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.81%. Comparing base (d8942db) to head (2ce5242).
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
apps/worker/tasks/save_commit_measurements.py 0.00% 5 Missing ⚠️
apps/worker/services/bundle_analysis/report.py 40.00% 3 Missing ⚠️
apps/worker/tasks/upload_finisher.py 40.00% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1869      +/-   ##
==========================================
- Coverage   91.83%   91.81%   -0.02%     
==========================================
  Files        1332     1332              
  Lines       51830    51842      +12     
  Branches     1647     1647              
==========================================
+ Hits        47598    47600       +2     
- Misses       3911     3921      +10     
  Partials      321      321              
Flag Coverage Δ
workerintegration 58.47% <0.00%> (-0.05%) ⬇️
workerunit 90.34% <26.66%> (-0.06%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants