Skip to content

fix(worker): prevent OperationalError from expired ORM objects in UploadFinisher - #1866

Open
sentry[bot] wants to merge 1 commit into
mainfrom
seer/fix/worker-upload-finisher-db-error
Open

fix(worker): prevent OperationalError from expired ORM objects in UploadFinisher#1866
sentry[bot] wants to merge 1 commit into
mainfrom
seer/fix/worker-upload-finisher-db-error

Conversation

@sentry

@sentry sentry Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

This PR addresses an OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly occurring in app.tasks.upload.UploadFinisher.

Root Cause:
The _handle_finisher_lock method in UploadFinisher performs significant work, including finish_reports_processing. After this, it calls db_session.commit() (line 597) to update the repository's updatestamp. Due to SQLAlchemy's default expire_on_commit=True behavior, all ORM objects in the session, including the commit object, are expired at this point.

Immediately after the commit, invalidate_caches is called (line 609). Inside invalidate_caches, accessing attributes like commit.repoid triggers a lazy-load from the database. If the PostgreSQL connection has been closed by the server (e.g., due to an idle timeout during the preceding long-running tasks), this lazy-load attempt results in the OperationalError.

Solution:
To prevent accessing expired ORM objects on a potentially closed database connection, the necessary commit and repository attributes (such as repoid, branch, commitid, repository.service, repository.author.username, repository.name, repository.branch) are now eagerly read and stored into local variables before db_session.commit() is called. The invalidate_caches method has been refactored to accept these primitive values directly, eliminating any post-commit ORM attribute access that could trigger a lazy-load.

Legal Boilerplate

Look, I get it. The entity doing business as "Codecov" is owned by Harness, Inc. In 2026 Harness acquired Codecov and as a result Harness is going to need some rights from me in order to utilize my contributions in this PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Harness can use, modify, copy, and redistribute my contributions, under Harness's choice of terms.

Fixes WORKER-Z5D

@codecov-notifications

codecov-notifications Bot commented Jul 31, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@codecov

codecov Bot commented Jul 31, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.83%. Comparing base (d8942db) to head (7e5da00).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1866   +/-   ##
=======================================
  Coverage   91.83%   91.83%           
=======================================
  Files        1332     1332           
  Lines       51830    51836    +6     
  Branches     1647     1647           
=======================================
+ Hits        47598    47604    +6     
  Misses       3911     3911           
  Partials      321      321           
Flag Coverage Δ
workerintegration 58.53% <100.00%> (+0.01%) ⬆️
workerunit 90.40% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants