Fix race condition creating duplicate data sources on fresh databases

[Flix6x]

Symptom

On a fresh database, running several concurrent scheduling jobs for the first time (e.g. 10 RQ workers all triggering StorageScheduler schedules) can leave the data_source table with two identical rows (same name, type, model and version). From that moment on, every subsequent scheduling job fails permanently with:

sqlalchemy.exc.MultipleResultsFound: Multiple rows were found when one or none was required

because the get-or-create lookup uses .scalar_one_or_none().

Race mechanism

Two workers race through the get-or-create logic (get_data_source in flexmeasures/data/utils.py and get_or_create_source in flexmeasures/data/services/data_sources.py):

1. Worker A looks up the scheduler's source → not found.
2. Worker B looks up the same source → not found (A hasn't committed yet).
3. Both insert → two rows.

One would expect the DB to reject the second insert: data_source has a UniqueConstraint("name", "user_id", "account_id", "model", "version", "attributes_hash"). But PostgreSQL treats NULLs as distinct in unique constraints, and scheduler/forecaster sources have NULL user_id and account_id (and, on the get_data_source path, NULL attributes_hash). So for exactly these sources the constraint never fires, and the duplicate insert succeeds silently.

Why fresh installs are affected

On a long-lived database the source row already exists, so the lookup always succeeds and the insert path is never raced. The bug bites exactly where the source does not exist yet: fresh installs, CI environments, and new deployments that start with concurrent scheduling load.

The fix (3 layers)

1. DB-level uniqueness that actually holds (migration c9d4f7a21e0b): replaces the NULL-blind unique constraint with a NULL-safe unique expression index, coalescing NULLs to sentinel values that cannot occur in real data (-1 for ids, '' for model/version, empty bytes for the attributes hash). The migration first deduplicates any existing duplicate rows: the lowest id is kept, and timed_belief, annotation and (if present) legacy power/price/weather rows are repointed to it, dropping only beliefs that collide on their full primary key with a belief of the kept source (i.e. duplicate recordings by the same logical source).
2. Race-safe get-or-create: both code paths now insert within a SAVEPOINT (session.begin_nested()) and, on IntegrityError, roll back to the savepoint and re-fetch the winning row — the standard race-safe get-or-create, which also keeps the enclosing transaction usable.
3. Defense in depth for wedged databases: lookups that expect one source now deterministically return the row with the lowest id (with a logged warning) when multiple rows match, instead of raising MultipleResultsFound. This lets databases that already contain duplicates (created before this fix, or rows differing only in fields the caller didn't filter on, such as attributes) degrade gracefully instead of failing every job.

Test coverage

New tests in flexmeasures/data/tests/test_data_source.py:

• test_get_or_create_source_survives_insert_race: forces the losing worker's code path (initial lookup misses while the row exists) and asserts the IntegrityError path re-fetches the winner and no duplicate is created.
• test_source_lookups_tolerate_duplicates: near-duplicate sources (differing only in attributes) no longer crash get_or_create_source / get_data_source lookups that don't filter on attributes; the oldest row is returned.
• test_exact_duplicate_sources_rejected_by_db: the NULL-safe unique index rejects exact duplicates with NULL key columns (the shape the old constraint let through).

Ran pytest flexmeasures/data/tests/test_data_source.py (19 passed) plus test_queries.py, test_scheduling_jobs.py, test_scheduling_repeated_jobs.py and test_utils.py (52 passed) against a sandboxed Postgres.

The migration was validated end-to-end on a fresh database (flexmeasures db upgrade), seeded at the previous head with exact-duplicate scheduler sources plus colliding and non-colliding timed_belief, annotation and legacy power rows: duplicates were removed, references repointed as intended, and a downgrade/upgrade roundtrip restores the old constraint and back.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WtuVTVfL4fQ9QSqbLXmAGD

[read-the-docs-community]

Documentation build overview

📚 flexmeasures | 🛠️ Build #33775936 | 📁 Comparing eeeb007 against latest (a3fb811)

  🔍 Preview build  

4 files changed

± changelog.html
± genindex.html
± _autosummary/flexmeasures.data.services.data_sources.html
± api/v3_0.html