Shrink timed_belief value columns from float8 to float4

[Ahmad-Wahid]

Closes #2331.

What

Store timed_belief's cumulative_probability and event_value as single-precision floats (float4/real, 4 bytes) instead of the double-precision (float8, 8 bytes) that timely_beliefs' TimedBeliefDBMixin declares. These are the two per-row numeric value columns on what is typically the largest table, so halving their width shrinks it on disk by roughly 15-20%.

Single precision keeps ~7 significant decimal digits, which is plenty for sensor readings and for a cumulative probability in [0, 1].

Changes

• ORM (flexmeasures/data/models/time_series.py): override both columns on FlexMeasures' TimedBelief subclass with db.Float(precision=24) (→ real), preserving cumulative_probability's primary-key membership, nullable=False, and default=0.5. This keeps fresh installs and the create_all()-based test schema consistent with migrated databases (tests build the schema from the ORM, not from migrations).
• Migration (9f2c1d7b3a44_shrink_timed_belief_values_to_float4.py): alter the existing columns via op.alter_column — upgrade to real, downgrade back to double precision. float8↔float4 casts are implicit in PostgreSQL, so no USING clause is needed.
• Changelog: entry under Infrastructure / Support.

Notes / caveats

• ALTER COLUMN ... TYPE rewrites the whole table and rebuilds the primary key (since cumulative_probability is part of it), so on a large timed_belief this is a heavy, table-locking operation — run during a maintenance window. Documented in the migration docstring.
• Existing stored values are rounded to the nearest float4 on upgrade. Any use case genuinely needing >7 significant digits would lose precision.

Verification

• Confirmed the ORM columns compile to FLOAT(24) (→ Postgres real) with the primary key intact.
• Confirmed the migration's op.alter_column emits ALTER COLUMN ... TYPE FLOAT(24) on upgrade and ... TYPE FLOAT on downgrade.
• black + flake8 + mypy pass (pre-commit).

🤖 Generated with Claude Code

[read-the-docs-community]

Documentation build overview

📚 flexmeasures | 🛠️ Build #33850812 | 📁 Comparing 3a79d8e against latest (36be120)

  🔍 Preview build  

63 files changed · + 4 added · ± 59 modified

+ Added

• _autodoc/INFLEXIBLE_CONSUMPTION.html
• _autodoc/INFLEXIBLE_PRODUCTION.html
• _autodoc/OPERATION_MODES.html
• _autosummary/flexmeasures.data.schemas.scheduling.groups.html

± Modified

• changelog.html
• configuration.html
• genindex.html
• http-routingtable.html
• py-modindex.html
• _autodoc/COMMITMENTS.html
• _autodoc/GROUP.html
• _autodoc/INFLEXIBLE_DEVICE_SENSORS.html
• _autosummary/flexmeasures.api.common.responses.html
• _autosummary/flexmeasures.api.common.schemas.generic_schemas.html
• and 49 more...

[Flix6x]

Reviewed this and pushed the fixes directly to the branch (a219646) rather than leaving a list of asks — details below so you can check my work. The direction is right, and the dedupe fix in particular turned out to be load-bearing rather than defensive: without it, re-submitting a high-precision value raises a duplicate key violation on timed_belief_pkey. I added a test that fails without it.

Fixed

1. The branch had two Alembic heads. The merge of main brought in 3c2f9e5a1d47, which already revises 4b0f2e9c1a6d — and this migration pointed at 4b0f2e9c1a6d too. Repointed down_revision at 3c2f9e5a1d47; there is a single head again.

2. sum() over a float4 column accumulates in float4. _get_sensor_stats does sa.func.sum(TimedBelief.event_value). PostgreSQL's sum(real) both returns and accumulates in real, so once the running total passes 2^24 it stops registering small addends entirely — a sensor's "Sum over values" silently drops whole units. Now sums over an explicit float8 cast. (avg(real) already accumulates in float8, and min/max return an existing value, so sum was the only one affected.)

3. "Unchanged" meant two different things in one function. drop_unchanged_beliefs deduplicated within the incoming batch on raw values, while the comparison against the database rounded to float4. Pulled the rounding into a shared round_to_stored_precision() and used it in both places. The helper works on a copy, so the frame we return now keeps the caller's original values instead of the rounded ones — previously the returned bdf was derived from the mutated a_df.

4. Tests (there were none):

• both value columns are declared float4 — this is the one that catches a future timely_beliefs release quietly reclaiming these columns;
• round_to_stored_precision does not mutate its input;
• re-submitting a value needing >7 significant digits is dropped as unchanged (fails with UniqueViolation without the fix);
• the sensor-stats sum does not accumulate in float4. Both values used are exactly representable in float4, so the test isolates the accumulator from storage rounding.

5. Docs. Changelog now cites the PR rather than the issue (matching the surrounding entries), plus a release warning covering the table rewrite and the irreversibility. The migration docstring now says the downgrade restores the type, not the lost digits.

Verification

Beyond "the SQL compiles", I ran the upgrade and downgrade against a real PostgreSQL 15 table shaped like timed_belief:

• columns become real, and the primary key survives with its column order intact;
• NaN is preserved;
• 0.1 still round-trips exactly — PG ≥12 emits shortest-round-trip text for real and we use psycopg2, so short decimals are safe. Worth remembering that this would not hold under psycopg3's binary protocol, if a driver migration ever comes up;
• the downgrade demonstrates the irreversibility concretely: 0.1 comes back as 0.10000000149011612.

Test suites against PG 15: flexmeasures/data 248 passed, flexmeasures/api 441 passed. The two test_closest_sensor failures pre-date this branch.

One thing I did not fix, deliberately

Redeclaring these columns on the subclass works, but SQLAlchemy orders mixin columns by creation order, so it moves both to the front of the table and of the primary key — making the nearly-always-0.5 cumulative_probability the leading key column:

	primary key column order
migrated DB (04f0e2d2924a)	event_start, belief_horizon, cumulative_probability, sensor_id, source_id
create_all() before this PR	source_id, event_start, belief_horizon, cumulative_probability, sensor_id
create_all() with this PR	cumulative_probability, source_id, event_start, belief_horizon, sensor_id

You can see it in the failure output of the new dedupe test: Key (cumulative_probability, source_id, event_start, belief_horizon, sensor_id)=....

Some drift already existed (from the source_id declared_attr override), and this only affects schemas built by create_all() — deployments run flexmeasures db upgrade — so it is a tolerable interim. But it does mean the test suite exercises a different index shape than production.

The clean fix is upstream, so I opened SeitaBV/timely-beliefs#242: a value_column_type hook on TimedBeliefDBMixin, defaulting to today's Float. Measured, it keeps column and primary-key order byte-identical for the default type and a narrowed one alike. Once a tb release carries it, this override becomes:

value_column_type = db.Float(precision=24)

There is a quieter reason to prefer it too: a subclass-body override silently discards whatever tb declares on those columns, so a future server_default, check constraint or index there would vanish with no error and no failing test. I left a TODO in time_series.py pointing at the upstream PR.

Worth deciding before merge

The changelog warning now tells hosts to check their data first, but we can't check it for them, and "is float4 lossless for what we already store?" currently has no better answer than ad-hoc SQL. I filed SeitaBV/timely-beliefs#243 for an audit helper that reports the max relative error a float4 cast would introduce, per sensor. For our own deployments it'd be good to run that before shipping this — sensors holding cumulative meter readings or currency totals in the millions are the ones with no headroom at ~7 significant digits.

🤖 Generated with Claude Code