Add a Pyomo-free scheduling backend (direct highspy model construction) - #2364
Draft
Flix6x wants to merge 8 commits into
Draft
Add a Pyomo-free scheduling backend (direct highspy model construction)#2364Flix6x wants to merge 8 commits into
Flix6x wants to merge 8 commits into
Conversation
Add flexmeasures/data/models/planning/highspy_optimization.py, which builds the device scheduler's LP/MILP directly with the HiGHS Python API (highspy), bypassing Pyomo's model construction and solution-ingestion overhead. The Pyomo implementation (linear_optimization.device_scheduler) remains the semantic reference; the module docstring prominently documents that the two models must be kept in sync, as well as the (verified) deviations: - ems_flow_commitment_equalities are not built: on the Pyomo path they are bound-less, i.e. free rows without any effect on the solution - rows no finite assignment can satisfy (bounds involving +/-inf quantities) are skipped, mirroring HiGHS rejecting such rows when appsi adds them - solver results and model are lightweight shims exposing the attributes callers consume (termination_condition/status strings, commitment_costs, commodity_costs, costs, and indexed variable views) The model is built with vectorized numpy arrays (addVars/addRows/ changeColsCost/changeColsIntegrality), constructing and solving typical battery problems in milliseconds, where the Pyomo layer needs seconds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WtuVTVfL4fQ9QSqbLXmAGD Signed-off-by: F.N. Claessen <claessen@seita.nl>
When FLEXMEASURES_LP_SOLVER is set to "highspy", device_scheduler delegates to device_scheduler_highspy with the same inputs and the same return contract. All other solver names keep using the Pyomo path unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WtuVTVfL4fQ9QSqbLXmAGD Signed-off-by: F.N. Claessen <claessen@seita.nl>
- Add "highspy" to the app_with_each_solver fixture params, so tests using it also run against the direct backend. - Add test_highspy_equivalence.py, running representative scenarios (battery with prices; soc targets incl. storage efficiency and stock delta; site capacity with breach and peak prices; two devices with a StockCommitment; an infeasible case) through both appsi_highs and highspy, asserting near-identical schedules and costs, and matching termination handling. - Make test case 2 of test_multiple_devices_simultaneous_scheduler assert solver-independent properties (aggregate schedule, total costs, total unmet demand): the problem has multiple optima, and only the site-level schedule is unique, while the per-device slot allocation is an arbitrary tie-break that depends on the solver backend. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WtuVTVfL4fQ9QSqbLXmAGD Signed-off-by: F.N. Claessen <claessen@seita.nl>
Flip the FLEXMEASURES_LP_SOLVER default from "appsi_highs" to "highspy" and update the configuration, installation and deployment docs accordingly. Any Pyomo-based solver remains available as before. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WtuVTVfL4fQ9QSqbLXmAGD Signed-off-by: F.N. Claessen <claessen@seita.nl>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WtuVTVfL4fQ9QSqbLXmAGD Signed-off-by: F.N. Claessen <claessen@seita.nl>
Documentation build overview
10 files changed ·
|
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new scheduling backend that bypasses Pyomo and constructs the equivalent optimization model directly with highspy (HiGHS), then makes it selectable via FLEXMEASURES_LP_SOLVER="highspy" and flips that to the default. This aims to substantially reduce scheduling-job overhead dominated by Pyomo model construction / ingestion.
Changes:
- Introduce
flexmeasures/data/models/planning/highspy_optimization.pyimplementing the device scheduler model directly in HiGHS (with lightweight result/model shims). - Add solver dispatch in
device_scheduler, expand the solver-parametrized planning test fixture, and add equivalence tests comparingappsi_highsvshighspy. - Update docs + changelog to reflect the new default solver backend and installation/deployment guidance.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| flexmeasures/utils/config_defaults.py | Switch default LP solver config to highspy |
| flexmeasures/data/models/planning/linear_optimization.py | Dispatch to the new highspy backend when configured |
| flexmeasures/data/models/planning/highspy_optimization.py | New direct-HiGHS scheduler model implementation + result/model shims |
| flexmeasures/data/models/planning/tests/conftest.py | Run planning tests under appsi_highs, cbc, and highspy |
| flexmeasures/data/models/planning/tests/test_solver.py | Make a previously solver-tie-broken test assert solver-independent properties |
| flexmeasures/data/models/planning/tests/test_highspy_equivalence.py | New scenario-based equivalence tests for appsi_highs vs highspy |
| documentation/configuration.rst | Document the backend split (highspy direct vs Pyomo solver interfaces) and new default |
| documentation/host/installation.rst | Update installation guidance to reflect highspy as the default backend |
| documentation/host/deployment.rst | Update deployment guidance for solver installation under the new default |
| documentation/changelog.rst | Add changelog entry announcing the new backend and default change |
Hoist convert_commitments_to_subcommitments to module level (it is solver-agnostic and closure-free) and share it between both scheduler backends, removing the duplicated copy from the highspy module. Splitting a commitment into per-group subcommitments now uses a single groupby pass (in order of first appearance, like pd.unique) instead of filtering the DataFrame once per group, and the price non-uniqueness checks are vectorized across all groups. This removes a cost that scaled quadratically with the number of time steps (each time step often forms its own group), benefiting both backends: a 2-device, 192-step benchmark drops from 0.47s to 0.28s via appsi_highs and from 0.37s to 0.12s via highspy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WtuVTVfL4fQ9QSqbLXmAGD Signed-off-by: F.N. Claessen <claessen@seita.nl>
- Do not disable HiGHS output unconditionally in the direct backend; only force output_flag false when LOGGING_LEVEL is "INFO", mirroring exactly how the Pyomo path builds its solver options profile, so verbose logging modes can still see solver output. Operator-configured FLEXMEASURES_LP_SOLVER_OPTIONS are still applied last and can override. - Clarify in the configuration docs that a separate solver installation is only needed for external solvers such as cbc; both HiGHS-based choices (highspy and appsi_highs) rely on the bundled highspy package. - Remove the now-inconsistent "pip install highspy" instruction from the deployment docs, which already state that highspy ships with FlexMeasures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WtuVTVfL4fQ9QSqbLXmAGD Signed-off-by: F.N. Claessen <claessen@seita.nl>
Comment on lines
+352
to
+362
| # commodity -> set(device indices) | ||
| commodity_devices: dict = {} | ||
| for df in commitments: | ||
| if "commodity" not in df.columns or "device" not in df.columns: | ||
| continue | ||
| for _, row in df[["commodity", "device"]].dropna().iterrows(): | ||
| devices = row["device"] | ||
| if not isinstance(devices, (list, tuple, set)): | ||
| devices = [devices] | ||
| commodity_devices.setdefault(row["commodity"], set()).update(devices) | ||
|
|
The commodity -> device indices lookup was only consumed by the Pyomo path's ems_flow_commitment_equalities, which the direct backend deliberately does not build (they are free rows). A breadcrumb comment keeps pointing readers at that documented skip. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WtuVTVfL4fQ9QSqbLXmAGD Signed-off-by: F.N. Claessen <claessen@seita.nl>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Building the scheduling model through Pyomo dominates scheduling-job time: profiled on a large co-simulation database, the Pyomo layer (expression-tree construction + appsi ingestion) accounted for ~1 s of a battery-only job and ~7 s of a two-device job with operation-mode power bands, while the solve itself finishes in seconds and belief reads are negligible (~0.3 s). A prototype building the identical battery model via highspy's array API lands at ~5 ms.
This PR adds a Pyomo-free backend that constructs the HiGHS model directly:
flexmeasures/data/models/planning/highspy_optimization.py— a faithful reimplementation ofdevice_scheduler's model semantics (device power up/down with efficiencies, stock dynamics incl. storage efficiency / stock delta / gain / usage, stock bounds and targets, device and EMS power capacities, the commitments framework incl. grouped equalities and subcommitments) using vectorized numpy + highspy. The module docstring prominently notes the two-models-in-sync maintenance trade-off and points atdevice_scheduleras the semantic reference.FLEXMEASURES_LP_SOLVER = "highspy".device_schedulerdispatches to the new module for that value; every other value keeps the Pyomo path untouched. Same inputs, same 4-tuple return contract (results/model shims cover every consumer usage:termination_condition,commitment_costs, indexed power views).app_with_each_solverfixture now runs["appsi_highs", "cbc", "highspy"], and a newtest_highspy_equivalence.pyasserts near-identical schedules across both HiGHS paths on five representative scenarios (battery+prices, soc-targets with storage efficiency and stock delta, site capacity with breach/peak prices, two devices with a StockCommitment, and an infeasible case).highspy(config_defaults), with docs updated (configuration, deployment, installation).Local verification: planning suite passes under all three solver params (263 passed × both HiGHS backends; cbc installed, nothing skipped), scheduling job tests and API scheduling tests pass under the new default, mypy/black/flake8 clean.
Discussion points (deliberate deviations, called out for review)
ems_flow_commitment_equalitiesis not built on the highspy path: on the Pyomo path it produces a bound-less (vacuous) constraint, and rows whose bounds involve ±inf commitment quantities are rejected by HiGHS when appsi adds them — so the net effect on main is identical. Documented in the module docstring.test_multiple_devices_simultaneous_schedulercase 2 asserted one specific optimum of a problem whose per-device allocation is non-unique (same total cost either way); it now asserts solver-independent properties (aggregate schedule, total cost, total unmet demand). Case 1 (unique optimum) keeps exact assertions.pd.concat([])).device_schedulercall, with identical costs. The largest wins appear on models with many constraint rows (e.g. the operation-mode power-band branch, where the Pyomo layer costs ~7 s/job).time_limitviaFLEXMEASURES_LP_SOLVER_OPTIONS(applied last on both paths).How to test
pytest flexmeasures/data/models/planning/tests(runs all three solver params), or setFLEXMEASURES_LP_SOLVER="appsi_highs"to keep the previous default behavior.🤖 Generated with Claude Code
https://claude.ai/code/session_01WtuVTVfL4fQ9QSqbLXmAGD