model.backend.update_parameter does not propagate in all equations
FLomb opened this issue · comments
What happened?
The functionality to update backend parameters does not seem to work as expected. If updating a given parameter, the parameter itself does end up being updated in the backend. However, the update does not seem to propagate to the equations using the parameter, which all remain fixated on the old value.
As an example, after building and solving the national-scale example model, one can try:
model.backend.update_parameter('cost_flow_cap', 0)
.
This will set all investment costs in capacity to a null value across all techs. If one inspects the parameters in the backend, one can see that the operation works smoothly and as expected.
Then, if re-running the model via model.solve(force=True)
, as suggested in the docs after updating a parameter, the solution does not reflect the expectations. In fact, model.results.cost_investment.to_series().dropna()
(or similar ways to inspect the outputs, including from the backend) will show that investment costs are still there for all technologies, including technologies that had no other investment cost than the one for flow_cap
. In other words, the updating of the parameter does not seem to propagate across the equations of the Pyomo model.
This was not the case in v0.6, so something must have gone wrong in the reformulation of our way to interface with Pyomo
Which operating systems have you used?
- macOS
- Windows
- Linux
Version
v0.7.0dev3
Relevant log output
No response
An update on this after additional debugging:
For the same example as above, if one tries model.backend.update_parameter('cost_flow_cap', 0)
, the result has the problem we discussed; the parameter does get updated, but the equations do not.
However, if one tries model.backend.update_parameter('cost_flow_cap', model.inputs.flow_cap_max*0)
, the parameter gets updated identically, but this time, it propagates as expected.
Edit: I was partially wrong on this one. The configuration is the only duplicated thing. Model.backend.inputs is a shallow copy. This should still be avoided, but it's not related to this issue!
However, if one tries
model.backend.update_parameter('cost_flow_cap', model.inputs.flow_cap_max*0)
, the parameter gets updated identically, but this time, it propagates as expected.
That's very interesting! This smells like a pointer / memory allocation issue, similar to #608
We are multiplying the model data in several locations, which should not be the case, as it can lead to sync problems!.
model._model_data # xarray object
model._model_data.attrs # model math and config
model.config # copy of model config
model.math # copy of model math
model.backend.inputs # copy of xarray object
model.backend.inputs.attrs # copy of model math and config
In particular, the backend completely duplicates the model configuration, while the math documentation does not:
id(model._model_data) # id gives us the location in memory of a variable
140507415076176
id(model.math_documentation.inputs)
140507415076176
id(model.backend.inputs) # Duplication!!!
140507414858960
This isn't an issue of how we manage the input data @irm-codebase. It's about how cross-referencing works between components of the backend. If you update a parameter, it has to know what else was reliant on it and then propagate those changes through the backend model.
I'm unable to reproduce your problem @FLomb. This is what I'm running:
import calliope
m = calliope.examples.national_scale()
m.build()
m.solve()
print(m.results.cost_investment_flow_cap.to_series().dropna())
m.backend.update_parameter('cost_flow_cap', 0)
m.solve(force=True)
print(m.results.cost_investment_flow_cap.to_series().dropna())
Produces:
Out:
nodes techs carriers costs
region1 ccgt power monetary 22500000.00
region1_to_region2 power monetary 323047.29
region1_1 csp power monetary 10000000.00
region1_2 csp power monetary 0.00
region1_3 csp power monetary 2533762.10
region2 region1_to_region2 power monetary 323047.29
Name: cost_investment_flow_cap, dtype: float64
nodes techs carriers costs
region1 ccgt power monetary 0.0
demand_power power monetary 0.0
region1_to_region1_1 power monetary 0.0
region1_to_region1_2 power monetary 0.0
region1_to_region1_3 power monetary 0.0
region1_to_region2 power monetary 0.0
region1_1 csp power monetary 0.0
region1_to_region1_1 power monetary 0.0
region1_2 csp power monetary 0.0
region1_to_region1_2 power monetary 0.0
region1_3 csp power monetary 0.0
region1_to_region1_3 power monetary 0.0
region2 battery power monetary 0.0
demand_power power monetary 0.0
region1_to_region2 power monetary 0.0
Name: cost_investment_flow_cap, dtype: float64
I tested the branch based on your PR #623, and that fixes the issue