Setting .exists: false for area-competing technologies bugs if not all locations are included

[Adrien Mellot]

Problem description

For scenario creation I remove some technologies from locations using the .exists: false override. This works well, except when I try to remove techs which compete for resource_area (wind_onshore_competing and open_field_pv from euro-calliope in this case). For some reason, if I don’t remove the techs from ALL the locations in the model but only some of them, running the model returns the following error:

ERROR: Rule failed when generating expression for Constraint
    resource_area_capacity_per_loc_constraint with index BL_BS: KeyError: "Index
    '' is not valid for indexed component 'resource_area'"
ERROR: Constructing component 'resource_area_capacity_per_loc_constraint' from
    data=None failed: KeyError: "Index '' is not valid for indexed component
    ‘resource_area’"

I’ve tried different set ups and it works fine when I remove the technologies in question from either all locations or none at all, so the problem is really coming from setting the .exists:false to some locations. Also, the problem doesn’t arise for technologies which don’t compete for resource_area, I can set .exists: false for selected locations without any issue.

Full error log:

Calliope 0.6.10 starting at 2023-01-30 11:43:21

[2023-01-30 11:43:21] INFO     Model: initialising
[2023-01-30 11:43:22] INFO     Loading overrides from scenario: techs_removing_problem 
[2023-01-30 11:43:22] INFO     Applying the following overrides from scenario definition: `['config_2016_electricity_swiss']` 
[2023-01-30 11:43:22] INFO     
`locations.AG,AI_AR,BE,BL_BS,FR,GE,GL,GR,JU,LU,NE,NW_OW,SG,SH,SO,SZ,TG,TI,UR,VD,VS,ZG,ZH.techs.open_field_pv.exists`:False applied from override as new configuration
`locations.AG,AI_AR,BE,BL_BS,FR,GE,GL,GR,JU,LU,NE,NW_OW,SG,SH,SO,SZ,TG,TI,UR,VD,VS,ZG,ZH.techs.wind_offshore.exists`:False applied from override as new configuration
`locations.AG,AI_AR,BE,BL_BS,FR,GE,GL,GR,JU,LU,NE,NW_OW,SG,SH,SO,SZ,TG,TI,UR,VD,VS,ZG,ZH.techs.wind_onshore_competing.exists`:False applied from override as new configuration

[2023-01-30 11:43:22] INFO     Model: preprocessing stage 1 (model_run)
[2023-01-30 11:43:25] INFO     Model: preprocessing stage 2 (model_data)
[2023-01-30 11:43:27] INFO     Model: time resampling/clustering complete
[2023-01-30 11:43:28] INFO     Model: preprocessing complete
Model name:   Swiss-Calliope model
Model size:   27 locations, 73 technologies, 8 timesteps

Starting model run...
[2023-01-30 11:43:28] INFO     Backend: starting model run
[2023-01-30 11:43:28] INFO     Loading sets
[2023-01-30 11:43:29] INFO     Loading parameters
[2023-01-30 11:43:29] INFO     constraints are loaded in the following order: ['capacity', 'dispatch', 'policy', 'energy_balance', 'costs', 'network', 'conversion', 'group', 'conversion_plus', 'export', 'milp']
[2023-01-30 11:43:29] INFO     creating capacity constraints
[2023-01-30 11:43:29] ERROR    Rule failed when generating expression for Constraint resource_area_capacity_per_loc_constraint with index BL_BS:
KeyError: "Index '' is not valid for indexed component 'resource_area'"
[2023-01-30 11:43:29] ERROR    Constructing component 'resource_area_capacity_per_loc_constraint' from data=None failed:
KeyError: "Index '' is not valid for indexed component 'resource_area'"

Error in _validate_index, /Users/adrienmellot/opt/anaconda3/envs/calliope/lib/python3.8/site-packages/pyomo/core/base/indexed_component.py:788
"Index '' is not valid for indexed component 'resource_area'"

Steps to reproduce the problem

The model above has locations AG,AI_AR,BE,BL_BS,FR,GE,GL,GR,JU,LU,NE,NW_OW,SG,SH,SO,SZ,TG,TI,UR,VD,VS,ZG,ZH as well as 4 more locations (from which I do not remove the technologies wind_onshore_competing and open_field_pv using the override .exists: false).

Calliope version

v0.6.10

[Bryn Pickering]

Good catch @adrienmellot. Ideally we have a MWE based on the example models to work on a fix for this, but there is a workaround so it isn't high-priority to fix.

Workaround is to set the maximum technology capacities ("energy_cap_equals" / "resource_area_equals") to zero in your override, rather than removing the technology existence. It's less pretty, but gurobi will clean out those variables in pre-processing so shouldn't add much, if any optimisation overhead.