Declarative YAML math interface

[brynpickering]

Closes #561

This started out as a port from Calliope text math backend, revised to plug into the linopy model object.

After the initial port, refactoring and cleaning of the modules was supported by Clause Code (see per-commit attributions).

Changes proposed in this Pull Request

• New linear expression container in the Linopy model object, to store expressions that may be used in multiple constraints (or are just useful "effects" to query the solution of after running the model)
• New declarative math directory with:
  • pydantic schema for YAML/dict math input
  • opinionated parsing grammar, parsed by pyparsing for safe AST generation and evaluation
  • linopy model builder using combination of parsed grammar and input xarray dataset
  • math documentation builder, using a latex math generator that lives alongside the linopy model component generator. It can render to a md/rst/latex documentation page. Since the equations can become very long and automatic line breaks in latex math are a pain to maintain, this only really works for online docs with horizontal scrolling.
  • data checker section independent of linopy model component object builder, to validate combinations of input data in the context of the given optimisation problem
  • "parameter", "lookup" and "dimension" yaml math sections, to record expected input data and dimensionality and to help the math expression parsing (as we don't need to guess what is a plain string and what is an expected input)

Open questions

• Do we want to rename lookups to something else? They are any input arrays that are going to be used to mask math arrays or group parameters. Unlike parameters, which must be numeric as they will be coefficients of the math expressions, they can be a range of types (string, bool, int, etc.)
• Should this live here as a directory of the core code or live in a separate repo?
• All arithmetic joins are assumed to be outer. Do we need some additional custom grammar to handle different join types (e.g. |+ or inner(left, right), etc.)?
• Plain strings could take the same name as arrays in a helper function. To disambiguate, should we therefore add parsing grammar for plain strings (e.g. ' either side) or for arrays (e.g. ``` either side or prepended with @), as is used in `pandas` query grammar?
• Do we need slices? It might be better to just reference slicing arrays directly in the square brackets. This will require disambiguation between strings and arrays (see above)
• Should we have sub expressions at all? They add complexity to the YAML but also allow us to make more efficient use of our data containers, by storing different cases in the same array when they share the same dimensions and have mutually exclusive masking. They also reduce the number of named expressions/constraints which might be easier on the user and in the math docs.
• How could we allow the user to supply inputs as entirely dense arrays, with no NaNs? Currently, all input data arrays are expected to all share the same dimensions (they are part of the same xarray dataset) but that could lead to a lot of bloat in memory for the inputs alone.
• Should we allow the user to supply a mapping of datasets, perhaps each with the same dimension names but not necessarily the same dimension members? How would they refer to these different input datasets in the YAML? E.g. generators.capacity_max to get data from the generators input array? Then what if we want to get data from some combination of input datasets? I'm hoping @coroa has some ideas here.

TODOs

• Add dims section to "parameter" and "lookup" sections to define the maximum dimensionality of input data. Will be used in input data validation.
• Add docs
• Add piecewise constraint generation
• Handle arbitrary, parsable masking strings in the mask helper function. Currently it can only handle refs to boolean arrays already defined in the inputs
• Clean up iterable globals and typevars; there are too many that are subsets of each other right now
• Validate against one_of in lookups. Currently included in schema but not used later.
• Automatically decide on expression order based on usage, rather than require order to be defined by the user. Since expressions are generated in the order they're defined, there can be cases of user error / YAML file merge error that leads to one expression being defined that references another expression that has not yet been defined. This should be easy enough to re-order after parsing the math for the first time, before evaluation
• time and memory benchmarking vs using the Python API
• Add a debug flag which allows a user to define math components to enter into a debugger for when loading. To make it easier to debug failing expressions (at parsing or evaluation) without access to placing debug breakpoints in the source code

Examples

Some snippets below. A more fleshed out example is here which works with this (zipped) input NetCDF.

Dimensions

dimensions:
  name:
    dtype: string
    ordered: false
  time:
    dtype: datetime
    ordered: true

Parameters

parameters
  dispatch_max:
    default: .inf
    unit: pu
    dims: [name, time]
  capacity_min:
    default: 0
    unit: MW
    dims: [name]
  capacity_max:
    default: .inf
    unit: MW
    dims: [name, time]
  capital_cost:
    default: 0
    unit: $\frac{currency}{nominal_capacity}$
    dims: [name]
  efficiency:
    default: 1
    unit: unitless    
    dims: [name, time]

Lookups

  active:
    dtype: bool
    default: true
    dims: [name, time]
  committable:
    dtype: bool
    default: false
    dims: [name]
  component:
    dtype: string
    dims: [name]
    one_of:
      - "buses"
      - "global_constraints"
      - "lines"
      - "transformers"
      - "links"
      - "loads"
      - "generators"
      - "processes"
      - "storage_units"
      - "stores"

Variables

variables:
  dispatch:
    foreach: [name, time]
    unit: MWh
    default: 0
    bounds:
      lower: -.inf
      upper: .inf
  capacity:
    foreach: [name]
    unit: MW
    default: .inf
    bounds: # refs to a parameter
      lower: capacity_min 
      upper: capacity_max
  spill:
    foreach: [name, time]
    mask: component == storage_units
    unit: MWh
    default: 0
    bounds:
      lower: 0
      upper: .inf

Expressions

expressions:
  capex:
    foreach: [name]
    mask: capital_cost
    equations:
      - expression: capacity * (capital_cost + fom_cost + $cc)
    sub_expressions:
      cc: # capital cost derived from overnight cost; NOTE: not all have been defined in parameters section above
        - mask: overnight_cost and discount_rate == 0 and lifetime == .inf
          expression: "0"
        - mask: overnight_cost and discount_rate == 0 and not lifetime == .inf
          expression: 1 / lifetime
        - mask: overnight_cost and discount_rate > 0 and not lifetime == .inf
          expression: overnight_cost * discount_rate / (1 - 1 / (1 + discount_rate) ** lifetime)
  opex:
    foreach: [name, time]
    mask: marginal_cost
    equations:
      - expression: dispatch * marginal_cost * time_weighting

Constraints

constraints:
  operation_lower:
    description: "Lower limit on dispatch"
    foreach: [name, time]
    mask: active and not committable
    equations:
      - mask: "not dispatch_min == 0"
        expression: dispatch >= dispatch_min * capacity
      - mask: "dispatch_min == 0"
        expression: p >= 0

Objective(s)

objectives: # could feasibly have multiple and then use `mask` as a way to decide between them based on input data
  cost_minimisation:
    equations:
      - expression: sum(opex, over=[time]) + sum(capex, over=name)
    sense: min

Checks

Arbitrary data validate checks that live alongside the math as they are closely related to the specific math formulation.

checks:
  overnight_capital_overlap:
    mask: overnight_cost and capital_cost
    message: can only define one of `overnight_cost` and `capital_cost`
    errors: raise

Checklist

• AI-generated content is marked (see AGENTS.md).
• Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
• Unit tests for new features were added (if applicable).
• A note for the release notes doc/release_notes.rst of the upcoming release is included.
• I consent to the release of this PR's code under the MIT license.

[codspeed-hq]

Merging this PR will degrade performance by 96.66%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

❌ 31 regressed benchmarks
✅ 144 untouched benchmarks
⏩ 175 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Memory	test_to_lp[masked-n=10]	4.9 KB	710.1 KB	-99.31%
❌	Memory	test_to_lp[milp-n=10]	4.9 KB	710.1 KB	-99.31%
❌	Memory	test_to_lp[sos-n=10]	4.9 KB	705.9 KB	-99.31%
❌	Memory	test_to_lp[piecewise-n=10]	4.9 KB	709.1 KB	-99.3%
❌	Memory	test_to_lp[knapsack-n=100]	4.8 KB	681.5 KB	-99.29%
❌	Memory	test_to_lp[basic-n=10]	5.7 KB	714.2 KB	-99.21%
❌	Memory	test_to_lp[qp-n=10]	6.7 KB	703.4 KB	-99.05%
❌	Memory	test_to_lp[cumsum-severity=0]	7 KB	712.1 KB	-99.01%
❌	Memory	test_to_lp[storage-n=10]	30.3 KB	2,966.7 KB	-98.98%
❌	Memory	test_to_lp[expression_arithmetic-n=10]	9.4 KB	720 KB	-98.69%
❌	Memory	test_to_lp[qp-n=1000]	20.4 KB	1,408.3 KB	-98.55%
❌	Memory	test_to_lp[milp-n=50]	43.9 KB	2,693.5 KB	-98.37%
❌	Memory	test_to_lp[storage-n=250]	663 KB	31,200.9 KB	-97.87%
❌	Memory	test_to_lp[sos-n=1000]	67.5 KB	3,038.5 KB	-97.78%
❌	Memory	test_to_lp[sparse_network-n=10]	21.2 KB	719.8 KB	-97.05%
❌	Memory	test_to_lp[knapsack-n=10000]	82.2 KB	2,299.5 KB	-96.42%
❌	Memory	test_to_lp[rolling-severity=0]	105.6 KB	2,947.7 KB	-96.42%
❌	Memory	test_to_lp[nodal_balance-severity=0]	129.1 KB	3,370.2 KB	-96.17%
❌	Memory	test_to_lp[piecewise-n=1000]	123.3 KB	3,069.5 KB	-95.98%
❌	Memory	test_to_lp[basic-n=250]	1.3 MB	27 MB	-95.34%
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing feature/declarative-yaml-interface (c7df8dd) with master (f268bff)}

Footnotes

1. 175 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩