Configuration transclusion and query language

[Bartosz Sławecki]

Describe the use case of a new functionality

Structuring complex configuration without repeating yourself.
In detail, the functionality should bring new query language for loading and managing the configuration.
This would definitely help to maintain a neat and tidy, yet complex configuration system.

Example Use

Having

class MyI18n(ConfigModel):
    locale: str
    timezone: str

we could configure it with a single file

# path/to/i18n.yaml
locale: pl_PL
timezone: Europe/Warsaw

and then load it with

MyI18n.load("path/to/path/to/i18n.yaml")

However, if we want to maintain multiple configuration files that should actually do something similar to 'transcluding' this configuration internally, we need to either have identical copies of the common configuration in them (and then if one changes, the other may be inconsistent with it), or have something that performs the transclusion from path/to/i18n.yaml in those configuration files. That transclusion would happen without changing the documents, since the configzen library is built on abstraction over file formats – thus, in a YAML configuration we could then even import a JSON config.

Assuming that both for production and development mode of an example considered application we use the following code to load configuration:

class MyConfig(ConfigModel):
    i18n: MyI18n
    # other independent config variables...

config = MyConfig.load("path/to/development.yaml" if __debug__ else "path/to/production.yaml")

then instead of maintaing separately i18n section in file

# path/to/development.yaml
i18n:  # as in path/to/i18n.yaml
  locale: pl_PL
  timezone: Europe/Warsaw
# other independent config variables...

and separately maintaing i18n section in file

# path/to/production.yaml
i18n:  # as in path/to/i18n.yaml
  locale: pl_PL
  timezone: Europe/Warsaw
# other independent config variables...

and wasting time on worrying whether they are the same as they should,
we could have

# path/to/development.yaml
i18n: 
  .import: path/to/i18n.yaml
# other independent config variables...

and

# path/to/production.yaml
i18n: 
  .import: path/to/i18n.yaml
# other independent config variables...

Please note that I could rewrite these examples into JSON, TOML or anything else supported by anyconfig and it would still work identically.

These import() statements could be even part of a rich query language, that could merge imported configurations, such as

options: 
 .merge:
   .import: path/to/base.yaml
   .import: path/to/overrides.yaml

to merge dictionaries out of imported base.yaml and overrides.yaml.
Namely, that would result in

AppropriateConfigModel(options={**anyconfig.load("base.yaml"), **anyconfig.load("overrides.yaml")})

Moreover, accessing particular scopes of the imported configurations would be considered useful.
We could make a default-settings.yaml with default configs dedicated for other models.

# path/to/default-settings.yaml
i18n:
  locale: pl_PL
  timezone: Europe/Warsaw
database:
  name: postgres
  password: myapp
  port: 5432

and then, in the application development configuration:

# path/to/development.yaml
i18n: 
  .import(i18n): path/to/default-settings.yaml
database:
  name: postgres_dev
  password: 
    .import(database.password): path/to/default-settings.yaml
  port: 
    .import(database.port): path/to/default-settings.yaml

and in the application production configuration:

# path/to/production.yaml
i18n: 
  .import(i18n): path/to/default-settings.yaml
database:
  .import: path/to/default-settings.yaml

# path/to/production.yaml
i18n:
  locale: pl_PL
  timezone: Europe/Warsaw
database:
  .import(database): path/to/default-settings.yaml
  name: postgres_dev

And this way, we would create inheritance of configurations:

# path/to/production.yaml
.import: path/to/default-settings.yaml  # base configuration
database:
  # inform that we inherit from path/to/default-settings.yaml database section, 
  # because now we just overwrite that imported section and YAML will 
  # otherwise forget about it (it does not merge)
  .import(database): .
  # overwrite desired option with our custom value
  name: postgres_dev

which would be equivalent to:

# path/to/production.yaml
# start import(path/to/default-settings.yaml)
i18n:
  locale: pl_PL
  timezone: Europe/Warsaw
database:
  name: postgres
  password: myapp
  port: 5432
# end .import: path/to/default-settings.yaml
database:
  # start .import(database): path/to/default-settings.yaml
  name: postgres
  password: myapp
  # end .import(database): path/to/default-settings.yaml
  name: postgres_dev

that eventually evaluates to

# path/to/production.yaml
i18n:
  locale: pl_PL
  timezone: Europe/Warsaw
database:
  name: postgres
  password: myapp
  name: postgres_dev

Additional context

No response

[Bartosz Sławecki]

All seems done for now. A bit different than expected.