Config file suggestions

[Belle Aerni]

It would be pretty handy to be able to define defaults in the config file that then applies to all databases and could reduce the size of a config file quite a bit.

Specifically I'm thinking about the types, optimization, and merge configuration options as those are likely to be pretty consistent for all databases that are being compiled.

For example, allowing just the types and optimization options to have defaults set would take my config from 185 lines down to 106. Defaults also means that a small, singular change could be a applied to all databases much more easily.

If this is already a thing and I'm just missing it, apologies, but I didn't see anything to indicate such functionality exists.

Thanks!

[Daniel Hååvi]

I think this is a great idea.

Just have to think about how to do this best.
I currently cannot differentiate between an unset value and a zero value, as they are the same. Would have to switch all the config to pointers for that.
Also, will arrays/slices be replaced or merged? If merged, how?

Maybe a cleaner solution is to just use the full types, optimize and merge keys as fallbacks. So, if that key is set, no defaults are used anymore.
This would be an easy change and also a very clean user experience. What do you think?

[Belle Aerni]

That's a great question.

I imagined defaults applied to all databases automatically and if a database sets a value that was already in the defaults then it would replace what was set via the defaults. Trying to handle merging the two sounds accident-prone to me.

Maybe a cleaner solution is to just use the full types, optimize and merge keys as fallbacks. So, if that key is set, no defaults are used anymore.

I think if I am understanding correctly that actually sounds a lot like what I original had in mind, however it's quite possible I'm not understanding correctly.

Here's kinda how I imagined it in practice:

databases:
  defaults:
    types:
      "country.iso_code": string
      "autonomous_system_organization": string
      "autonomous_system_number": uint32
    optimize:
      floatDecimals: 3
      forceIPVersion: true
      maxPrefix: 0

  # This follows the defaults as set above
  - name: "IPv4 - CC0"
    mmdb:
      ipVersion: 4
      recordSize: 24
    inputs:
      - file: "DBs/geo-whois-asn-country-ipv4.csv"
        fields: ["from", "to", "country.iso_code"]
    output: build/cc0-v4.mmdb

  # This would use the defaults except for optimize which would override `floatDecimals` and `maxPrefix` with it's own values
  - name: "IPv6 - CC0"
    mmdb:
      ipVersion: 6
      recordSize: 24
    inputs:
      - file: "DBs/geo-whois-asn-country-ipv6.csv"
        fields: ["from", "to", "country.iso_code"]
    output: build/cc0-v6.mmdb
    optimize:
      floatDecimals: 2
      maxPrefix: 24

If I understand correctly what you meant in the previous comment that would mean something like this where the whole types, optimize, and merge keys must be supplied, which I also think would be sufficient.

databases:
  defaults:
    types:
      "country.iso_code": string
      "autonomous_system_organization": string
      "autonomous_system_number": uint32
    optimize:
      floatDecimals: 3
      forceIPVersion: true
      maxPrefix: 0

  # This follows the defaults as set above
  - name: "IPv4 - CC0"
    mmdb:
      ipVersion: 4
      recordSize: 24
    inputs:
      - file: "DBs/geo-whois-asn-country-ipv4.csv"
        fields: ["from", "to", "country.iso_code"]
    output: build/cc0-v4.mmdb

  # This would use the defaults except for optimize which would override `floatDecimals` and `maxPrefix` with it's own values
  - name: "IPv6 - CC0"
    mmdb:
      ipVersion: 6
      recordSize: 24
    inputs:
      - file: "DBs/geo-whois-asn-country-ipv6.csv"
        fields: ["from", "to", "country.iso_code"]
    output: build/cc0-v6.mmdb
    optimize:
      floatDecimals: 2
      forceIPVersion: true #Even though `forceIPVersion` is unchanged from the originally set defaults, it must still be supplied
      maxPrefix: 24

[Daniel Hååvi]

I went with the first version, inheriting settings separately and sometimes merging them.

This is now v0.3: https://github.com/safing/mmdbmeld/releases/tag/v0.3.0

Check the updated README for details.

[Belle Aerni]

That was pretty quick, thanks!
I'll give that a go and if I have any issues I'll let you know