Using environment variables to replace variables inside plugin configs does not seem to trigger

Question

Using environment variables to replace variables inside plugin configs does not seem to trigger

janerjak opened this issue 3 months ago · comments

Describe the problem

Hi,

I'm using the environment variable replacement feature for config files described here:
https://docker-minecraft-server.readthedocs.io/en/latest/configuration/interpolating/#replacing-variables-inside-configs
https://docker-minecraft-server.readthedocs.io/en/latest/mods-and-plugins/#optional-plugins-mods-and-config-attach-points

I am trying to use environment variables defined in Portainer in plugin config files.
My plugins are stored in the host volume mounted to /plugins as described in the docs.

A file /plugins/test.yaml has the following contents:

just-some-mapping:
  just-some-variable: ${TEST_SECRET}

The container logs show that the file is being interpolated to the /data/plugins directory:

[init] Copying any plugins from /plugins to /data/plugins
[mc-image-helper] 16:15:19.325 INFO  : Interpolating /plugins/test.yaml -> /data/plugins/test.yaml

The file is present, but the contents are the following:

root@minecraft:/data/plugins# cat test.yaml 
just-some-mapping:
  just-some-variable: ${TEST_SECRET}

to clarify: The variable name is not replaced by any value.

I specifically encased the variable name in curly brackets, since this is stated to be required in the docs:

Variables that you want to replace need to be declared inside curly brackets and prefixed with a dollar sign, such as ${CFG_YOUR_VARIABLE}, which is same as many scripting languages.
https://docker-minecraft-server.readthedocs.io/en/latest/configuration/interpolating/#replacing-variables-inside-configs

I enabled environment variable substitution during the sync stage using the appropriate environment variable within the compose file:

environment:
    # Enable environment variable substitution
    REPLACE_ENV_DURING_SYNC: true
    REPLACE_ENV_SUFFIXES: "yml,yaml,txt,cfg,conf,properties,hjson,json,tml,toml"

Note that I additionally specified REPLACE_ENV_SUFFIXES, since the default value stated in the docs was not present when performing echo $REPLACE_ENV_SUFFIXES. But that still does not cause substitution within test.yaml.
NOTE: The variable isn't mentioned in the variable glossary from the docs:
https://docker-minecraft-server.readthedocs.io/en/latest/variables/

Although $TEST_SECRET is not explicitly defined within the environment mapping of the compose file, an env_file is specified:

env_file:
    - minecraft.adapter.env

which has the following contents, defining $TEST_SECRET:

TEST_SECRET=$TEST_SECRET

Querying $TEST_SECRET from within the container works as expected:

root@minecraft:/data/plugins# echo $TEST_SECRET
Replaced by value from environment variable

(Replaced by value from environment variable is the value of $TEST_SECRET as defined in Portainer)
Am I using the feature correctly? What can I check / try?

Thanks and best regards,
Jan

Container definition

services:
  minecraft-server:
    container_name: minecraft-server
    image: itzg/minecraft-server:latest
    env_file:
      - minecraft.adapter.env
    environment:
      UID: 2000
      GID: 2000

      EULA: TRUE
      SNOOPER_ENABLED: false

      ## Server.properties
      OVERRIDE_SERVER_PROPERTIES: true
      DUMP_SERVER_PROPERTIES: true

      # Branding
      SERVER_NAME: $SERVER_NAME

      # World generation and source
      LEVEL: $CURRENT_WORLD # World_Erich_1
      SEED: $CURRENT_SEED # SpeeBlock

      # Whitelist
      ENFORCE_WHITELIST: true
      WHITELIST: |
        D4rkn
      #WHITELIST_FILE: /user-lists/white-list.json
      EXISTING_WHITELIST_FILE: MERGE

      OPS: |
        D4rkn
      #OPS_FILE: /user-lists/ops.json
      EXISTING_OPS_FILE: MERGE

      ## Server type and mods
      VERSION: $MINECRAFT_VERSION
      TYPE: PAPER
      #PAPERBUILD: 496 #1.20.4
      # 1.20.6 #86 dev build
      PAPER_DOWNLOAD_URL: https://api.papermc.io/v2/projects/paper/versions/1.20.6/builds/86/downloads/paper-1.20.6-86.jar

      # Enable environment variable substitution
      REPLACE_ENV_DURING_SYNC: true
      REPLACE_ENV_SUFFIXES: "yml,yaml,txt,cfg,conf,properties,hjson,json,tml,toml"

      # Miscellaneous
      MODE: survival
      DIFFICULTY: normal
      FORCE_GAMEMODE: true
      HARDCORE: false

      GENERATE_STRUCTURES: true
      MAX_BUILD_HEIGHT: 320
      VIEW_DISTANCE: $VIEW_DISTANCE
      SIMULATION_DISTANCE: $SIMULATION_DISTANCE
      
      MAX_PLAYERS: 50
      PLAYER_IDLE_TIMEOUT: 10
      ANNOUNCE_PLAYER_ACHIEVEMENTS: false
      
      BROADCAST_CONSOLE_TO_OPS: true

      # Auto-pausing
      ENABLE_AUTOPAUSE: true
      AUTOPAUSE_TIMEOUT_EST: 60
      AUTOPAUSE_TIMEOUT_INIT: 600
      AUTOPAUSE_TIMEOUT_KN: 120
      AUTOPAUSE_PERIOD: 60
      AUTOPAUSE_KNOCK_INTERFACE: eth0
      # TODO: Is rootless auto-pause required?

      # Disable server watchdog, as resuming a pause would otherwise force a restart of the server. See: https://docker-minecraft-server.readthedocs.io/en/latest/misc/autopause-autostop/autopause/
      MAX_TICK_TIME: -1

      # Additional protocols (Rcon, gamespy query)
      ENABLE_RCON: true
      RCON_PASSWORD: $RCON_PASSWORD
      ENABLE_QUERY: false
      QUERY_PORT: 25566

      # Commands to run event based
      RCON_CMDS_STARTUP:  |-
        team add NewArrivals
        team add ToBeAssigned
        team add Assigned
        chunkgen start $CHUNKGEN_BLOCK_RANGE ${CURRENT_WORLD}
        chunkgen start $CHUNKGEN_BLOCK_RANGE ${CURRENT_WORLD}_nether
        chunkgen start $CHUNKGEN_BLOCK_RANGE ${CURRENT_WORLD}_the_end
      RCON_CMDS_ON_CONNECT: |-
        team join NewArrivals @a[team=]
        team join ToBeAssigned @a[team=NewArrivals]
      RCON_CMDS_FIRST_CONNECT: |-
        chunkgen cancel
        weather clear 3600
        time set day
      RCON_CMDS_LAST_DISCONNECT: |-
        weather clear
        kill @e[type=minecraft:boat]
        chunkgen start $CHUNKGEN_BLOCK_RANGE ${CURRENT_WORLD}
        chunkgen start $CHUNKGEN_BLOCK_RANGE ${CURRENT_WORLD}_nether
        chunkgen start $CHUNKGEN_BLOCK_RANGE ${CURRENT_WORLD}_the_end

      ## RAM and JVM arguments
      # GENERAL_ARGS="-Xms1G -Xmx20G"
      INIT_MEMORY: $JVM_INIT_MEMORY
      MAX_MEMORY: $JVM_MAX_MEMORY
      # TODO: Test if quotes do anything
      JVM_XX_OPTS: "-XX:+UseG1GC -XX:+ParallelRefProcEnabled -XX:MaxGCPauseMillis=200 -XX:+UnlockExperimentalVMOptions -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -XX:G1NewSizePercent=30 -XX:G1MaxNewSizePercent=40 -XX:G1HeapRegionSize=8M -XX:G1ReservePercent=20 -XX:G1HeapWastePercent=5 -XX:G1MixedGCCountTarget=4 -XX:InitiatingHeapOccupancyPercent=15 -XX:G1MixedGCLiveThresholdPercent=90 -XX:G1RSetUpdatingPauseTimePercent=5 -XX:SurvivorRatio=32 -XX:+PerfDisableSharedMem -XX:MaxTenuringThreshold=1"
      # Disable watchdog additionally explicitly using JVM option
      JVM_DD_OPTS: disable.watchdog=true

    volumes:
      - /truenas/Rapid/app-data-vm/game-servers/minecraft/data:/data
      
      - /truenas/Rapid/app-data-vm/game-servers/minecraft/plugin-vault/1.20.6/mods:/mods
      - /truenas/Rapid/app-data-vm/game-servers/minecraft/plugin-vault/1.20.6/plugins:/plugins
      - /truenas/Rapid/app-data-vm/game-servers/minecraft/config:/config
      
      - /truenas/Rapid/app-data-vm/game-servers/minecraft/user-lists:/user-lists
    tty: true
    stdin_open: true
    restart: unless-stopped
        
    deploy:
      resources:
        limits:
          memory: $CONTAINER_MEMORY_LIMIT

    profiles:
      - on-demand


networks:
  macvlan:
    external: true

Container logs

[init] Copying any plugins from /plugins to /data/plugins
[mc-image-helper] 16:15:19.325 INFO  : Interpolating /plugins/test.yaml -> /data/plugins/test.yaml

Geoff Bourne · Answer 1 · Fri May 24 2024 04:48:40 GMT+0800 (China Standard Time)

The variable name prefix is important:

https://docker-minecraft-server.readthedocs.io/en/latest/configuration/interpolating/#replacing-variables-inside-configs:~:text=You%20can%20also%20change%20REPLACE_ENV_VARIABLE_PREFIX%2C%20which%20defaults%20to%20%22CFG_%22%2C%20to%20limit%20which%20environment%20variables%20are%20allowed%20to%20be%20used.

By default, for security reasons, it will only access variables that have a name that starts with "CFG_".

Jan Karuc · Answer 2 · Thu May 30 2024 23:11:24 GMT+0800 (China Standard Time)

I have to apologize for missing that crucial line in the wiki. That explains my struggles, thanks! Changing REPLACE_ENV_VARIABLE_PREFIX to _ and prefixing every variable worked perfectly. Although curly braces as in ${VAR_NAME} seem to be required as stated in the wiki ($VAR_NAME does not work).

If I understand the good intention behind the required prefix correctly - somebody could push a malicious mod / plugin update that scans for secrets in environment variables and steal them otherwise?

Is that the reason you can not disable this? Setting REPLACE_ENV_VARIABLE_PREFIX to the empty string doesn't seem to work.

Geoff Bourne · Answer 3 · Fri May 31 2024 10:21:59 GMT+0800 (China Standard Time)

I have to apologize for missing that crucial line in the wiki. That explains my struggles, thanks! Changing REPLACE_ENV_VARIABLE_PREFIX to _ and prefixing every variable worked perfectly. Although curly braces as in ${VAR_NAME} seem to be required as stated in the wiki ($VAR_NAME does not work).

If I understand the good intention behind the required prefix correctly - somebody could push a malicious mod / plugin update that scans for secrets in environment variables and steal them otherwise?

Correct.

Is that the reason you can not disable this? Setting REPLACE_ENV_VARIABLE_PREFIX to the empty string doesn't seem to work.

I just tested and setting to empty string disables the prefix as documented:

I created a config/test.cfg file that contained

eula=${EULA}

and used the compose file

services:
  mc:
    image: itzg/minecraft-server
    environment:
      REPLACE_ENV_VARIABLE_PREFIX: ""
      EULA: true
    volumes:
      - ./config:/config
      - ./data:/data

and the file data/config/test.cfg now contains

eula=true

Jan Karuc · Answer 4 · Fri May 31 2024 21:17:13 GMT+0800 (China Standard Time)

I have to correct, that works exactly as expected.

Is that the reason you can not disable this? Setting REPLACE_ENV_VARIABLE_PREFIX to the empty string doesn't seem to work.

Before writing this I tested it with my setup multiple times to no avail - but now it works fine.
The only thing I can assume is that there were external factors like Portainer not correctly mapping my stack environment variables. I assume this because the only variables not replaced in my test.cfg are those that have no value. Maybe that was the issue.

Nevertheless, this is resolved for me. I'm still baffled. Sorry for wasting your time.

Jan Karuc · Answer 5 · Fri May 31 2024 21:20:22 GMT+0800 (China Standard Time)

Closing this issue. I made some mistake somewhere.

REPLACE_ENV_VARIABLE_PREFIX decides which variables are replaced. The default value is CFG_ meaning only variables like CFG_VAR_1 are replaced. Setting REPLACE_ENV_VARIABLE_PREFIX to the empty string makes docker-minecraft-server replace every variable.