Troubles connecting to redis with secret

Question

Troubles connecting to redis with secret

marinmo opened this issue a month ago · comments

marinmo commented a month ago

Version

v4.38.8

Deployment Method

Other

Reverse Proxy

NGINX

Reverse Proxy Version

1.25.5

Description

I'm having issues connecting to redis as of 4.38.8. I can't really tell when it stopped working, but it's quite recent. I'm using rootless podman to run authelia and auxiliary services (nginx proxy, vaultwarden etc), all on the same machine. There are a few things I've observed which might lead to the issues I'm seeing;

I run the bitnami redis container. This container does not listen on IPv6 (which is not enabled in the podman networking anyway). However, entering the container name in authelia's configuration file (eg. host: 'redis') leads the authelia container to resolve the IPv6 address of the LAN interface.
Entering the IPv4 address of the LAN interface in the host directive leads to an error as described in the logs section. However, I am able to access and store/read entries to redis both from the (redis) container itself, from the server the container is running on and from another client on the LAN.
Other containers on the container network are able to connect to each other (eg. vaultwarden -> pgsql)
Logging on the redis container (loglevel debug) doesn't show any attempt of connection from the authelia container.

Reproduction

I'm obviously running this in containers;
Nginx:latest (1.25.5)
Authelia:latest (4.38.8)
PostgreSQL:16 (16.2.0)
Redis:latest (7.2.4)

Nginx, PGSQL and Redis are packaged by bitnami

Expectations

Successful connection from Authelia to Redis like as such;

user@othercomputer ~> redis-cli -h SERVERIP -a SECRET
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
SERVERIP:6379> get mykey
(nil)
SERVERIP:6379> set mykey helloworld
OK
SERVERIP:6379> get mykey
"helloworld"

Configuration (Authelia)

server:
  address: 'tcp://0.0.0.0:9091'

log:
  level: 'debug'
  keep_stdout: 'true'
  file_path: /config/authlog.log

duo_api:
  hostname: REDACTED
  integration_key: REDACTED

authentication_backend:
  password_reset:
    disable: 'false'
  file:
    path: '/config/users_database.yml'
    password:
      algorithm: 'argon2id'
      iterations: '1'
      key_length: '32'
      salt_length: '16'
      memory: '128'
      parallelism: '8'

access_control:
  default_policy: 'deny'
  rules:
    - domain:
      - 'auth.example.com'
      policy: 'bypass'
      resources:
        - "^/api.*$"
        - "^/identity.*$"
    - domain:
      - 'example.com'
      - '*.example.com'
      policy: 'two_factor'

session:
  cookies:
    - name: 'authelia_session'
      domain: 'example.com'
      authelia_url: 'https://auth.example.com'
      expiration: '1 hour'
      inactivity: '5 minutes'
  redis:
    host: 'IPv4-address'
    port: '6379'

regulation:
  max_retries: '3'
  find_time: '2 minutes'
  ban_time: '5 minutes'

storage:
  postgres:
    address: 'tcp://REDACTED'
    database: 'authelia'
    username: 'authelia'

notifier:
  disable_startup_check: 'false'
  filesystem:
    filename: '/config/notification.txt'

Build Information

podman run docker.io/authelia/authelia authelia build-info
Last Tag: v4.38.8
State: tagged clean
Branch: v4.38.8
Commit: cd32d5ce0b3cc2d581f63700c5cec1174c027718
Build Number: 28628
Build OS: linux
Build Arch: amd64
Build Compiler: gc
Build Date: Mon, 15 Apr 2024 13:44:28 +1000
Extra: 

Go:
    Version: go1.22.1
    Module Path: github.com/authelia/authelia/v4
    Executable Path: github.com/authelia/authelia/v4/cmd/authelia

Logs (Authelia)

time="2024-05-02T22:05:03+02:00" level=fatal msg="redis connection error: dial tcp IPADDRESS:6379: connect: connection refused" stack="github.com/authelia/authelia/v4/internal/commands/context.go:150 (*CmdCtx).LoadProviders\ngithub.com/authelia/authelia/v4/internal/commands/root.go:72     (*CmdCtx).RootRunE\ngithub.com/spf13/cobra@v1.8.0/command.go:983                     (*Command).execute\ngithub.com/spf13/cobra@v1.8.0/command.go:1115                    (*Command).ExecuteC\ngithub.com/spf13/cobra@v1.8.0/command.go:1039                    (*Command).Execute\ngithub.com/authelia/authelia/v4/cmd/authelia/main.go:10          main\nruntime/internal/atomic/types.go:194                             (*Uint32).Load\nruntime/asm_amd64.s:1695                                         goexit"

Logs (Proxy / Application)

× authelia.service - Authelia authentication system
     Loaded: loaded (/home/user/.config/systemd/user/authelia.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/user/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: exit-code) since Thu 2024-05-02 22:17:53 CEST; 2min 28s ago
   Duration: 876ms
    Process: 807280 ExecStartPre=/bin/rm -f /run/user/1000/authelia.service.ctr-id (code=exited, status=0/SUCCESS)
    Process: 807305 ExecStart=/usr/bin/podman run --cidfile=/run/user/1000/authelia.service.ctr-id --cgroups=no-conmon --label io.containers.autoupdate=registry --rm --sdnotify=conmon --name=authelia --hostname authelia --replace -d --log-driver journald -p 9091:9091 --tz=local --init -v /home/user/containers/authelia:/config:Z --env AUTHELIA_IDENTITY_VALIDATION_RESET_PASSWORD_JWT_SECRET_FILE=/config/secrets/jwt --env AUTHELIA_DUO_API_SECRET_KEY_FILE=/config/secrets/duo --env AUTHELIA_SESSION_SECRET_FILE=/config/secrets/session --env AUTHELIA_STORAGE_ENCRYPTION_KEY_FILE=/config/secrets/encryption --env AUTHELIA_STORAGE_POSTGRES_PASSWORD_FILE=/config/secrets/postgres --env AUTHELIA_SESSION_REDIS_PASSWORD_FILE=/config/secrets/redis docker.io/authelia/authelia (code=exited, status=1/FAILURE)
    Process: 807363 ExecStopPost=/usr/bin/podman rm -f --ignore --cidfile=/run/user/1000/authelia.service.ctr-id (code=exited, status=0/SUCCESS)
   Main PID: 807305 (code=exited, status=1/FAILURE)
        CPU: 343ms

May 02 22:17:52 SERVER authelia[807305]: time="2024-05-02T22:17:52+02:00" level=debug msg="Loaded Configuration Sources" files="[/config/configuration.yml]" filters="[]"
May 02 22:17:52 SERVER authelia[807305]: time="2024-05-02T22:17:52+02:00" level=debug msg="Logging Initialized" fields.level=debug file=/config/authlog.log format= keep_stdout=true
May 02 22:17:52 SERVER authelia[807305]: time="2024-05-02T22:17:52+02:00" level=debug msg="Process user information" gid=0 gids="1,2,3,4,6,10,11,20,26,27" name=root uid=0 username=root
May 02 22:17:52 SERVER authelia[807305]: time="2024-05-02T22:17:52+02:00" level=info msg="Authelia v4.38.8 is starting"
May 02 22:17:52 SERVER authelia[807305]: time="2024-05-02T22:17:52+02:00" level=info msg="Log severity set to debug"
May 02 22:17:52 SERVER authelia[807305]: time="2024-05-02T22:17:52+02:00" level=fatal msg="redis connection error: dial tcp IPADDRESS:6379: connect: connection refused" stack="github.com/authelia/authelia/v4/internal/commands/context.go:150 (*CmdCtx).LoadProviders\ngithub.com/authelia/authelia/v4/internal/commands/root.go:72     (*CmdCtx).RootRunE\ngithub.com/spf13/cobra@v1.8.0/command.go:983                     (*Command).execute\ngithub.com/spf13/cobra@v1.8.0/command.go:1115                    (*Command).ExecuteC\ngithub.com/spf13/cobra@v1.8.0/command.go:1039                    (*Command).Execute\ngithub.com/authelia/authelia/v4/cmd/authelia/main.go:10          main\nruntime/internal/atomic/types.go:194                             (*Uint32).Load\nruntime/asm_amd64.s:1695                                         goexit"
May 02 22:17:53 SERVER podman[807343]: 2024-05-02 22:17:53.470561212 +0200 CEST m=+0.505451431 container died 6f44f7d49e47834b827c51dedb1c8fb8b39fad2e6cd42396c71c1db38b7c7dac (image=docker.io/authelia/authelia:latest, name=authelia, org.opencontainers.image.documentation=https://www.authelia.com, org.opencontainers.image.revision=cd32d5ce0b3cc2d581f63700c5cec1174c027718, PODMAN_SYSTEMD_UNIT=authelia.service, org.opencontainers.image.title=authelia, org.opencontainers.image.source=https://github.com/authelia/authelia/tree/cd32d5ce0b3cc2d581f63700c5cec1174c027718, org.opencontainers.image.vendor=Authelia, org.opencontainers.image.base.name=docker.io/library/alpine:3.19.1, org.opencontainers.image.version=4.38.8, io.containers.autoupdate=registry, org.opencontainers.image.description=Authelia is an open-source authentication and authorization server providing two-factor authentication and single sign-on (SSO) for your applications via a web portal., org.opencontainers.image.url=https://github.com/authelia/authelia/pkgs/container/authelia, org.opencontainers.image.created=2024-04-15T13:48:06+10:00, org.opencontainers.image.licenses=Apache-2.0)
May 02 22:17:53 SERVER podman[807343]: 2024-05-02 22:17:53.641347276 +0200 CEST m=+0.676237540 container remove 6f44f7d49e47834b827c51dedb1c8fb8b39fad2e6cd42396c71c1db38b7c7dac (image=docker.io/authelia/authelia:latest, name=authelia, org.opencontainers.image.documentation=https://www.authelia.com, org.opencontainers.image.url=https://github.com/authelia/authelia/pkgs/container/authelia, org.opencontainers.image.created=2024-04-15T13:48:06+10:00, org.opencontainers.image.revision=cd32d5ce0b3cc2d581f63700c5cec1174c027718, org.opencontainers.image.base.name=docker.io/library/alpine:3.19.1, org.opencontainers.image.description=Authelia is an open-source authentication and authorization server providing two-factor authentication and single sign-on (SSO) for your applications via a web portal., org.opencontainers.image.vendor=Authelia, PODMAN_SYSTEMD_UNIT=authelia.service, org.opencontainers.image.version=4.38.8, io.containers.autoupdate=registry, org.opencontainers.image.title=authelia, org.opencontainers.image.licenses=Apache-2.0, org.opencontainers.image.source=https://github.com/authelia/authelia/tree/cd32d5ce0b3cc2d581f63700c5cec1174c027718)
May 02 22:17:53 SERVER systemd[1328]: authelia.service: Main process exited, code=exited, status=1/FAILURE
May 02 22:17:53 SERVER systemd[1328]: authelia.service: Failed with result 'exit-code'.

Documentation

Redis is obviously configured with a secret. I did not try removing the secret, but I did try to pass it to authelia as an environment variable.

Pre-Submission Checklist

I agree to follow the Code of Conduct
This is a bug report and not a support request
I have read the security policy and this bug report is not a security issue or security related issue
I have either included the complete configuration file or I am sure it's unrelated to the configuration
I have either included the complete debug / trace logs or the output of the build-info command if the logs are not relevant
I have provided all of the required information in full with the only alteration being reasonable sanitization in accordance with the Troubleshooting Sanitization reference guide
I have checked for related proxy or application logs and included them if available
I have checked for related issues and checked the documentation

glime-ai · Answer 1 · Wed May 08 2024 04:17:09 GMT+0800 (China Standard Time)

Please note this response was created using generative AI leveraging previously available information from the repository. This information may be helpful however it is our strong recommendation that you carefully consider the information before blindly applying it.

Based on the detailed description and logs you provided, it seems like the issue is related to the network connectivity between the Authelia container and the Redis container. Here are a few steps and checks you can perform to troubleshoot and potentially resolve the issue:

Network Configuration: Ensure that both the Authelia and Redis containers are on the same Docker network. This allows them to communicate with each other using their container names as hostnames.
Redis Host Configuration: In your Authelia configuration, you have specified the Redis host using an IPv4 address. Instead, try using the container name of the Redis server as the host. For example:
```
redis:
  host: 'redis'  # Use the container name instead of the IP address
  port: '6379'
```
Check Redis Logs: Since the Redis logs do not show any attempt of connection from the Authelia container, it's possible that the connection attempts are not reaching the Redis container. This could be due to network misconfiguration or firewall rules.
Firewall and Security Groups: Check if there are any firewall rules or security groups that might be blocking the connection between the containers.
Redis Configuration: Ensure that the Redis container is configured to accept connections from other containers. This can typically be controlled through Redis configuration files or startup command flags.
Container Restart: Sometimes, simply restarting the containers can resolve transient network issues.
Use Docker Compose: If you are not already using Docker Compose, it might help to define your containers and their network settings in a docker-compose.yml file. This ensures that all containers are configured consistently.
Debugging Logs: Increase the logging level in both Authelia and Redis to get more detailed logs that might provide additional insights into the issue.

If after trying these steps the issue persists, it might be helpful to provide the configuration of your Docker network and any additional logs that might have relevant information.

This comment was generated by Glime.

James Elliott · Answer 2 · Wed May 08 2024 04:26:52 GMT+0800 (China Standard Time)

To me this looks like a network issue (the improperly redacted IP is refusing connections on that port). Please show the output of running a separate redis CLI container on the same container network as authelia using the CLI to connect as well as the redis logs for both the authelia connection attempts and the CLI connection attempts.

marinmo · Answer 3 · Thu May 09 2024 02:20:30 GMT+0800 (China Standard Time)

Yep, this was a network issue; I didn't really bother to figure out what went wrong (update from podman 4 -> 5 seems like a likely culprit but couldn't tell more closely). I put all containers into a pod and used localhost as host and it all works, seems - probably - the podman upgrade broke inter-container communication since the redis container was reachable from non-container locations as described in the bug report.

marinmo · Answer 4 · Thu May 09 2024 02:20:53 GMT+0800 (China Standard Time)

Closing this as not a bug with Authelia.