Ability to use specific postgrespro parameters in patroni

Question

Ability to use specific postgrespro parameters in patroni

Eugene10082024 opened this issue 4 months ago · comments

What happened?

I use a failover cluster based on patroni.
Cluster composition: etcd-3.5.0, patroni-3.0.2, postgrespro-std-16 (ver. 16.1.1)
I tried to configure connection pooler built-in postgrespro-std-16.
To enable the pooler you must set the following parameters: connection_pool_workers, max_sessions,session_pool_size
The values of the specified parameters were set in DCS via the command - patronictl -c /etc/patroni/patroni.yml edit-config

But the entered and saved parameter values were not applied and pooler did not start
I found the following messages in log patroni.service
Jan 30 17:30:17 server_02.domain.local patroni[65801]: 2024-01-30 17:30:17,581 INFO: no action. I am (vdc01-piecmdb02), the leader with the lock
Jan 30 17:30:17 server_02.domain.local patroni[65801]: 2024-01-30 17:30:17,590 INFO: Changed connection_pool_workers from 2 to 3 (restart might be required)
Jan 30 17:30:17 server_02.domain.local patroni[65801]: 2024-01-30 17:30:17,591 INFO: Changed max_sessions from 1000 to 200 (restart might be required)
Jan 30 17:30:17 server_02.domain.local patroni[65801]: 2024-01-30 17:30:17,591 INFO: Changed session_pool_size from 0 to 4 (restart might be required)
Jan 30 17:30:17 server_02.domain.local patroni[65801]: 2024-01-30 17:30:17,592 WARNING: Removing unexpected parameter=connection_pool_workers value=3 from the config
Jan 30 17:30:17 server_02.domain.local patroni[65801]: 2024-01-30 17:30:17,592 WARNING: Removing unexpected parameter=max_sessions value=200 from the config
Jan 30 17:30:17 server_02.domain.local patroni[65801]: 2024-01-30 17:30:17,592 WARNING: Removing unexpected parameter=session_pool_size value=4 from the config
Jan 30 17:30:17 server_02.domain.local patroni[65801]: 2024-01-30 17:30:17,593 INFO: Reloading PostgreSQL configuration.
Jan 30 17:30:17 server_02.domain.local patroni[70372]: server signaled

Removed the above options from DCS and moved to /etc/patroni/patroni.yml - the parameters section, where values are set for a specific node.

I found the following messages in log patroni.service
Jan 30 17:39:44 server_02.domain.local patroni[65801]: 2024-01-30 17:39:44,041 WARNING: Removing unexpected parameter=connection_pool_workers value=4 from the config
Jan 30 17:39:44 server_02.domain.local patroni[65801]: 2024-01-30 17:39:44,042 WARNING: Removing unexpected parameter=max_sessions value=500 from the config
Jan 30 17:39:44 server_02.domain.local patroni[65801]: 2024-01-30 17:39:44,042 WARNING: Removing unexpected parameter=session_pool_size value=4 from the config

How can I set the parameters such as: connection_pool_workers, max_sessions, session_pool_size so that they are applied and allow the pooler to be activated?
Where can I see a list of parameters that can be placed in the DCS, after which their values will be applied in postgrespro?
Will the parameters connection_pool_workers, max_sessions, session_pool_size be added to any version of patroni?
If yes, please provide the version number

How can we reproduce it (as minimally and precisely as possible)?

etcd-3.5.0, patroni-3.0.2, postgrespro-std-16

What did you expect to happen?

I expected that the values of the connection_pool_workers, max_sessions, session_pool_size parameters would be applied on the postgrespro nodes

Patroni/PostgreSQL/DCS version

Patroni version: 3.0.2
PostgreSQL version: postgrespro-std-16
DCS (and its version): etcd 3.5.0

Patroni configuration file

name: server_01
namespace: /patroni/
scope: patroni-cluster
restapi:
  listen: 192.168.122.170:8008
  connect_address: server_01.mydomain.local:8008
  authentication:
    username: patroni
    password: patroni
log:
  level: DEBUG
etcd3:
  hosts: 192.168.122.170:2379,192.168.122.171:2379,192.168.122.173:2379
  username: root
  password: root
bootstrap:
  dcs:
    ttl: 30
    loop_wait: 10
    retry_timeout: 10
    maximum_lag_on_failover: 1048576
    master_start_timeout: 300
    postgresql:
      use_pg_rewind: true
      use_slots: true
  initdb:
   - auth-host: scram-sha-256
   - auth-local: peer
   - encoding: UTF8
   - data-checksums
   - locale: en_US.UTF-8
   - waldir: /pgwal/wal/wal
 pg_hba:
   - local all all trust
   - host all all 192.168.122.170/32 trust
   - host all all 192.168.122.171/32 trust
   - host all all 0.0.0.0/0 scram-sha-256
   - host replication replicator 192.168.0.0/16 scram-sha-256
   - host replication replicator 127.0.0.1/32 scram-sha-256
postgresql:
  create_replica_methods:
   - basebackup
  basebackup:
   - waldir: /pgwal/wal/wal
   - checkpoint: 'fast'
   - verbose
  listen: 0.0.0.0:5432
  connect_address: server_01.mydomain.local:5432
  config_dir: /pgdata/16/data
  bin_dir: /opt/pgpro/ent-16/bin
  data_dir: /pgdata/16/data
  pgpass: /tmp/pgpass
  use_unix_socket: true
  authentication:
    superuser:
      username: postgres
      password: passw0rd
    replication:
      username: replicator
      password: passw0rd
    rewind:
      username: rewind_user
      password: passw0rd
  parameters:
    unix_socket_directories: '/tmp'
watchdog:
  mode: 'off'
tags:
  nofailover: false
  noloadbalance: false
clonefrom: false

patronictl show-config

loop_wait: 10
master_start_timeout: 300
maximum_lag_on_failover: 1048576
postgresql:
  parameters:
    autovacuum_analyze_scale_factor: 0.01
    autovacuum_analyze_threshold: 50
    autovacuum_max_workers: 2
    autovacuum_vacuum_cost_delay: 5ms
    autovacuum_vacuum_cost_limit: 400
    autovacuum_vacuum_insert_scale_factor: 0.05
    autovacuum_vacuum_scale_factor: 0.02
    autovacuum_vacuum_threshold: 50
    checkpoint_timeout: 30min
    connection_pool_workers: 4
    default_toast_compression: lz4
    effective_cache_size: 4GB
    maintenance_work_mem: 820MB
    max_connections: 2000
    max_parallel_maintenance_workers: 2
    max_sessions: 500
    max_wal_size: 4GB
    min_wal_size: 1GB
    password_encryption: scram-sha-256
    session_pool_size: 4
    shared_buffers: 4GB
    shared_preload_libraries: pgpro_stats
    vacuum_cost_limit: 200
    wal_compression: lz4
    wal_keep_size: 20GB
    work_mem: 524
  pg_hba:
  - local all postgres peer
  - local all all scram-sha-256
  - host all all 127.0.0.1/32 scram-sha-256
  - host all all 192.168.0.0/16 scram-sha-256
  - host replication replicator 0.0.0.0/0 scram-sha-256
  - host replication replicator 127.0.0.1/32 scram-sha-256
  use_pg_rewind: true
  use_slots: true
retry_timeout: 10
synchronous_mode: false
synchronous_mode_restrict: false
ttl: 30

Patroni log files

Jan 30 17:30:17 server_02.domain.local  patroni[65801]: 2024-01-30 17:30:17,581 INFO: no action. I am (vdc01-piecmdb02), the leader with the lock
Jan 30 17:30:17 server_02.domain.local  patroni[65801]: 2024-01-30 17:30:17,590 INFO: Changed connection_pool_workers from 2 to 3 (restart might be required)
Jan 30 17:30:17 server_02.domain.local  patroni[65801]: 2024-01-30 17:30:17,591 INFO: Changed max_sessions from 1000 to 200 (restart might be required)
Jan 30 17:30:17 server_02.domain.local  patroni[65801]: 2024-01-30 17:30:17,591 INFO: Changed session_pool_size from 0 to 4 (restart might be required)
Jan 30 17:30:17 server_02.domain.local  patroni[65801]: 2024-01-30 17:30:17,592 WARNING: Removing unexpected parameter=connection_pool_workers value=3 from the config
Jan 30 17:30:17 server_02.domain.local  patroni[65801]: 2024-01-30 17:30:17,592 WARNING: Removing unexpected parameter=max_sessions value=200 from the config
Jan 30 17:30:17 server_02.domain.local  patroni[65801]: 2024-01-30 17:30:17,592 WARNING: Removing unexpected parameter=session_pool_size value=4 from the config
Jan 30 17:30:17 server_02.domain.local  patroni[65801]: 2024-01-30 17:30:17,593 INFO: Reloading PostgreSQL configuration.
Jan 30 17:30:17 server_02.domain.local  patroni[70372]: server signaled

PostgreSQL log files

No

Have you tried to use GitHub issue search?

Yes

Anything else we need to know?

No response

Alexander Kukushkin · Answer 1 · Mon Feb 05 2024 21:31:58 GMT+0800 (China Standard Time)

Please upgrade to the latest Patroni version, 3.2.2. It will already make it to work, but there will be no validation for these parameters.
To enable extended validation you can either add new parameters to https://github.com/zalando/patroni/blob/master/patroni/postgresql/available_parameters/0_postgres.yml or add them to the new file under https://github.com/zalando/patroni/blob/master/patroni/postgresql/available_parameters/