max_connection value doesn't work unless you do with patronictl edit-config

Question

max_connection value doesn't work unless you do with patronictl edit-config

anaskhantpl opened this issue 20 days ago · comments

What happened?

We have patroni cluster running production, but due to some reason, we need to change the max_connections from default value 100 to 500.

How can we reproduce it (as minimally and precisely as possible)?

i have tried to add max_connections in different positions, even though in /data/patroni/postgresql.base.conf file but doesn't work!
1-

bootstrap:
  dcs:
    ttl: 30
    loop_wait: 10
    retry_timeout: 10
    maximum_lag_on_failover: 1048576
    max_connections: 500
    postgresql:
      parameters:
      use_pg_rewind: true
      use_slots: true

2-

bootstrap:
  dcs:
    ttl: 30
    loop_wait: 10
    retry_timeout: 10
    maximum_lag_on_failover: 1048576
    postgresql:
      parameters:
          max_connections: 500
      use_pg_rewind: true
      use_slots: true

3-

 parameters:
      max_connections: 500
      unix_socket_directories: '.'
      log_statement: 'all'
      log_destination: 'stderr'
      logging_collector: on
      log_directory: '/var/log/postgresql'
      log_min_duration_statement: 0
      log_line_prefix: 'time=%t, pid=%p %q db=%d, usr=%u, client=%h, app=%a, line=%l'

What did you expect to happen?

i want to make changes in to persistent file which located /etc/patroni.yml , rather than using patronictl edit-config command.

Patroni/PostgreSQL/DCS version

Patroni version: 3.2.2
PostgreSQL version: 16.2
DCS (and its version): etcd-3.5.13

Patroni configuration file

scope: postgres
namespace: /db/
name: pg-n3

restapi:
    listen: 192.168.100.1:8008
    connect_address: 192.168.100.1:8008

etcd:
    hosts: 192.168.100.1:2379,192.168.100.2:2379,192.168.100.3:2379

bootstrap:
  dcs:
    ttl: 30
    loop_wait: 10
    retry_timeout: 10
    maximum_lag_on_failover: 1048576
#    max_connections: 500
    postgresql:
      parameters:
    #    max_connections: 500
      use_pg_rewind: true
      use_slots: true

  initdb:
  - encoding: UTF8
  - data-checksums

  pg_hba:
  - local all postgres peer
  - local all all peer
  - host all all 0.0.0.0/0 md5
  - host replication replicator 127.0.0.1/32 md5
  - host replication replicator 192.168.100.1/0 md5
  - host replication replicator 192.168.100.2/0 md5
  - host replication replicator 192.168.100.3/0 md5
  - host all all 0.0.0.0/0 md5

  users:
    admin:
      password: admin
      options:
        - createrole
        - createdb

postgresql:
  use_pg_bouncer: true
  listen: 192.168.100.1:5432
  connect_address: 192.168.100.1:5432
  data_dir: /data/patroni
  pgpass: /tmp/pgpass
  authentication:
    replication:
      username: replicator
      password: Replicator
    superuser:
      username: postgres
      password: Postgres123
  parameters:
   #   max_connections: 500
      unix_socket_directories: '.'
      log_statement: 'all'
      log_destination: 'stderr'
      logging_collector: on
      log_directory: '/var/log/postgresql'
      log_min_duration_statement: 0
      log_line_prefix: 'time=%t, pid=%p %q db=%d, usr=%u, client=%h, app=%a, line=%l'

tags:
    nofailover: false
    noloadbalance: false
    clonefrom: false
    nosync: false

patronictl show-config

loop_wait: 10
maximum_lag_on_failover: 1048576
postgresql:
  parameters: null
  use_pg_rewind: true
  use_slots: true
retry_timeout: 10
ttl: 30

Patroni log files

time=2024-05-08 23:00:56 UTC, pid=109158 LOG:  started streaming WAL from primary at 0/5000000 on timeline 13
time=2024-05-08 23:00:57 UTC, pid=109163  db=postgres, usr=postgres, client=192.168.100.1, app=Patroni heartbeat, line=1LOG:  statement: SELECT CASE WHEN pg_catalog.pg_is_in_recovery() THEN 0 ELSE ('x' || pg_catalog.substr(pg_catalog.pg_walfile_name(pg_catalog.pg_current_wal_lsn()), 1, 8))::bit(32)::int END, CASE WHEN pg_catalog.pg_is_in_recovery() THEN 0 ELSE pg_catalog.pg_wal_lsn_diff(pg_catalog.pg_current_wal_flush_lsn(), '0/0')::bigint END, pg_catalog.pg_wal_lsn_diff(pg_catalog.pg_last_wal_replay_lsn(), '0/0')::bigint, pg_catalog.pg_wal_lsn_diff(COALESCE(pg_catalog.pg_last_wal_receive_lsn(), '0/0'), '0/0')::bigint, pg_catalog.pg_is_in_recovery() AND pg_catalog.pg_is_wal_replay_paused(), 0, CASE WHEN latest_end_lsn IS NULL THEN NULL ELSE received_tli END, slot_name, conninfo, status, pg_catalog.current_setting('restore_command'), NULL, 'on', '', NULL FROM pg_catalog.pg_stat_get_wal_receiver()
time=2024-05-08 23:00:57 UTC, pid=109163  db=postgres, usr=postgres, client=192.168.100.1, app=Patroni heartbeat, line=2LOG:  duration: 1.345 ms
time=2024-05-08 23:00:57 UTC, pid=109163  db=postgres, usr=postgres, client=192.168.100.1, app=Patroni heartbeat, line=3LOG:  statement: SELECT name, setting, unit, vartype, context, sourcefile FROM pg_catalog.pg_settings  WHERE pg_catalog.lower(name) = ANY(ARRAY['archive_cleanup_command','primary_conninfo','primary_slot_name','promote_trigger_file','recovery_end_command','recovery_min_apply_delay','recovery_target','recovery_target_lsn','recovery_target_name','recovery_target_time','recovery_target_timeline','recovery_target_xid','restore_command'])
time=2024-05-08 23:00:57 UTC, pid=109163  db=postgres, usr=postgres, client=192.168.100.1, app=Patroni heartbeat, line=4LOG:  duration: 1.977 ms
time=2024-05-08 23:00:57 UTC, pid=109163  db=postgres, usr=postgres, client=192.168.100.1, app=Patroni heartbeat, line=5LOG:  statement: SELECT slot_name, slot_type, pg_catalog.pg_wal_lsn_diff(restart_lsn, '0/0')::bigint, plugin, database, datoid, catalog_xmin, pg_catalog.pg_wal_lsn_diff(confirmed_flush_lsn, '0/0')::bigint FROM pg_catalog.pg_replication_slots WHERE NOT temporary
time=2024-05-08 23:00:57 UTC, pid=109163  db=postgres, usr=postgres, client=192.168.100.1, app=Patroni heartbeat, line=6LOG:  duration: 1.108 ms
time=2024-05-08 23:01:06 UTC, pid=109163  db=postgres, usr=postgres, client=192.168.100.1, app=Patroni heartbeat, line=7LOG:  statement: SELECT CASE WHEN pg_catalog.pg_is_in_recovery() THEN 0 ELSE ('x' || pg_catalog.substr(pg_catalog.pg_walfile_name(pg_catalog.pg_current_wal_lsn()), 1, 8))::bit(32)::int END, CASE WHEN pg_catalog.pg_is_in_recovery() THEN 0 ELSE pg_catalog.pg_wal_lsn_diff(pg_catalog.pg_current_wal_flush_lsn(), '0/0')::bigint END, pg_catalog.pg_wal_lsn_diff(pg_catalog.pg_last_wal_replay_lsn(), '0/0')::bigint, pg_catalog.pg_wal_lsn_diff(COALESCE(pg_catalog.pg_last_wal_receive_lsn(), '0/0'), '0/0')::bigint, pg_catalog.pg_is_in_recovery() AND pg_catalog.pg_is_wal_replay_paused(), 0, CASE WHEN latest_end_lsn IS NULL THEN NULL ELSE received_tli END, slot_name, conninfo, status, pg_catalog.current_setting('restore_command'), NULL, 'on', '', NULL FROM pg_catalog.pg_stat_get_wal_receiver()
time=2024-05-08 23:01:06 UTC, pid=109163  db=postgres, usr=postgres, client=192.168.100.1, app=Patroni heartbeat, line=8LOG:  duration: 0.332 ms
time=2024-05-08 23:01:10 UTC, pid=109157 LOG:  redo starts at 0/542CCE0
time=2024-05-08 23:01:16 UTC, pid=109163  db=postgres, usr=postgres, client=192.168.100.1, app=Patroni heartbeat, line=9LOG:  statement: SELECT CASE WHEN pg_catalog.pg_is_in_recovery() THEN 0 ELSE ('x' || pg_catalog.substr(pg_catalog.pg_walfile_name(pg_catalog.pg_current_wal_lsn()), 1, 8))::bit(32)::int END, CASE WHEN pg_catalog.pg_is_in_recovery() THEN 0 ELSE pg_catalog.pg_wal_lsn_diff(pg_catalog.pg_current_wal_flush_lsn(), '0/0')::bigint END, pg_catalog.pg_wal_lsn_diff(pg_catalog.pg_last_wal_replay_lsn(), '0/0')::bigint, pg_catalog.pg_wal_lsn_diff(COALESCE(pg_catalog.pg_last_wal_receive_lsn(), '0/0'), '0/0')::bigint, pg_catalog.pg_is_in_recovery() AND pg_catalog.pg_is_wal_replay_paused(), 0, CASE WHEN latest_end_lsn IS NULL THEN NULL ELSE received_tli END, slot_name, conninfo, status, pg_catalog.current_setting('restore_command'), NULL, 'on', '', NULL FROM pg_catalog.pg_stat_get_wal_receiver()
time=2024-05-08 23:01:16 UTC, pid=109163  db=postgres, usr=postgres, client=192.168.100.1, app=Patroni heartbeat, line=10LOG:  duration: 0.291 ms
time=2024-05-08 23:01:26 UTC, pid=109163  db=postgres, usr=postgres, client=192.168.100.1, app=Patroni heartbeat, line=11LOG:  statement: SELECT CASE WHEN pg_catalog.pg_is_in_recovery() THEN 0 ELSE ('x' || pg_catalog.substr(pg_catalog.pg_walfile_name(pg_catalog.pg_current_wal_lsn()), 1, 8))::bit(32)::int END, CASE WHEN pg_catalog.pg_is_in_recovery() THEN 0 ELSE pg_catalog.pg_wal_lsn_diff(pg_catalog.pg_current_wal_flush_lsn(), '0/0')::bigint END, pg_catalog.pg_wal_lsn_diff(pg_catalog.pg_last_wal_replay_lsn(), '0/0')::bigint, pg_catalog.pg_wal_lsn_diff(COALESCE(pg_catalog.pg_last_wal_receive_lsn(), '0/0'), '0/0')::bigint, pg_catalog.pg_is_in_recovery() AND pg_catalog.pg_is_wal_replay_paused(), 0, CASE WHEN latest_end_lsn IS NULL THEN NULL ELSE received_tli END, slot_name, conninfo, status, pg_catalog.current_setting('restore_command'), NULL, 'on', '', NULL FROM pg_catalog.pg_stat_get_wal_receiver()
time=2024-05-08 23:01:26 UTC, pid=109163  db=postgres, usr=postgres, client=192.168.100.1, app=Patroni heartbeat, line=12LOG:  duration: 0.357 ms
time=2024-05-08 23:01:36 UTC, pid=109163  db=postgres, usr=postgres, client=192.168.100.1, app=Patroni heartbeat, line=13LOG:  statement: SELECT CASE WHEN pg_catalog.pg_is_in_recovery() THEN 0 ELSE ('x' || pg_catalog.substr(pg_catalog.pg_walfile_name(pg_catalog.pg_current_wal_lsn()), 1, 8))::bit(32)::int END, CASE WHEN pg_catalog.pg_is_in_recovery() THEN 0 ELSE pg_catalog.pg_wal_lsn_diff(pg_catalog.pg_current_wal_flush_lsn(), '0/0')::bigint END, pg_catalog.pg_wal_lsn_diff(pg_catalog.pg_last_wal_replay_lsn(), '0/0')::bigint, pg_catalog.pg_wal_lsn_diff(COALESCE(pg_catalog.pg_last_wal_receive_lsn(), '0/0'), '0/0')::bigint, pg_catalog.pg_is_in_recovery() AND pg_catalog.pg_is_wal_replay_paused(), 0, CASE WHEN latest_end_lsn IS NULL THEN NULL ELSE received_tli END, slot_name, conninfo, status, pg_catalog.current_setting('restore_command'), NULL, 'on', '', NULL FROM pg_catalog.pg_stat_get_wal_receiver()
time=2024-05-08 23:01:36 UTC, pid=109163  db=postgres, usr=postgres, client=192.168.100.1, app=Patroni heartbeat, line=14LOG:  duration: 0.510 ms
time=2024-05-08 23:01:37 UTC, pid=109166  db=postgres, usr=postgres, client=192.168.100.1, app=psql, line=1LOG:  statement: SELECT name, setting FROM pg_settings WHERE name = 'max_connections';
time=2024-05-08 23:01:37 UTC, pid=109166  db=postgres, usr=postgres, client=192.168.100.1, app=psql, line=2LOG:  duration: 1.760 ms

PostgreSQL log files

NULL

Have you tried to use GitHub issue search?

Yes

Anything else we need to know?

i dont see anything in logs to resolve that issue!

Sophia Ruan · Answer 1 · Thu May 09 2024 16:45:07 GMT+0800 (China Standard Time)

Please see the instruction in https://github.com/zalando/patroni/blob/master/docs/patroni_configuration.rst

PostgreSQL parameters controlled by Patroni
Some of the PostgreSQL parameters must hold the same values on the primary and the replicas. For those, values set either in the local patroni configuration files or via the environment variables take no effect. To alter or set their values one must change the shared configuration in the DCS. Below is the actual list of such parameters together with the default values:

max_connections: 100
max_locks_per_transaction: 64
max_worker_processes: 8
max_prepared_transactions: 0
wal_level: hot_standby
track_commit_timestamp: off

So max_connections is controlled by patroni, you must use either patronictl_edit_config tool or Patroni REST API <rest_api> to change it in dynamic configuration.

anaskhantpl · Answer 2 · Fri May 10 2024 00:55:03 GMT+0800 (China Standard Time)

@XiuhuaRuan Thank you for clarifying. Another question arises: Is the DCS section in /etc/patroni.yml not included in the shared configuration?

Polina Bungina · Answer 3 · Fri May 10 2024 03:14:21 GMT+0800 (China Standard Time)

https://patroni.readthedocs.io/en/latest/yaml_configuration.html#bootstrap-configuration

Please use Slack for questions.