postgres is reloaded before the pga_hba.conf is written to disk leading to latest configs not being applied to postgresql service

[Jason Barnett]

👻 Brief Description

If any resource triggers a :delayed notification to service[postgres] before the first postgresql_access resource is compiled then you're left in a state where the service reload of postgresql happens before the pg_hba.conf is written to disk. In other words, you end up with a running postgresql process that hasn't yet realized the pg_hba.conf change(s).

🥞 Cookbook version

v8.0.2

👩‍🍳 Chef-Infra Version

v15.13.8

🎩 Platform details

RHEL 7.8

Steps To Reproduce

Write a simple recipe like so... See below

The key here is that the postgresql_server_conf is triggering a :delayed notification to service[postgresql] before the postgresql_access is able to trigger a :delayed notification to template[#{conf_dir}/pg_hba.conf] (see this line of code).

postgresql_server_install 'PostgreSQL 11' do
  version '11'
  password nil
  action [:install, :create]
end

postgresql_server_conf 'PostgreSQL 11 Config' do
  version '11'
  additional_config({
    'listen_addresses' => '*'
  })

  notifies :reload, 'service[postgresql]'
end

postgresql_access 'app user' do
  access_type 'host'
  access_db 'app'
  access_user 'app'
  access_addr '192.168.0.0/24'
  access_method 'md5'

  notifies :reload, 'service[postgresql]'
end

It's very easy to workaround this problem by putting the postgresql_server_conf resource after the postgresql_access resource, like so:

postgresql_server_install 'PostgreSQL 11' do
  version '11'
  password nil
  action [:install, :create]
end

postgresql_access 'app user' do
  access_type 'host'
  access_db 'app'
  access_user 'app'
  access_addr '192.168.0.0/24'
  access_method 'md5'

  notifies :reload, 'service[postgresql]'
end

postgresql_server_conf 'PostgreSQL 11 Config' do
  version '11'
  additional_config({
    'listen_addresses' => '*'
  })

  notifies :reload, 'service[postgresql]'
end

🚓 Expected behavior

I would expect the postgresql service reload to always happen after pg_hba.conf is written to disk.

➕ Additional context

• Chef Infra Client executes delayed notifications last, https://github.com/chef/chef/blob/v16.8.8/lib/chef/runner.rb#L140-L146
• Delayed notifications are implemented as a fifo (First-In-First-Out), https://github.com/chef/chef/blob/v16.8.8/lib/chef/runner.rb#L151-L161
• delayed_action is a delayed notification, https://github.com/chef/chef/blob/v16.8.8/lib/chef/resource.rb#L182
• delayed_actions is an array, https://github.com/chef/chef/blob/v16.8.8/lib/chef/run_context.rb#L164

[Jason Barnett]

I assume this either needs to be clearly documented or come up with a creative way to force the creation of pg_hba.conf to the beginning of the delayed_actions array.

[Lance Albertson]

I'm a little confused why this is happening. @tas50 @lamont-granquist any ideas why this might be happening?

[Jason Barnett]

@ramereth this is happening due to the compilation order and the architecture of the chef infra client.

Any delayed notification (or delayed_action) gets added to a fifo queue that gets processed at the end of the chef run.

In this specific case I'm just calling out that if a person creates a resource that notifies service[postgresql] :reload before they create the postgresql_access resource then the delayed_actions queue will look like this:

1. :reload service[postgresql]
2. :create template[#{conf_dir}/pg_hba.conf]

Because it's a fifo, the postgresql service will be reloaded first, then the template[#{conf_dir}/pg_hba.conf] will be created. That ultimately leads to the pg_bha.conf configuration change not being picked up by the service.

[Lamont Granquist]

If you send an immediate notification from the template resource to the reload the service does that fix the problem?

If the template resource isn't always delayed and you need to not do that forced immediate reload, then you probably need to introduce two different template resources for the immediate case (with a delayed notification to reload) and the delayed case (with an immediate notification to reload) or something like that.

But more or less you gotta rethink what you're doing here.

Delayed the template is also really weird, it may be easiest to stop doing that.

It may also be useful to wrap the template into a custom resource, which is then sent a delayed notification so that you get the sub-resource collection compile-converge to handle this kind of orchestration.

[Lamont Granquist]

But "force the template to the head of the delayed notifications" is almost certainly the wrong way to think about solving this problem.

That template resource has a dependency on reloading the service correctly which isn't being expressed. Either it needs to notify a :reload or it needs to be a composite action expressed as a custom resource with a template and a service resource.

If the service reload running first before the template runs creates a problem then you probably need to introduce a composite resource which is responsible for the whole thing and everything needs to be changed to notifying that postgresql_service resource and consumers need to stop sending :reload to the service[postgresql] service entirely.

Hacking it up so that the template moves to the start though is asserting a dependency without specifying it explicitly, and is going to lead to an entire hack-up-the-delayed-notifications API needing to be created, which ultimately will require a depsolver for the delayed notification phase which is a very, very hard "will not fix".