[BUG] Custom pillarenv issue on multimaster installation
denyka opened this issue · comments
Description
We have multimaster setup with the same configuration on both masters.
Looks like salt master continues to not respect pillarenv's (while another master continues to do so). It does however respect saltenv aka will look at alternative branches for state files (just not pillar).
We have tried:
Rebuilding the entire salt-master.
Using both gitpython and libgit2 git libraries for the gitfs backend.
Downgrading gitpython versions
The impact is that regardless of what you specify as pillarenv= when applying states or fetching pillar entries, it always fetches from the master branch.
Setup
Both masters are on Almalinux8
For gitfs we have configured
gpg_keydir: /etc/salt/master.d/gpgkeys
top_file_merging_strategy: same
state_top_saltenv: base
fileserver_backend:
- roots
- git
file_roots:
base:
- /srv/salt
# - set salt states gitfs
gitfs_provider: gitpython
gitfs_remotes:
- ssh://git@github.<redacted>/salt.git
# - set salt pillar gitfs
git_pillar_provider: gitpython
ext_pillar:
- git:
- __env__ ssh://git@github.<redacted>/salt.git:
- root: pillar
# the pillarenv value will assume the value of the effective saltenv when running states.
pillarenv_from_saltenv: true
For minions we have:
log_file: /var/log/salt/minion
#to prevent the minion from setting itself back to default base environment which is the master branch.
default_top: nonexistent_branch
master_shuffle: True
verify_master_pubkey_sign: True
master:
- salt-master1
- salt-master2
Steps to Reproduce the behavior
When we run command like this
salt 'test-minion' pillar.item ceph_release_codename pillarenv=some_test_env
we get the different output on both masters
on salt-master1
test-minion:
----------
ceph_release_codename:
knottyliss
on salt-master2
test-minion:
----------
ceph_release_codename:
Nautilus
minion log on working master
[INFO ] User sudo_dneichev Executing command pillar.item with jid 20220609145451892648
[DEBUG ] Command details {'fun': 'pillar.item', 'arg': ['ceph_release_codename', {'__kwarg__': True, 'pillarenv': 'some_test_env'}], 'tgt': 'test-minion', 'jid': '20220609145451892648', 'ret': '', 'tgt_type': 'glob', 'user': 'sudo_dneichev'}
[DEBUG ] Subprocess ProcessPayload-Job-20220609145451892648 added
[DEBUG ] Using importlib_metadata to load entry points
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] LazyLoaded yaml.render
[INFO ] Starting a new job 20220609145451892648 with PID 581062
[DEBUG ] Could not LazyLoad direct_call.allow_missing_func: 'direct_call.allow_missing_func' is not available.
[DEBUG ] LazyLoaded pillar.item
[DEBUG ] Determining pillar cache
[DEBUG ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'test-minion', 'tcp://salt-master:4506')
[DEBUG ] Connecting the Minion to the Master URI (for the return server): tcp://salt-master:4506
[DEBUG ] Trying to connect to: tcp://salt-master:4506
[DEBUG ] salt.crypt.get_rsa_key: Loading private key
[DEBUG ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG ] salt.crypt.verify_signature: Loading public key
[DEBUG ] salt.crypt.get_rsa_pub_key: Loading public key
[DEBUG ] salt.crypt.verify_signature: Verifying signature
[DEBUG ] Closing AsyncZeroMQReqChannel instance
[DEBUG ] Minion return retry timer set to 5 seconds (randomized)
[INFO ] Returning information for job: 20220609145451892648
[DEBUG ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'test-minion', 'tcp://salt-master:4506')
[DEBUG ] Connecting the Minion to the Master URI (for the return server): tcp://salt-master:4506
[DEBUG ] Trying to connect to: tcp://salt-master:4506
[DEBUG ] Closing AsyncZeroMQReqChannel instance
[DEBUG ] minion return: {'success': True, 'return': {'ceph_release_codename': 'knottyliss'}, 'retcode': 0, 'jid': '20220609145451892648', 'fun': 'pillar.item', 'fun_args': ['ceph_release_codename', {'pillarenv': 'some_test_env'}]}
[DEBUG ] Subprocess ProcessPayload-Job-20220609145451892648 cleaned up
minion log with problematic master
[INFO ] User sudo_dneichev Executing command pillar.item with jid 20220609145541104944
[DEBUG ] Command details {'fun': 'pillar.item', 'arg': ['ceph_release_codename', {'__kwarg__': True, 'pillarenv': 'some_test_env'}], 'tgt': 'test-minion', 'jid': '20220609145541104944', 'ret': '', 'tgt_type': 'glob', 'user': 'sudo_dneichev'}
[DEBUG ] Subprocess ProcessPayload-Job-20220609145541104944 added
[DEBUG ] Using importlib_metadata to load entry points
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] LazyLoaded yaml.render
[INFO ] Starting a new job 20220609145541104944 with PID 581392
[DEBUG ] Could not LazyLoad direct_call.allow_missing_func: 'direct_call.allow_missing_func' is not available.
[DEBUG ] LazyLoaded pillar.item
[DEBUG ] Determining pillar cache
[DEBUG ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'test-minion', 'tcp://salt-master:4506')
[DEBUG ] Connecting the Minion to the Master URI (for the return server): tcp://salt-master:4506
[DEBUG ] Trying to connect to: tcp://salt-master:4506
[DEBUG ] salt.crypt.get_rsa_key: Loading private key
[DEBUG ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG ] salt.crypt.verify_signature: Loading public key
[DEBUG ] salt.crypt.get_rsa_pub_key: Loading public key
[DEBUG ] salt.crypt.verify_signature: Verifying signature
[DEBUG ] Closing AsyncZeroMQReqChannel instance
[DEBUG ] Minion return retry timer set to 8 seconds (randomized)
[INFO ] Returning information for job: 20220609145541104944
[DEBUG ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'test-minion', 'tcp://salt-master:4506')
[DEBUG ] Connecting the Minion to the Master URI (for the return server): tcp://salt-master:4506
[DEBUG ] Trying to connect to: tcp://salt-master:4506
[INFO ] User sudo_dneichev Executing command saltutil.find_job with jid 20220609145546246664
[DEBUG ] Command details {'fun': 'saltutil.find_job', 'arg': ['20220609145541104944'], 'tgt': ['test-minion'], 'jid': '20220609145546246664', 'ret': '', 'tgt_type': 'list', 'user': 'sudo_dneichev'}
[DEBUG ] Subprocess ProcessPayload-Job-20220609145546246664 added
[DEBUG ] Using importlib_metadata to load entry points
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] LazyLoaded yaml.render
[INFO ] Starting a new job 20220609145546246664 with PID 581438
[DEBUG ] Could not LazyLoad direct_call.allow_missing_func: 'direct_call.allow_missing_func' is not available.
[DEBUG ] LazyLoaded saltutil.find_job
[DEBUG ] Minion return retry timer set to 10 seconds (randomized)
[INFO ] Returning information for job: 20220609145546246664
[DEBUG ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'test-minion', 'tcp://salt-master:4506')
[DEBUG ] Connecting the Minion to the Master URI (for the return server): tcp://salt-master:4506
[DEBUG ] Trying to connect to: tcp://salt-master:4506
[DEBUG ] Closing AsyncZeroMQReqChannel instance
[DEBUG ] minion return: {'success': True, 'return': {'ceph_release_codename': 'Nautilus'}, 'retcode': 0, 'jid': '20220609145541104944', 'fun': 'pillar.item', 'fun_args': ['ceph_release_codename', {'pillarenv': 'some_test_env'}]}
[DEBUG ] Closing AsyncZeroMQReqChannel instance
[DEBUG ] minion return: {'success': True, 'return': {}, 'retcode': 0, 'jid': '20220609145546246664', 'fun': 'saltutil.find_job', 'fun_args': ['20220609145541104944']}
[DEBUG ] Subprocess ProcessPayload-Job-20220609145541104944 cleaned up
[DEBUG ] Subprocess ProcessPayload-Job-20220609145546246664 cleaned up
If we compare the output on the working master it creates only one jid, while on the second master it creates two
Expected behavior
The valid output on both masters should be
test-minion:
----------
ceph_release_codename:
knottyliss
Screenshots
If applicable, add screenshots to help explain your problem.
Versions Report
master1
Salt Version:
Salt: 3004.1
Dependency Versions:
cffi: 1.11.5
cherrypy: unknown
dateutil: 2.6.1
docker-py: Not Installed
gitdb: 4.0.9
gitpython: 3.1.18
Jinja2: 2.10.1
libgit2: 0.26.8
M2Crypto: 0.35.2
Mako: Not Installed
msgpack: 0.6.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.14
pycrypto: Not Installed
pycryptodome: Not Installed
pygit2: 0.26.4
Python: 3.6.8 (default, Apr 29 2022, 13:46:02)
python-gnupg: Not Installed
PyYAML: 3.12
PyZMQ: 19.0.0
smmap: 5.0.0
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: almalinux 8.6 Sky Tiger
locale: UTF-8
machine: x86_64
release: 4.18.0-372.9.1.el8.x86_64
system: Linux
version: AlmaLinux 8.6 Sky Tiger
master2
Salt Version:
Salt: 3004.1
Dependency Versions:
cffi: 1.11.5
cherrypy: unknown
dateutil: 2.6.1
docker-py: 1.10.6
gitdb: 4.0.9
gitpython: 3.1.18
Jinja2: 2.10.1
libgit2: Not Installed
M2Crypto: 0.35.2
Mako: Not Installed
msgpack: 0.6.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.14
pycrypto: Not Installed
pycryptodome: Not Installed
pygit2: Not Installed
Python: 3.6.8 (default, Apr 29 2022, 13:46:02)
python-gnupg: 0.4.8
PyYAML: 3.12
PyZMQ: 19.0.0
smmap: 5.0.0
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: almalinux 8.6 Sky Tiger
locale: UTF-8
machine: x86_64
release: 4.18.0-372.9.1.el8.x86_64
system: Linux
version: AlmaLinux 8.6 Sky Tiger
Additional context
None
Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:
- Community Wiki
- Salt’s Contributor Guide
- Join our Community Slack
- IRC on LiberaChat
- Salt Project YouTube channel
- Salt Project Twitch channel
There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!