[BUG] Salt-formula don't render pillar from imported yaml map file
schirka opened this issue · comments
Your setup
Formula commit hash / release tag
1.8.5
Versions reports (master & minion)
salt-master:
Salt Version:
Salt: 3002.6
Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: Not Installed
docker-py: Not Installed
gitdb: 0.6.4
gitpython: 1.0.1
Jinja2: 3.0.1
libgit2: Not Installed
M2Crypto: 0.35.2
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: Not Installed
pycryptodome: 3.10.1
pygit2: Not Installed
Python: 3.6.8 (default, Nov 16 2020, 16:55:22)
python-gnupg: Not Installed
PyYAML: 5.4.1
PyZMQ: 22.1.0
smmap: 0.9.0
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: centos 7 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-1160.25.1.el7.x86_64
system: Linux
version: CentOS Linux 7 Core
Salt-minion
Salt Version:
Salt: 3002.6
Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: Not Installed
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.0.1
libgit2: Not Installed
M2Crypto: 0.35.2
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: Not Installed
pycryptodome: 3.10.1
pygit2: Not Installed
Python: 3.6.8 (default, Nov 16 2020, 16:55:22)
python-gnupg: Not Installed
PyYAML: 5.4.1
PyZMQ: 22.1.0
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: centos 7 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-1160.25.1.el7.x86_64
system: Linux
version: CentOS Linux 7 Core
Pillar / config used
/etc/salt/master
gitfs_provider: gitpython
gitfs_remotes:
- https://github.com/saltstack-formulas/salt-formula
pillar_roots:
base:
- /srv/pillar
---
Bug details
Describe the bug
salt['pillar.get'] is not rendered by import_yaml
map.jinja(line 6,7.8)
...
{% import_yaml "salt/osfamilymap.yaml" as osfamilymap %}
{% import_yaml "salt/osmap.yaml" as osmap %}
{% import_yaml "salt/osfingermap.yaml" as osfingermap %}
...
<osfamilymap.yaml(line 5)>
...
{%- set py_ver_repr = salt['pillar.get']('salt:py_ver', '') %}
...
<osmap.yaml(line 5)>
...
{%- set py_ver_repr = salt['pillar.get']('salt:py_ver', '') %}
...
<osfingermap.yaml(line 5)>
...
{%- set py_ver_repr = salt['pillar.get']('salt:py_ver', '') %}
...
Steps to reproduce the bug
$ salt-call state.apply salt.minion -l debug
> local:
> Data failed to compile:
> ----------
> Rendering SLS 'base:salt.minion' failed: Jinja variable 'salt' is undefined
> /var/cache/salt/minion/files/base/salt/osfamilymap.yaml(5):
> ---
> # -*- coding: utf-8 -*-
> # vim: ft=yaml
> ---
>
> {%- set py_ver_repr = salt['pillar.get']('salt:py_ver', '') %} <======================
>
> {%- set osrelease = salt['grains.get']('osrelease', '') %}
> {%- set salt_release = salt['pillar.get']('salt:release', 'latest') %}
> {%- if salt_release.split('.')|length >= 3 %}
> {%- set salt_release = 'archive/' ~ salt_release %}
> [...]
> ---
> ----------
Expected behaviour
Expect the formula to render
Attempts to fix the bug
For salt['pillar.get'] to be rendered by import_yaml, it needs with context
map.jinja(line 6,7.8)
...
{% import_yaml "salt/osfamilymap.yaml" as osfamilymap with context %}
{% import_yaml "salt/osmap.yaml" as osmap with context %}
{% import_yaml "salt/osfingermap.yaml" as osfingermap with context %}
...
Additional context
salt-formula is now rendering properly.
There is an issue with the use of Jinja2 3.0 saltstack/salt#60188.
Thanks @baby-gnu that helped a lot.
We've ended up downgrading jinja2 after the salt bootstrap in vagrant and terraform provisionner:
pip3 install jinja2==2.11.3
I think we could close this bug report, right?
I have a similar import_yaml issue on opensuse leap 15.4 minion running the same salt 3002.6 version as @schirka. I tried downgrading jinja2==2.11.3 and jinja2==2.8.1 matching with jinja2 version from its master but could not get passed the error.
master:
# salt --versions-report
Salt Version:
Salt: 3002.6
Dependency Versions:
cffi: Not Installed
cherrypy: unknown
dateutil: Not Installed
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.8.1
libgit2: Not Installed
M2Crypto: 0.33.0
Mako: Not Installed
msgpack: 0.6.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: Not Installed
pycryptodome: Not Installed
pygit2: Not Installed
Python: 3.6.8 (default, Nov 16 2020, 16:55:22)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 17.0.0
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.1.4
System Versions:
dist: centos 7 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-1160.42.2.el7.x86_64
system: Linux
version: CentOS Linux 7 Core
minion:
# salt --versions-report
Salt Version:
Salt: 3002.6
Dependency Versions:
cffi: 1.13.2
cherrypy: Not Installed
dateutil: Not Installed
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.0.3
libgit2: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.5
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.17
pycrypto: Not Installed
pycryptodome: 3.17
pygit2: Not Installed
Python: 3.6.15 (default, Sep 23 2021, 15:41:43) [GCC]
python-gnupg: Not Installed
PyYAML: 6.0
PyZMQ: 25.0.2
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: opensuse 15.4
locale: UTF-8
machine: x86_64
release: 5.14.21-150400.24.33-default
system: Linux
version: openSUSE Leap 15.4
map.jinja:
{## Start imports ##}
{% import_yaml "salt/defaults.yaml" as defaults %}
{% import_yaml "salt/osfamilymap.yaml" as osfamilymap %}
{% import_yaml "salt/osmap.yaml" as osmap %}
state.apply error:
----------
Rendering SLS 'base:salt.master' failed: Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/jinja2/debug.py", line 162, in fake_traceback
exec(code, globals, locals)
File "/var/cache/salt/minion/files/base/salt/osfamilymap.yaml", line 5, in top-level template code
{% set ospyver = salt['grains.filter_by'](ospyvermap, salt['grains.get']('os_family')) or {} %}
File "/usr/lib/python3.6/site-packages/jinja2/sandbox.py", line 303, in getitem
#: .. versionadded:: 2.6
jinja2.exceptions.UndefinedError: 'salt' is undefined
I have tested downgrading Jinja2 version starting from 3.0.3 down to 2.8.1 without success. Here is a list from opensuse minion
(from versions: 2.0rc1, 2.0, 2.1, 2.1.1, 2.2, 2.2.1, 2.3, 2.3.1, 2.4, 2.4.1, 2.5, 2.5.1, 2.5.2, 2.5.3, 2.5.4, 2.5.5, 2.6, 2.7, 2.7.1, 2.7.2, 2.7.3, 2.8, 2.8.1, 2.9, 2.9.1, 2.9.2, 2.9.3, 2.9.4, 2.9.5, 2.9.6, 2.10, 2.10.1, 2.10.2, 2.10.3, 2.11.0, 2.11.1, 2.11.2, 2.11.3, 3.0.0a1, 3.0.0rc1, 3.0.0rc2, 3.0.0, 3.0.1, 3.0.2, 3.0.3)
I have tried to match PyYAML version 3.11 on the minion too without success.
Besides Jinja2 and PyYAML package what else should I try to overcome this error? Any suggestion is greatly appreciated. Thanks in advance.
I figured it out. Adding with context to each import_yaml line in map.jinja resolved my issue and it works with Jinja2 3.0.3.
{## Start imports ##}
{% import_yaml "salt/defaults.yaml" as defaults with context %}
{% import_yaml "salt/osfamilymap.yaml" as osfamilymap with context %}
{% import_yaml "salt/osmap.yaml" as osmap with context %}