Jinja 2.7.2 on RHEL 7.9 vs. v1.0.1?

Question

Jinja 2.7.2 on RHEL 7.9 vs. v1.0.1?

bunchrt opened this issue 3 years ago · comments

bunchrt commented 3 years ago

on RHEL 7.9, this task cannot be bypassed by either:

cat mpg_playbook.yml

hosts: all
become: yes
become_user: root
roles:
- { role: MindPointGroup.RHEL7-STIG }
  vars:
  rhel_07_020680: false

...or in the packer config:

    {
        "type": "ansible-local",
        "playbook_file": "mpg_playbook.yml",
        "galaxy_file": "mpg_requirements.yml",
        "extra_arguments": [
            "-vvv",
            "--skip-tags [RHEL-07-020680]"
                ]
    },

...neither approach successfully bypasses this check.

I've tried updating Jinja2 from 2.7.2 w/ pip, but that seems to just clobber Jinja2 altogether. (python2-jinja package contents inside site-packages is gone). I'm --assuming-- that Jinja is the issue at hand, as it's at the bottom of the stack trace, and I've found related issues at https://stackoverflow.com/questions/44660161/how-to-solve-templateruntimeerror-no-test-named-equalto-in-ansible.

Confirmed this is against v1.0.1. python 2.7.5, python 3.6.8.

./roles/MindPointGroup.RHEL7-STIG/meta/.galaxy_install_info:{install_date: 'Mon Jun 14 01:20:03 2021', version: v1.0.1}

amazon-ebs: TASK [MindPointGroup.RHEL7-STIG : MEDIUM | RHEL-07-020680 | AUDIT | The Red Hat Enterprise Linux operating system must be configured so that all files and directories contained in local interactive user home directories have a mode of 0750 or less permissive.] ***
amazon-ebs: task path: /tmp/packer-provisioner-ansible-local/60c6a6e1-af29-4125-01d3-495d8d271665/roles/MindPointGroup.RHEL7-STIG/tasks/fix-cat2.yml:1211
amazon-ebs: The full traceback is:
amazon-ebs: Traceback (most recent call last):
amazon-ebs:   File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 105, in run
amazon-ebs:     items = self._get_loop_items()
amazon-ebs:   File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 232, in _get_loop_items
amazon-ebs:     convert_bare=False)
amazon-ebs:   File "/usr/lib/python2.7/site-packages/ansible/utils/listify.py", line 33, in listify_lookup_plugin_terms
amazon-ebs:     terms = templar.template(terms.strip(), convert_bare=convert_bare, fail_on_undefined=fail_on_undefined)
amazon-ebs:   File "/usr/lib/python2.7/site-packages/ansible/template/__init__.py", line 618, in template
amazon-ebs:     disable_lookups=disable_lookups,
amazon-ebs:   File "/usr/lib/python2.7/site-packages/ansible/template/__init__.py", line 877, in do_template
amazon-ebs:     res = j2_concat(rf)
amazon-ebs:   File "<template>", line 12, in root
amazon-ebs:   File "/usr/lib/python2.7/site-packages/jinja2/filters.py", line 740, in do_list
amazon-ebs:     return list(value)
amazon-ebs:   File "/usr/lib/python2.7/site-packages/jinja2/filters.py", line 839, in do_map
amazon-ebs:     for item in seq:
amazon-ebs:   File "/usr/lib/python2.7/site-packages/jinja2/filters.py", line 930, in _select_or_reject
amazon-ebs:     for item in seq:
amazon-ebs:   File "/usr/lib/python2.7/site-packages/jinja2/filters.py", line 931, in _select_or_reject
amazon-ebs:     if modfunc(func(transfunc(item))):
amazon-ebs:   File "/usr/lib/python2.7/site-packages/jinja2/filters.py", line 925, in <lambda>
amazon-ebs:     name, item, args, kwargs)
amazon-ebs:   File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 438, in call_test
amazon-ebs:     raise TemplateRuntimeError('no test named %r' % name)
amazon-ebs: TemplateRuntimeError: no test named '>='
amazon-ebs: fatal: [127.0.0.1]: FAILED! => {
amazon-ebs:     "msg": "Unexpected failure during module execution.",
amazon-ebs:     "stdout": ""
amazon-ebs: }

this task also blows up (previously disabled it).

amazon-ebs: TASK [MindPointGroup.RHEL7-STIG : MEDIUM | RHEL-07-020600 | AUDIT | The Red Hat Enterprise Linux operating system must be configured so that all local interactive users have a home directory assigned in the /etc/passwd file.] ***
amazon-ebs: task path: /tmp/packer-provisioner-ansible-local/60c6ade7-a15a-81a5-1cd0-b1adc4733058/roles/MindPointGroup.RHEL7-STIG/tasks/fix-cat2.yml:1016
amazon-ebs: <127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: maintuser
amazon-ebs: <127.0.0.1> EXEC /bin/sh -c 'echo ~maintuser && sleep 0'
amazon-ebs: <127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/maintuser/.ansible/tmp `"&& mkdir "` echo /home/maintuser/.ansible/tmp/ansible-tmp-1623633774.75-14206-74568539780571 `" && echo ansible-tmp-1623633774.75-14206-74568539780571="` echo /home/maintuser/.ansible/tmp/ansible-tmp-1623633774.75-14206-74568539780571 `" ) && sleep 0'
amazon-ebs: Using module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py
amazon-ebs: <127.0.0.1> PUT /home/maintuser/.ansible/tmp/ansible-local-10694P3fPfX/tmpUFxFWv TO /home/maintuser/.ansible/tmp/ansible-tmp-1623633774.75-14206-74568539780571/AnsiballZ_command.py
amazon-ebs: <127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/maintuser/.ansible/tmp/ansible-tmp-1623633774.75-14206-74568539780571/ /home/maintuser/.ansible/tmp/ansible-tmp-1623633774.75-14206-74568539780571/AnsiballZ_command.py && sleep 0'
amazon-ebs: <127.0.0.1> EXEC /bin/sh -c 'sudo -H -S -n  -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-tdxdcpokanaoxlicxcznykrsqdsmrtvj ; /usr/bin/python /home/maintuser/.ansible/tmp/ansible-tmp-1623633774.75-14206-74568539780571/AnsiballZ_command.py'"'"' && sleep 0'
amazon-ebs: <127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/maintuser/.ansible/tmp/ansible-tmp-1623633774.75-14206-74568539780571/ > /dev/null 2>&1 && sleep 0'
amazon-ebs: fatal: [127.0.0.1]: FAILED! => {
amazon-ebs:     "msg": "The conditional check 'rhel7stig_07_20600_audit | length > 0' failed. The error was: An unhandled exception occurred while templating '{{ rhel7stig_passwd | selectattr('uid', '>=', rhel7stig_int_gid) | selectattr('id', 'in', ld_users) | list }}'. Error was a <class 'jinja2.exceptions.TemplateRuntimeError'>, original message: no test named '>='"
amazon-ebs: }

George Nalen · Answer 1 · Fri Jul 02 2021 00:49:25 GMT+0800 (China Standard Time)

@bunchrt,
I think you are correct that this is indeed a jinja2 issue. You mentioned that you tried to upgrade from version 2.7.2 but ran into issues. That is a fairly old version of jinja2 (released January 2014). I'm thinking the best way forward would be to upgrade to a newer version of jinja2, basically first figure out the issue preventing the upgrade and then upgrade. If you go that route you could utilize a venv so you don't interfere with your main environment and other setup. I have also tried to replicate this issue using more current versions of jinja2 and don't have the same problem.

-George

bunchrt · Answer 2 · Fri Jul 02 2021 01:51:49 GMT+0800 (China Standard Time)

RHEL 7 (staying in the mainstream of available RPMs) means I'm kinda stuck on jinja2 2.7.2.

if I step aside from the RPM and try to do a pip upgrade, it actually leaves things in a much messier place. now granted, I'm not a python guy -- but I know when I did a pip upgrade for jinja2, it appears to bring down the latest, but it blew up other stuff that was in place via the RPM --- leaving jinja2 basically lobotomized.

can you share the means by which you successfully upgraded jinja2 to 2.8++? and as root or non-priv'd user?

uk-bolly · Answer 3 · Mon Jul 05 2021 23:48:12 GMT+0800 (China Standard Time)

Hi @bunchrt

There are a few ways to be able to achieve the requested results, we do however understand you maybe restricted by what you are able to do within your environments and on your control node with regard to support restrictions and access. While we can't advise which route you can take below are a few options that may work for you

Probably the most commonly used is python virtual environments.
This would give you the ability to run isolated instances of ansible(any python package) using differing python packages without affecting the system supplied rpm files. Often used to test differing version of products before upgrading. e.g able to test different ansible versions to help better manage impact at upgrade time.
These can be managed by a pip requirements file also so you can maintain control over which versions are within each environment.
A good introduction and relative explanation can be found here
There are several good sites that maybe able to help you.
This maybe worth running in conjunction with a python3 install (see below), giving you alot more ability to test.

Another alternative you may have it to install python3 alongside your existing python2 installation, using RPM's (again dependant on the restrictions within your environment) you can then install the appropriate python3 rpms or use pip3 to again isolate the the files between to 2 versions.

This could be achieved by using rhscl (redhat software collections) and enabling more options. This may also help as if covered they will support this software collection provided files. RH dev blog.

Or if you are able to use an alternate approved repository install alongside existing python 2.
3rd party repo py3 setup

I hope that goes so way to help.

uk-bolly

bunchrt · Answer 4 · Tue Aug 10 2021 22:46:43 GMT+0800 (China Standard Time)

was finally able to circle back to this; no joy. I captured the playbook from /tmp, placed it in a temp instance that was prepped, stood up the virtualenv (python2 style, which resulted in python3 inside), ran the command line for running the playbook, and resulted in the same behavior. In this case, trying the later 1.1.0 playbook.

[root@...]# . ~maintuser/python-virtual-environments/env2/bin/activate
(env2) [root@...]# cd /tmp/packer-provisioner-ansible-local/61128610-c64d-839d-6135-f677ee816bcd
(env2) [root@...]# ANSIBLE_FORCE_COLOR=1 PYTHONUNBUFFERED=1 ansible-playbook /tmp/packer-provisioner-ansible-local/61128610-c64d-839d-6135-f677ee816bcd/mpg_playbook.yml --extra-vars "packer_build_name=amazon-ebs packer_builder_type=amazon-ebs packer_http_addr= -o IdentitiesOnly=yes" -vvv --skip-tags=none -c local -i /tmp/packer-provisioner-ansible-local/61128610-c64d-839d-6135-f677ee816bcd/packer-provisioner-ansible-local802522285

`
fatal: [127.0.0.1]: FAILED! => {
"msg": "The conditional check 'rhel7stig_07_20600_audit | length > 0' failed. The error was: An unhandled exception occurred while templating '{{ rhel7stig_passwd | selectattr('uid', '>=', rhel7stig_int_gid) | selectattr('id', 'in', ld_users) | list }}'. Error was a <class 'jinja2.exceptions.TemplateRuntimeError'>, original message: no test named '>='"
}

:
:

(env2) [root@...]# echo $PATH
/home/maintuser/python-virtual-environments/env2/bin:/usr/local/sbin:/sbin:/bin:/usr/sbin:/usr/bin:/opt/aws/bin:/root/bin
(env2) [root@...]# cd ~maintuser/python-virtual-environments/env2
(env2) [root@...]# ls
bin lib lib64 pyvenv.cfg
(env2) [root@...]# ls lib*/py*/site*
lib64/python3.6/site-packages:
markupsafe MarkupSafe-2.0.1.dist-info

lib/python3.6/site-packages:
_distutils_hack pkg_resources _virtualenv.py
distutils-precedence.pth pycache wheel
jinja2 setuptools wheel-0.36.2.dist-info
Jinja2-3.0.1.dist-info setuptools-57.4.0.dist-info wheel-0.36.2.virtualenv
pip setuptools-57.4.0.virtualenv
pip-21.2.3.dist-info _virtualenv.pth
`

uk-bolly · Answer 5 · Wed Aug 11 2021 20:51:05 GMT+0800 (China Standard Time)

hi @bunchrt

This appears to still be complaining about jinja2 being an issue.
This maybe easier if we have a direct contact to try and assist, you can ask us a question here as part of a trial of our counselor product.

https://www.mindpointgroup.com/cybersecurity-products/ansible-counselor
(link to ask us a question at the bottom).

We should be able to directly connect with you, so we can discuss as we go if you think this will help?

If you raise the call and start the issue as per this github link, hopefully we can get you going.

it will only accept non mass mailer email addresses e.g. gmail, outlook etc

Thanks

uk-bolly

Michael Angel · Answer 6 · Fri Feb 04 2022 05:16:22 GMT+0800 (China Standard Time)

@bunchrt I'm having this same issue on RHEL 7.9 with the latest updates. Where you able to get this resolved?

As a workaround, can we get all of these item tagged so we can skip these checks and do the remainder of the role?
setting the variable rhel_07_020680 does not bypass these issues as not all of the lasts are tagged.

uk-bolly · Answer 7 · Fri Feb 04 2022 22:50:10 GMT+0800 (China Standard Time)

hi @Michael-Angel-Sec

If you can let us know the versions of ansible/jinja2 you are running.
I am making a change now in the stig_v3r6 branch that may fix it as a way round these some versions.

I would throughly recommend updating jinja2 if you are able noting that the RHEL version may not be the latest, if the python2-jinja package is installed. If possible look at the Centos repo appears to have one of a later version.

There is a push toward ansible on python3 that may also get round some of the incompatibility issues.

regards

uk-bolly

JP · Answer 8 · Fri Jul 08 2022 10:24:40 GMT+0800 (China Standard Time)

Hello @Michael-Angel-Sec,
v1.3.2

I've found a way to resolve this issue. This was due to Jinja2 that was suck on 2.7 which isn't supported. You have to update it to 2.8+ but more errors came up with pip so I had to upgrade pip to 20.3 and upgrade setuptools and from there, I didn't have the issue.

This is what I've wrote:

"sudo yum -y install python-pip", # Added this due to pip command not found
"sudo pip install --upgrade pip==20.3", # Try pip install --upgrade pip==20.3. pip 21.0 and later no longer support Python 2. See https://pip.pypa.io/en/stable/development/release-process/#python-2-support
"sudo pip install --upgrade setuptools",
"sudo pip install --upgrade Jinja2"

Hopefully, this helps!
-JP

uk-bolly · Answer 9 · Wed Apr 12 2023 16:04:31 GMT+0800 (China Standard Time)

Thank you @jmp9 @Michael-Angel-Sec @bunchrt

This appears to have now been resolved I will close this issue.