[Bug] - lambda/nodejs:20 AL2023 based container: aws command doesn't work after installing awscli-2
gtach2o opened this issue · comments
Describe the bug
aws
command doesn't work after installing
To Reproduce
Steps to reproduce the behavior:
FROM public.ecr.aws/lambda/nodejs:20
RUN dnf install -y awscli-2
RUN aws --version
build the image
docker build --platform=linux/amd64 --pull -t lambda-node -f Dockerfile --progress=plain .
fails with error
#6 [3/3] RUN aws --version
#6 0.470 Traceback (most recent call last):
#6 0.470 File "/usr/bin/aws", line 19, in <module>
#6 0.471 import awscli.clidriver
#6 0.471 File "/usr/lib/python3.9/site-packages/awscli/clidriver.py", line 21, in <module>
#6 0.471 import botocore.session
#6 0.471 File "/usr/lib/python3.9/site-packages/awscli/botocore/session.py", line 27, in <module>
#6 0.472 import botocore.client
#6 0.472 File "/usr/lib/python3.9/site-packages/awscli/botocore/client.py", line 16, in <module>
#6 0.472 from botocore import UNSIGNED, waiter, xform_name
#6 0.472 File "/usr/lib/python3.9/site-packages/awscli/botocore/waiter.py", line 17, in <module>
#6 0.472 from botocore.docs.docstring import WaiterDocstring
#6 0.472 File "/usr/lib/python3.9/site-packages/awscli/botocore/docs/__init__.py", line 15, in <module>
#6 0.472 from botocore.docs.service import ServiceDocumenter
#6 0.472 File "/usr/lib/python3.9/site-packages/awscli/botocore/docs/service.py", line 13, in <module>
#6 0.472 from botocore.docs.bcdoc.restdoc import DocumentStructure
#6 0.473 File "/usr/lib/python3.9/site-packages/awscli/botocore/docs/bcdoc/restdoc.py", line 15, in <module>
#6 0.473 from botocore.compat import OrderedDict
#6 0.473 File "/usr/lib/python3.9/site-packages/awscli/botocore/compat.py", line 31, in <module>
#6 0.473 from urllib3 import exceptions
#6 0.473 File "/usr/lib/python3.9/site-packages/urllib3/__init__.py", line 7, in <module>
#6 0.474 from .connectionpool import HTTPConnectionPool, HTTPSConnectionPool, connection_from_url
#6 0.474 File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 11, in <module>
#6 0.474 from .exceptions import (
#6 0.474 File "/usr/lib/python3.9/site-packages/urllib3/exceptions.py", line 2, in <module>
#6 0.474 from .packages.six.moves.http_client import IncompleteRead as httplib_IncompleteRead
#6 0.474 File "/usr/lib/python3.9/site-packages/urllib3/packages/__init__.py", line 3, in <module>
#6 0.474 from . import ssl_match_hostname
#6 0.474 File "/usr/lib/python3.9/site-packages/urllib3/packages/ssl_match_hostname.py", line 1, in <module>
#6 0.474 from ssl import match_hostname, CertificateError
#6 0.474 File "/usr/lib64/python3.9/ssl.py", line 99, in <module>
#6 0.475 import _ssl # if we can't import it, let the error propagate
#6 0.475 ImportError: /usr/lib64/python3.9/lib-dynload/_ssl.cpython-39-x86_64-linux-gnu.so: undefined symbol: COMP_get_type, version OPENSSL_3.0.0
#6 ERROR: process "/bin/sh -c aws --version" did not complete successfully: exit code: 1
------
> [3/3] RUN aws --version:
0.474 from .exceptions import (
0.474 File "/usr/lib/python3.9/site-packages/urllib3/exceptions.py", line 2, in <module>
0.474 from .packages.six.moves.http_client import IncompleteRead as httplib_IncompleteRead
0.474 File "/usr/lib/python3.9/site-packages/urllib3/packages/__init__.py", line 3, in <module>
0.474 from . import ssl_match_hostname
0.474 File "/usr/lib/python3.9/site-packages/urllib3/packages/ssl_match_hostname.py", line 1, in <module>
0.474 from ssl import match_hostname, CertificateError
0.474 File "/usr/lib64/python3.9/ssl.py", line 99, in <module>
0.475 import _ssl # if we can't import it, let the error propagate
0.475 ImportError: /usr/lib64/python3.9/lib-dynload/_ssl.cpython-39-x86_64-linux-gnu.so: undefined symbol: COMP_get_type, version OPENSSL_3.0.0
Expected behavior
aws --version
prints it's version
It's not the awscli-2
that's causing the issue. In a python shell try importing ssl
:
bash-5.2# python
Python 3.9.16 (main, Sep 8 2023, 00:00:00)
[GCC 11.4.1 20230605 (Red Hat 11.4.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib64/python3.9/ssl.py", line 99, in <module>
import _ssl # if we can't import it, let the error propagate
ImportError: /usr/lib64/python3.9/lib-dynload/_ssl.cpython-39-x86_64-linux-gnu.so: undefined symbol: COMP_get_type, version OPENSSL_3.0.0
dnf install -y awscli-2
installs a lot if stuff like perl etc. Why ssl is not a dependency for awscli-2?
also it is not possible to use pip because there will be an error that ssl is not enabled or something like that.
As far as I can tell, this seems to be unique to the lambda containers. Note the following:
public.ecr.aws/amazonlinux/amazonlinux:2023-minimal
bash-5.2# ldd /usr/lib64/python3.9/lib-dynload/_ssl.cpython-39-x86_64-linux-gnu.so
linux-vdso.so.1 (0x00007fffe25df000)
libssl.so.3 => /lib64/libssl.so.3 (0x00007fb0033f4000)
libcrypto.so.3 => /lib64/libcrypto.so.3 (0x00007fb002fbb000)
libc.so.6 => /lib64/libc.so.6 (0x00007fb002db3000)
libz.so.1 => /lib64/libz.so.1 (0x00007fb002d99000)
/lib64/ld-linux-x86-64.so.2 (0x00007fb0034d1000)
bash-5.2# rpm -qf /lib64/libcrypto.so.3
openssl-libs-3.0.8-1.amzn2023.0.10.x86_64
public.ecr.aws/lambda/nodejs:20
bash-5.2# ldd /usr/lib64/python3.9/lib-dynload/_ssl.cpython-39-x86_64-linux-gnu.so
linux-vdso.so.1 (0x00007ffcc5f58000)
libssl.so.3 => /var/lang/lib/libssl.so.3 (0x00007fe4689d2000)
libcrypto.so.3 => /var/lang/lib/libcrypto.so.3 (0x00007fe46835d000)
libc.so.6 => /lib64/libc.so.6 (0x00007fe468155000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fe468150000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fe46814b000)
/lib64/ld-linux-x86-64.so.2 (0x00007fe468ca2000)
bash-5.2# rpm -qf /var/lang/lib/libcrypto.so.3
file /var/lang/lib/libcrypto.so.3 is not owned by any package
So the lambda containers are installing some other libcrypto?
openssl-libs
is present in the lambda container also. If you copy /usr/lib64/libsso.so.3
and /usr/lib64/libcrypto.so.3
and override the files in /var/lang/lib/
then all symbols are resolved and aws --version
works:
bash-5.2# aws --version
aws-cli/2.14.5 Python/3.9.16 Linux/5.15.146.1-microsoft-standard-WSL2 source/x86_64.amzn.2023 prompt/off
However, the files in /var/lang/lib/
are runtime libraries for the lambdas and might nuke the container!
openssl-libs
is present in the lambda container also. If you copy/usr/lib64/libsso.so.3
and/usr/lib64/libcrypto.so.3
and override the files in/var/lang/lib/
then all symbols are resolved andaws --version
works:bash-5.2# aws --version aws-cli/2.14.5 Python/3.9.16 Linux/5.15.146.1-microsoft-standard-WSL2 source/x86_64.amzn.2023 prompt/off
However, the files in
/var/lang/lib/
are runtime libraries for the lambdas and might nuke the container!
I don’t think this can be a solution.
The lambda container has LD_LIBRARY_PATH
environment variable set to /var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib:/opt/lib
so libraries in /var/lang/lib/
will take precedence over the ones in/usr/lib64/
Run printenv
or echo $LD_LIBRARY_PATH
to show the contents.
It's a docker image. The echo output is the same.
My question was that SSL is not properly configured. It is not even possible to run pip command.
IS this only docker image reflated issue. Should I better ask there https://github.com/aws/aws-lambda-base-images ?
It's a docker image. The echo output is the same. My question was that SSL is not properly configured. It is not even possible to run pip command. IS this only docker image reflated issue. Should I better ask there https://github.com/aws/aws-lambda-base-images ?
The image in question is made by the Lambda team on top of the base AL2023 minimal container image - so cutting an issue over there is likely a good idea. Let's keep this one open here as well until we resolve the issue.
Maybe they messed it up by setting LD_LIBRARY_PATH
environment variable value.
I got it working with ENV LD_LIBRARY_PATH=""
workaround.
The nodejs20
is built on top of aws-lambda-base-images
and the Dockerfile sets some environment variables:
FROM scratch
ADD x86_64/22f00718d013305cd61f4906577540cc6804d6603602567f1bff83781ebe6893.tar.xz /
ADD x86_64/88f2931da348e4df07deb80db8d3d1f42f27bac499f4d786543b5d3520b360c8.tar.xz /
ADD x86_64/8ac98ca8d59bf7ca6c4889d72b65a954cf14af58b8e5672c42469dc0c36cd8a4.tar.xz /
ADD x86_64/9a75445380a50339a32e7a9710a586be0a0203193182c5c4a65789c540e99318.tar.xz /
ADD x86_64/e8312a6d39ae266ae459772d2cea9fac729fe7ed9f58f28bc0cccf6d7b227c4c.tar.xz /
ADD x86_64/feb815887f22ce6ea2814b716320e5a1b255f00f3ed05a1f21f0d8640b117208.tar.xz /
ENV LANG=en_US.UTF-8
ENV TZ=:/etc/localtime
ENV PATH=/var/lang/bin:/usr/local/bin:/usr/bin/:/bin:/opt/bin
ENV LD_LIBRARY_PATH=/var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib:/opt/lib
ENV LAMBDA_TASK_ROOT=/var/task
ENV LAMBDA_RUNTIME_DIR=/var/runtime
WORKDIR /var/task
ENTRYPOINT ["/lambda-entrypoint.sh"]
See details at https://raw.githubusercontent.com/aws/aws-lambda-base-images/nodejs20.x/Dockerfile.nodejs20.x
LD_LIBRARY_PATH
is of importance to the node's runtime but not to aws
shell. So when invoking the shell just run LD_LIBRARY_PATH='' aws --version
w/out having to mess the environment set by the publisher.
Oof, this took me a while to find. Hopefully this gets picked up by Google to save someone else time:
If you are on Amazon Linux 2023 and pip install
is failing with an error message:
WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
Then the issue is python unable to import the ssl module, as explained above. To verify the issue, start python
and try to import the ssl module:
$ python
>>> import ssl
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib64/python3.9/ssl.py", line 99, in <module>
import _ssl # if we can't import it, let the error propagate
ImportError: /usr/lib64/python3.9/lib-dynload/_ssl.cpython-39-x86_64-linux-gnu.so: undefined symbol: COMP_get_type, version OPENSSL_3.0.0
The work around for this issue is to prefix pip install
with LD_LIBRARY_PATH=''
as explained above. Such as:
LD_LIBRARY_PATH='' pip install your_package_here
Looks like this was already raised in the lambda project and closed without fixing.