Runtime exited with error with customized Lambda image

[Aaron Yi]

Hi folks

I been exploring our container support in Lambda, for both official container image (public.ecr.aws/lambda/python:3.7) and customized image with RIC/RIE (python:buster). The official image works fine, however customized image failed with error for local invoke curl -X POST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{"payload":"hello world!"}', or Lambda console invoke. Not sure it's dockerfile compose, RIC/RIE or other issue, anyone had been met the same issue?

local env:

$ docker run -p 9000:8080 local-lambda-python3.8-custom-ocr:latest
time="2020-12-28T04:18:10.654" level=info msg="exec '/usr/local/bin/python' (cwd=/opt, handler=app.handler)"START RequestId: ad5c6337-7132-4619-8165-b7456ea4ae01 Version: $LATEST
time="2020-12-28T04:18:13.918" level=info msg="extensionsDisabledByLayer(/opt/disable-extensions-jwigqn8j) -> stat /opt/disable-extensions-jwigqn8j: no such file or directory"
time="2020-12-28T04:18:13.918" level=warning msg="Cannot list external agents" error="open /opt/extensions: no such file or directory"
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/awslambdaric/__main__.py", line 21, in <module>
    main(sys.argv)
  File "/opt/awslambdaric/__main__.py", line 17, in main
    bootstrap.run(app_root, handler, lambda_runtime_api_addr)
  File "/opt/awslambdaric/bootstrap.py", line 416, in run
    event_request = lambda_runtime_client.wait_next_invocation()
  File "/opt/awslambdaric/lambda_runtime_client.py", line 76, in wait_next_invocation
    response_body, headers = runtime_client.next()
AttributeError: 'NoneType' object has no attribute 'next'
Executing 'app.handler' in function directory '/opt'
time="2020-12-28T04:18:14.276" level=warning msg="First fatal error stored in appctx: Runtime.ExitError"
time="2020-12-28T04:18:14.276" level=warning msg="Process 16(python) exited: Runtime exited with error: exit status 1"
time="2020-12-28T04:18:14.276" level=error msg="Init failed" InvokeID= error="Runtime exited with error: exit status 1"
time="2020-12-28T04:18:14.276" level=warning msg="Failed to send default error response: ErrInvalidInvokeID"
time="2020-12-28T04:18:14.277" level=error msg="INIT DONE failed: Runtime.ExitError"
time="2020-12-28T04:18:14.277" level=warning msg="Reset initiated: ReserveFail"

Lambda console:

START RequestId: 0737419e-378e-4989-8d59-e54092369fed Version: $LATEST
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/awslambdaric/__main__.py", line 21, in <module>
    main(sys.argv)
  File "/opt/awslambdaric/__main__.py", line 17, in main
    bootstrap.run(app_root, handler, lambda_runtime_api_addr)
  File "/opt/awslambdaric/bootstrap.py", line 416, in run
    event_request = lambda_runtime_client.wait_next_invocation()
  File "/opt/awslambdaric/lambda_runtime_client.py", line 76, in wait_next_invocation
    response_body, headers = runtime_client.next()
AttributeError: 'NoneType' object has no attribute 'next'
Executing 'app.handler' in function directory '/opt'
END RequestId: 0737419e-378e-4989-8d59-e54092369fed
REPORT RequestId: 0737419e-378e-4989-8d59-e54092369fed	Duration: 5804.95 ms	Billed Duration: 5805 ms	Memory Size: 128 MB	Max Memory Used: 18 MB	
RequestId: 0737419e-378e-4989-8d59-e54092369fed Error: Runtime exited with error: exit status 1
Runtime.ExitError

My dockerfile:

# Define function directory
ARG LAYER_DIR="/opt"

FROM lambci/lambda-base:build as build-image

# Install aws-lambda-cpp build dependencies
RUN yum -y install \
  g++ \
  make \
  cmake \
  unzip \
  libcurl4-openssl-dev

# Include global arg in this stage of the build
ARG LAYER_DIR
# Create function directory
RUN mkdir -p ${LAYER_DIR}
# Install the runtime interface client
RUN pip install \
        --target ${LAYER_DIR} \
        awslambdaric

# Add layer start
ARG AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION:-"cn-northwest-1"}
ARG AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID:-"xxxx"}
ARG AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY:-"xxxx"}
ENV AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION}
ENV AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
ENV AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}

RUN yum install -y curl unzip

RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
RUN unzip awscliv2.zip
RUN ./aws/install

# layer1
RUN curl $(aws lambda get-layer-version-by-arn --arn arn:aws-cn:lambda:cn-northwest-1:xxxx:layer:xxxx:3 --query 'Content.Location' --output text) --output layer.zip
RUN unzip layer.zip -d ${LAYER_DIR}
RUN rm layer.zip

# layer2
RUN curl $(aws lambda get-layer-version-by-arn --arn arn:aws-cn:lambda:cn-northwest-1:xxxx:layer:xxxx:1 --query 'Content.Location' --output text) --output layer.zip
RUN unzip layer.zip -d ${LAYER_DIR}
RUN rm layer.zip
# Add layer done


# Multi-stage build: grab a fresh copy of the base image, use custom image instead of official one
FROM python:buster

# Include global arg in this stage of the build
ARG LAYER_DIR

# Copy in the build image dependencies
WORKDIR ${LAYER_DIR}
COPY --from=build-image ${LAYER_DIR} .

COPY app.py .

ADD https://github.com/aws/aws-lambda-runtime-interface-emulator/releases/latest/download/aws-lambda-rie /usr/bin/aws-lambda-rie
RUN chmod 755 /usr/bin/aws-lambda-rie

COPY entry.sh /
RUN chmod 755 /entry.sh

# Production env
ENTRYPOINT [ "/entry.sh" ]
CMD [ "app.handler" ]

And entry.sh embedded

#!/bin/sh
if [ -z "${AWS_LAMBDA_RUNTIME_API}" ]; then
    exec /usr/bin/aws-lambda-rie /usr/local/bin/python -m awslambdaric $1
else
    exec /usr/local/bin/python -m awslambdaric $1
fi

[Michael Hart]

I think this is an issue for https://github.com/aws/aws-lambda-runtime-interface-emulator