Not defining a global provider.runtime breaks dockerizePip: true

Question

Not defining a global provider.runtime breaks dockerizePip: true

ToneVDB opened this issue 10 months ago · comments

Tone Vanden Bergh commented 10 months ago

Are you certain it's a bug?

Yes, it looks like a bug

Are you using the latest plugin release?

Yes, I'm using the latest plugin release

Is there an existing issue for this?

I have searched existing issues, it hasn't been reported yet

Issue description

We have a project that uses different runtimes per function:
function1: docker
function2: python3.9

Because of this, in the provider section, we did not define a runtime setting, as we did this on an individual function level.
(package.individually was set to true)
When deploying this stack it caused no issue at all, untill we needed to set the dockerizePip to true

Doing this caused the following error to appear:

× Stack application-dev failed to deploy (2s)
Environment: linux, node 18.16.1, framework 3.33.0 (local), plugin 6.2.3, SDK 4.3.2
Credentials: Local, environment variables
Docs:        docs.serverless.com
Support:     forum.serverless.com
Bugs:        github.com/serverless/serverless/issues

Error:
Running "docker run --rm -v /home/vsts/.cache/serverless-python-requirements/312994c1479362d239d9d5bf26421488a6a04688248779b8d3d3ec9ec8d32241_x86_64_slspyc:/var/task:z -v /home/vsts/.cache/serverless-python-requirements/downloadCacheslspyc:/var/useDownloadCache:z public.ecr.aws/sam/build-undefined:latest-x86_64 /bin/sh -c chown -R 0\:0 /var/useDownloadCache && python -m pip install -t /var/task/ -r /var/task/requirements.txt --cache-dir /var/useDownloadCache && chown -R 1001\:122 /var/task && chown -R 1001\:122 /var/useDownloadCache" failed with: "Unable to find image 'public.ecr.aws/sam/build-undefined:latest-x86_64' locally
docker: Error response from daemon: repository public.ecr.aws/sam/build-undefined not found: name unknown: The repository with name 'build-undefined' does not exist in the registry with id 'sam'.
See 'docker run --help'."
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Tracing back the error caused me to look where public.ecr.aws/sam/build- was used in the repository.
It led me here
As in our usecase serverless.service.provider.runtime was not set on a project lvl, it caused the code to replace this with the string undefined.

This image ofcourse does not exist, causing the docker command to crash out.
Setting the provider.runtime string to python3.9 fixes this issue.

Service configuration (serverless.yml) content

service: application

frameworkVersion: '3'

package:
  individually: true
  patterns:
    - '!./**/**'
    - .requirements.zip

provider:
  name: aws
  stage: ${sls:stage}
  timeout: 900 
  memorySize: 2048

  # if you do not define it here, it breaks the dockerizePip: true
  # remove the comment on the below line to fix the error
  # runtime: python3.9

  environment:
    ENV: ${sls:stage}
    DEPLOYED: TRUE

  # Lambda needs access to S3, secretsmanager, dynamodb and logs
  iam:
    role:
      statements:
        - Effect: Allow
          Action:
            - "s3:*"
          Resource: "*"

        - Effect: Allow
          Action:
            - "secretsmanager:GetSecretValue"
          Resource: "*"

        - Effect: Allow
          Action:
            - "dynamodb:*"
          Resource: "*"

        - Effect: Allow
          Action:
            - "logs:*"
          Resource: "*"

  # Serverless stores our docker image of the app in ECR
  ecr:
    images:
      appimage_scrape:
        path: ./
        file: scrape.dockerfile

plugins:
  - serverless-python-requirements

custom:
  pythonRequirements:
    usePoetry: true
    dockerizePip: true
    requirePoetryLockFile: true
    zip: true
    slim: true
    strip: false
    

functions:
  ma_scrape:
    environment:
      ENV: ${sls:stage}
    image:
      name: appimage_scrape
    events:
      # Invoke Lambda function every day at 6:15h UTC from Mon-Fri
      - schedule: cron(15 6 ? * MON-FRI *)

  ma_clean:
    handler: src.handler.run_clean
    runtime: python3.9
    environment:
      ENV: ${sls:stage}
    events:
      - s3:
          bucket: ${self:service}-${sls:stage}-databucket
          event: s3:ObjectCreated:*
          rules:
            - prefix: raw/
          existing: true
          forceDeploy: true
    package:
      patterns:
        - src/application/clean/**
        - src/application/common/**
        - src/handler.py

  ma_score:
    handler: src.handler.run_score
    runtime: python3.9
    environment:
      ENV: ${sls:stage}
    events:
      - s3:
          bucket: ${self:service}-${sls:stage}-databucket
          event: s3:ObjectCreated:*
          rules:
            - prefix: clean/
            - suffix: info.json
          existing: true
          forceDeploy: true
    package:
      patterns:
        - src/application/score/**
        - src/application/common/**
        - src/handler.py

  ma_match:
    handler: src.handler.run_match
    runtime: python3.9
    environment:
      ENV: ${sls:stage}
    events:
      # Invoke Lambda function every day at 6:45h UTC from Mon-Fri
      - schedule: cron(45 6 ? * MON-FRI *)
    
    package:
      patterns:
        - src/application/match/**
        - src/application/common/**
        - src/handler.py

  ma_send:
    handler: src.handler.run_send
    runtime: python3.9
    environment:
      ENV: ${sls:stage}
    events:
      # Invoke Lambda function every day at 7h UTC from Mon-Fri
      - schedule: cron(0 7 ? * MON-FRI *)
    
    package:
      patterns:
        - src/application/send/**
        - src/application/common/**
        - src/handler.py

Command name and used flags

serverless deploy --verbose --region eu-west-1 --stage dev

Command output

$ /home/vsts/work/1/s/node_modules/.bin/serverless deploy --verbose --region eu-west-1 --stage dev

Deploying application to stage dev (eu-west-1)

Adding Python requirements helper to .
Generating requirements.txt from poetry.lock
Parsed requirements.txt from pyproject.toml in /home/vsts/work/1/s/.serverless/requirements.txt
Installing requirements from "/home/vsts/.cache/serverless-python-requirements/312994c1479362d239d9d5bf26421488a6a04688248779b8d3d3ec9ec8d32241_x86_64_slspyc/requirements.txt"
Docker Image: public.ecr.aws/sam/build-undefined:latest-x86_64
Using download cache directory /home/vsts/.cache/serverless-python-requirements/downloadCacheslspyc
Running docker run --rm -v /home/vsts/.cache/serverless-python-requirements/312994c1479362d239d9d5bf26421488a6a04688248779b8d3d3ec9ec8d32241_x86_64_slspyc\:/var/task\:z -v /home/vsts/.cache/serverless-python-requirements/downloadCacheslspyc\:/var/useDownloadCache\:z public.ecr.aws/sam/build-undefined\:latest-x86_64 /bin/sh -c 'chown -R 0\\:0 /var/useDownloadCache && python -m pip install -t /var/task/ -r /var/task/requirements.txt --cache-dir /var/useDownloadCache && chown -R 1001\\:122 /var/task && chown -R 1001\\:122 /var/useDownloadCache'...

× Stack application-dev failed to deploy (2s)
Environment: linux, node 18.16.1, framework 3.33.0 (local), plugin 6.2.3, SDK 4.3.2
Credentials: Local, environment variables
Docs:        docs.serverless.com
Support:     forum.serverless.com
Bugs:        github.com/serverless/serverless/issues

Error:
Running "docker run --rm -v /home/vsts/.cache/serverless-python-requirements/312994c1479362d239d9d5bf26421488a6a04688248779b8d3d3ec9ec8d32241_x86_64_slspyc:/var/task:z -v /home/vsts/.cache/serverless-python-requirements/downloadCacheslspyc:/var/useDownloadCache:z public.ecr.aws/sam/build-undefined:latest-x86_64 /bin/sh -c chown -R 0\:0 /var/useDownloadCache && python -m pip install -t /var/task/ -r /var/task/requirements.txt --cache-dir /var/useDownloadCache && chown -R 1001\:122 /var/task && chown -R 1001\:122 /var/useDownloadCache" failed with: "Unable to find image 'public.ecr.aws/sam/build-undefined:latest-x86_64' locally
docker: Error response from daemon: repository public.ecr.aws/sam/build-undefined not found: name unknown: The repository with name 'build-undefined' does not exist in the registry with id 'sam'.
See 'docker run --help'."
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
yarn run v1.22.19
warning package.json: No license field

Environment information

Framework Core: 3.33.0 (local)
Plugin: 6.2.3
SDK: 4.3.2


serverless-python-requirements: 6.0.0