gcloud functions deploy fails Container Healthcheck
johnnyoshika opened this issue · comments
TL;DR
When trying to deploy a google cloud function (with runtime python311), the build succeeds (I can see the successful build in GCP Console's Cloud Build page), but container healthcheck fails. This failure started happening sometime be Jan 5, 2024 (the last successful deployment) and May 7, 2024 (the first failed deployment).
The exact command when deploying from my macBook Pro succeeds without any issue. The deployment command is:
gcloud functions deploy essayVerbatim --gen2 --project examind-dev --runtime python311 --region us-central1 --source . --entry-point essay_verbatim --set-env-vars "PYTHON_ENV=production" --set-secrets "{redacted}" --trigger-http --no-allow-unauthenticated --service-account {redacted}@examind-dev.iam.gserviceaccount.com --timeout 300 --memory 4GiB --quiet
On my macBook Pro, I see this successful output:
Preparing function...done.
X Updating function (may take a while)...
✓ [Build] Logs are available at [https://console.cloud.google.com/cloud-build/builds;region=us-central1/b5ed63b8-7367-49ea-8a7b-4c43c85dbfbf?project=485160045751]
[Service]
. [ArtifactRegistry]
. [Healthcheck]
. [Triggercheck]
Completed with warnings:
[INFO] A new revision will be deployed serving with 100% traffic.
You can view your function in the Cloud Console here: https://console.cloud.google.com/functions/details/us-central1/essayVerbatim?project=examind-dev
In GitHub Workflow, I see the following error:
Preparing function...
done.
Updating function (may take a while)...
[Build]............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................done
[Service]..........................................................................................................................................................................................................................................................................................................................................................................................................................................................failed
Failed.
ERROR: (gcloud.functions.deploy) OperationError: code=3, message=Could not create or update Cloud Run service essayverbatim, Container Healthcheck failed. Revision 'essayverbatim-000[22](https://github.com/examind-ai/examind-web/actions/runs/8194276520/job/22448423753#step:9:23)-pok' is not ready and cannot serve traffic. The user-provided container failed to start and listen on the port defined provided by the PORT=8080 environment variable. Logs for this revision might contain more information.
Logs URL: https://console.cloud.google.com/logs/viewer?project=examind-dev&resource=cloud_run_revision/service_name/essayverbatim/revision_name/essayverbatim-00022-pok&advancedFilter=resource.type%3D%22cloud_run_revision%22%0Aresource.labels.service_name%3D%22essayverbatim%22%0Aresource.labels.revision_name%3D%22essayverbatim-00022-pok%22
For more troubleshooting guidance, see https://cloud.google.com/run/docs/troubleshooting#container-failed-to-start
Error: Process completed with exit code 1.
This is what I see in the log page:
2024-03-08 10:23:48.356
Traceback (most recent call last): File "/layers/google.python.pip/pip/bin/functions-framework", line 8, in <module> sys.exit(_cli())
2024-03-08 10:23:48.356
^^^^^^
2024-03-08 10:23:48.356
File "/layers/google.python.pip/pip/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
2024-03-08 10:23:48.356
return self.main(*args, **kwargs)
2024-03-08 10:23:48.356
^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-08 10:23:48.356
File "/layers/google.python.pip/pip/lib/python3.11/site-packages/click/core.py", line 1078, in main
2024-03-08 10:23:48.357
rv = self.invoke(ctx)
2024-03-08 10:23:48.357
^^^^^^^^^^^^^^^^
2024-03-08 10:23:48.357
File "/layers/google.python.pip/pip/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
2024-03-08 10:23:48.357
return ctx.invoke(self.callback, **ctx.params)
2024-03-08 10:23:48.358
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-08 10:23:48.358
File "/layers/google.python.pip/pip/lib/python3.11/site-packages/click/core.py", line 783, in invoke
2024-03-08 10:23:48.358
return __callback(*args, **kwargs)
2024-03-08 10:23:48.358
^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-08 10:23:48.358
File "/layers/google.python.pip/pip/lib/python3.11/site-packages/functions_framework/_cli.py", line 36, in _cli
2024-03-08 10:23:48.358
app = create_app(target, source, signature_type)
2024-03-08 10:23:48.359
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-08 10:23:48.359
File "/layers/google.python.pip/pip/lib/python3.11/site-packages/functions_framework/__init__.py", line 380, in create_app
2024-03-08 10:23:48.359
raise e from None
2024-03-08 10:23:48.359
File "/layers/google.python.pip/pip/lib/python3.11/site-packages/functions_framework/__init__.py", line 361, in create_app
2024-03-08 10:23:48.359
spec.loader.exec_module(source_module)
2024-03-08 10:23:48.360
File "<frozen importlib._bootstrap_external>", line 940, in exec_module
2024-03-08 10:23:48.360
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
2024-03-08 10:23:48.360
File "/workspace/main.py", line 5, in <module>
2024-03-08 10:23:48.360
from services.verbatim_sentences import verbatim_sentences
2024-03-08 10:23:48.360
File "/workspace/services/verbatim_sentences.py", line 1, in <module>
2024-03-08 10:23:48.360
from services.sentences import get_min_distance, get_embeddings, text_to_sentences
2024-03-08 10:23:48.360
File "/workspace/services/sentences.py", line 20, in <module>
2024-03-08 10:23:48.361
minilm_model = SentenceTransformer(minilm_model_dir)
2024-03-08 10:23:48.361
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-08 10:23:48.361
File "/layers/google.python.pip/pip/lib/python3.11/site-packages/sentence_transformers/SentenceTransformer.py", line 95, in __init__
2024-03-08 10:23:48.364
modules = self._load_sbert_model(model_path)
2024-03-08 10:23:48.364
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-08 10:23:48.365
File "/layers/google.python.pip/pip/lib/python3.11/site-packages/sentence_transformers/SentenceTransformer.py", line 840, in _load_sbert_model
2024-03-08 10:23:48.365
module = module_class.load(os.path.join(model_path, module_config['path']))
2024-03-08 10:23:48.365
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-08 10:23:48.365
File "/layers/google.python.pip/pip/lib/python3.11/site-packages/sentence_transformers/models/Pooling.py", line 120, in load
2024-03-08 10:23:48.372
return Pooling(**config)
2024-03-08 10:23:48.372
^^^^^^^^^^^^^^^^^
2024-03-08 10:23:49.391
TypeError: Pooling.__init__() got an unexpected keyword argument 'pooling_mode_weightedmean_tokens'
2024-03-08 10:23:50.638
Container called exit(1).
2024-03-08 10:23:50.879
Default STARTUP TCP probe failed 1 time consecutively for container "worker" on port 8080. The instance was not started.
2024-03-08 10:23:50.896
run.googleapis.com
v1
…nd-dev/revisions/essayverbatim-00022-pok
Ready condition status changed to False for Revision essayverbatim-00022-pok with message: The user-provided container failed to start and listen on the port defined provided by the PORT=8080 environment variable. Logs for this revision might contain more information. Logs URL: https://console.cloud.google.com/logs/viewer?project=examind-dev&resource=cloud_run_revision/service_name/essayverbatim/revision_name/essayverbatim-00022-pok&advancedFilter=resource.type%3D%22cloud_run_revision%22%0Aresource.labels.service_name%3D%22essayverbatim%22%0Aresource.labels.revision_name%3D%22essayverbatim-00022-pok%22 For more troubleshooting guidance, see https://cloud.google.com/run/docs/troubleshooting#container-failed-to-start
Expected behavior
The function to deploy
Observed behavior
The function fails to deploy because of container healthcheck
Action YAML
name: Deploy essayVerbatim
on:
workflow_call:
inputs:
projectId:
required: true
type: string
region:
required: true
type: string
secrets:
gcloudServiceAccount:
required: true
jobs:
deploy_essayVerbatim:
name: Deploy essayVerbatim
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.11'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install nltk sentence-transformers
- name: Download NLTK data
working-directory: cloud/functions/essayVerbatim
run: |
python -m nltk.downloader -d nltk_data punkt
- name: Download Mini LM model
working-directory: cloud/functions/essayVerbatim
run: python ./save_minilm_model.py
- id: auth
name: Authenticate to Google Cloud
uses: google-github-actions/auth@v2
with:
credentials_json: ${{secrets.gcloudServiceAccount}}
- name: Set up Cloud SDK
uses: google-github-actions/setup-gcloud@v2
with:
project_id: ${{inputs.projectId}}
- name: Deploy essayVerbatim cloud function
working-directory: cloud/functions/essayVerbatim
run: >
gcloud functions deploy essayVerbatim
--gen2
--runtime python311
--region ${{inputs.region}}
--source .
--entry-point essay_verbatim
--set-env-vars "PYTHON_ENV=production"
--set-secrets "{redacted}"
--trigger-http
--no-allow-unauthenticated
--service-account {redacted}@${{inputs.projectId}}.iam.gserviceaccount.com
--timeout 300
--memory 4GiB
--quietLog output
Run gcloud functions deploy essayVerbatim --gen2 --runtime python311 --region us-central1 --source . --entry-point essay_verbatim --set-env-vars "PYTHON_ENV=production" --set-secrets "{redacted}" --trigger-http --no-allow-unauthenticated --service-account {redacted}@examind-dev.iam.gserviceaccount.com --timeout 300 --memory 4GiB --quiet
Preparing function...
done.
Updating function (may take a while)...
[Build].........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................done
[Service]....................................................................................................................................................................................................................................................................................................................................................................................................................................failed
Failed.
ERROR: (gcloud.functions.deploy) OperationError: code=3, message=Could not create or update Cloud Run service essayverbatim, Container Healthcheck failed. Revision 'essayverbatim-00025-rob' is not ready and cannot serve traffic. The user-provided container failed to start and listen on the port defined provided by the PORT=8080 environment variable. Logs for this revision might contain more information.
Logs URL: https://console.cloud.google.com/logs/viewer?project=examind-dev&resource=cloud_run_revision/service_name/essayverbatim/revision_name/essayverbatim-00025-rob&advancedFilter=resource.type%3D%22cloud_run_revision%22%0Aresource.labels.service_name%3D%22essayverbatim%22%0Aresource.labels.revision_name%3D%22essayverbatim-00025-rob%22
For more troubleshooting guidance, see https://cloud.google.com/run/docs/troubleshooting#container-failed-to-start
Error: Process completed with exit code 1.
Additional information
The contents of save_minilm_model.py, which is referenced in the action, is this:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')
model.save("./minilm_model")
Does this work outside of setup-gcloud?
gcloud functions deploy essayVerbatim --gen2 --runtime python311 --region us-central1 --source . --entry-point essay_verbatim --set-env-vars "PYTHON_ENV=production" --set-secrets "{redacted}" --trigger-http --no-allow-unauthenticated --service-account {redacted}@examind-dev.iam.gserviceaccount.com --timeout 300 --memory 4GiB --quiet
Yes, it works fine when I run that command from my Terminal in my MacBook Pro. I've included outputs of that in my original message above.
Do you have a gcloudignore ot gitignore file that is preventing files from being present on the GitHub Action runner? Unless it's a different in gcloud versions, this GitHub Action literally just runs gcloud commands :)
My .gitignore of the working directory of the monorepo:
/.env
/venv
/certs
/nltk_data
__pycache__
My .gcloudignore of the working directory of the monorepo:
# Without .gcloudignore, .gitignore will be used by `gcloud functions deploy`,
# which will prevent `nltk_data` folder from deploying.
.gcloudignore
.gitignore
The strangest thing is that it deployed fine from GitHub Actions fine before. The last time it deployed successfully was Jan 5, 2024. Then the next time the workflow file ran was Mar 7, 2024 and it failed. Nothing has changing inside the working directory since other than upgrading node versions, but I tried rolling that back to see if it works and it still doesn't. Here's a history of the workflow running:
Hi @johnnyoshika - sorry but I'm not sure how to help. Without a small reproduction case and access to the entire environment, my ability to assist is limited.
Googling some error messages, it seems like "got an unexpected keyword argument 'pooling_mode_weightedmean_tokens'" is a fairly common error. Forums suggest updating sentence-transformers helps.
@sethvargo Thanks for pointing me in the right direction. I pinned the sentence_transformers package in my GitHub Action to a previous version and it now works again.
I changed:
pip install nltk sentence-transformers
to
pip install nltk==3.8.1 sentence-transformers==2.2.2
I don't think I needed to pin the nltk version as it hasn't changed in a while, but everything is working now.