Authentication can take up to 20mn on self-hosted runners in k8s (GCP) - ReOpen

[Roberto Gutiérrez]

TL;DR

Sometimes it takes 10-20 minutes to run certain setup actions, like google-github-actions/setup-gcloud@v1 and google-github-actions/setup-gcloud@v0

Expected behavior

google-github-actions/setup-gcloud@v1 should take some seconds

Observed behavior

google-github-actions/setup-gcloud@v1 can take up to 20 minutes

Action YAML

name: Development deployment

on:
  workflow_dispatch:
    inputs:
      reason:
        description: 'Why you trigger manually'
        required: true
        type: string
  push:
    branches: [ "BE*", "AS*" ]
  pull_request:
    types: [ closed ]
env:
  PROJECT_ID: "REDACTED"
  LOCATION: "REDACTED"

jobs:
  create:
    name: Deploy composer dags/plugin/data on development
    permissions:
      contents: 'read'
      id-token: 'write'
    runs-on: preemptible-runners
    steps:
      - name: 'Checkout'
        uses: actions/checkout@v3

      - name: 'Google Auth'
        id: 'auth'
        uses: 'google-github-actions/auth@v1'
        with:
          token_format: "access_token"
          workload_identity_provider: '${{ secrets.WIF_PROVIDER_NAME }}'
          service_account: '${{ secrets.GSA_COMPOSER_EMAIL }}'
          project_id: '${{ env.PROJECT_ID }}'

      - name: 'Set up Cloud SDK'
        uses: 'google-github-actions/setup-gcloud@v1'
        with:
          project_id: '${{ env.PROJECT_ID }}'
          install_components: 'alpha'

      - name: 'Gcloud info'
        run: 'gcloud info'

      - name: 'Run tests'
        uses: actions/setup-python@v4
        with:
          python-version: '3.9'
      - run: scripts/composer-admin.sh -v --dev --run-test '${{ secrets.GSA_COMPOSER_JSON_BASE64 }}'

      - name: 'Checking code style'
        uses: actions/setup-python@v4
        with:
          python-version: '3.9'
      - run: scripts/composer-admin.sh -v --dev --check-code-style

      - name: 'Upload dags'
        run: scripts/composer-admin.sh -v --dev --upload-dags

      - name: 'Upload plugins'
        run: scripts/composer-admin.sh -v --dev --upload-plugins

      - name: 'Upload data'
        run: scripts/composer-admin.sh -v --dev --upload-data

### Log output

```text
I'll attach the logs to the issue

Additional information

I'm reopening #620

[Roberto Gutiérrez]

logs_good.zip

These are the logs when the workflow works as expected

I'll upload the "failing ones" ASAP

[Seth Vargo]

Hi @robgutsopedra

I don't think there's anything we can do to speed up the install times. I recommend opening an issue with the gcloud team. There are many reasons why the installation might be slow on self-hosted runners, including poor egress network connection, rate limiting, etc. You can pre-install the SDK on your self-hosted runners and skip the setup-gcloud step entirely if you'd like.

I'm interested to see the logs from one of the long installs, just to see where it's hanging.