Giters

pulumi / actions

Deploy continuously to your cloud of choice, using your favorite language, Pulumi, and GitHub!

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

PULUMI_ACCESS_TOKEN must be set error when using azure blob as backend

dkezri opened this issue · comments

commented

What happened?

Hi,

I m trying to use azure blob as backend that is required by my organization, but i get error:

PULUMI_ACCESS_TOKEN must be set for login during non-interactive CLI sessions

i have this environment variables

env:
AZCLI_VERSION: latest
AZURE_WRITE: false
DOTNET_VERSION: 7.0.x
AZURE_REGION: norwayeast
PULUMI_WORK_DIR: PulumiAzure
PULUMI_STACK_NAME: dev
DOTNET_ROOT: /usr/share/dotnet
AZURE_HTTP_USER_AGENT:
AZUREPS_HOST_ENVIRONMENT:
AZURE_STORAGE_ACCOUNT: pulumistateaccountdev
AZURE_CONTAINER_NAME: pulumistatecontainerdev
AZURE_STORAGE_KEY: rI/otrVwLuo0WrV+GUC7V5azr23RVc56AewKt5wv10SLDnXqpH4I**********
AZURE_STORAGE_SAS_TOKEN: se=.******************
ARM_USE_OIDC: true
ARM_CLIENT_ID: ***
ARM_TENANT_ID: ***
ARM_SUBSCRIPTION_ID: ***

Configured range: ^3
/opt/hostedtoolcache/pulumi/3.77.1/x64/pulumi version
v3.77.1
Pulumi version 3.77.1 is already installed on this machine. Skipping download
Logging into azblob://pulumistatecontainerdev

/home/runner/work/_actions/pulumi/actions/v4/webpack:/pulumi-github-action/node_modules/@pulumi/pulumi/automation/errors.js:77
: new CommandError(result);
^
CommandError: code: -2
stdout:
stderr: Command failed with exit code 255: pulumi stack select --stack dev --non-interactive
error: PULUMI_ACCESS_TOKEN must be set for login during non-interactive CLI sessions
err?: Error: Command failed with exit code 255: pulumi stack select --stack dev --non-interactive
error: PULUMI_ACCESS_TOKEN must be set for login during non-interactive CLI sessions

at Object.createCommandError (/home/runner/work/_actions/pulumi/actions/v4/webpack:/pulumi-github-action/node_modules/@pulumi/pulumi/automation/errors.js:77:1)
at Object.<anonymous> (/home/runner/work/_actions/pulumi/actions/v4/webpack:/pulumi-github-action/node_modules/@pulumi/pulumi/automation/cmd.js:76:1)
at Generator.throw (<anonymous>)
at rejected (/home/runner/work/_actions/pulumi/actions/v4/webpack:/pulumi-github-action/node_modules/@pulumi/pulumi/automation/cmd.js:19:1)
at processTicksAndRejections (node:internal/process/task_queues:96:5)

Expected Behavior

the pulumi state is not created in azure blob

Steps to reproduce

i use github action with azure cli

Output of pulumi about

Configured range: ^3
/opt/hostedtoolcache/pulumi/3.77.1/x64/pulumi version
v3.77.1
Pulumi version 3.77.1 is already installed on this machine. Skipping download
Logging into azblob://pulumistatecontainerdev

/home/runner/work/_actions/pulumi/actions/v4/webpack:/pulumi-github-action/node_modules/@pulumi/pulumi/automation/errors.js:77
: new CommandError(result);
^
CommandError: code: -2
stdout:
stderr: Command failed with exit code 255: pulumi stack select --stack dev --non-interactive
error: PULUMI_ACCESS_TOKEN must be set for login during non-interactive CLI sessions
err?: Error: Command failed with exit code 255: pulumi stack select --stack dev --non-interactive
error: PULUMI_ACCESS_TOKEN must be set for login during non-interactive CLI sessions

at Object.createCommandError (/home/runner/work/_actions/pulumi/actions/v4/webpack:/pulumi-github-action/node_modules/@pulumi/pulumi/automation/errors.js:77:1)
at Object.<anonymous> (/home/runner/work/_actions/pulumi/actions/v4/webpack:/pulumi-github-action/node_modules/@pulumi/pulumi/automation/cmd.js:76:1)
at Generator.throw (<anonymous>)
at rejected (/home/runner/work/_actions/pulumi/actions/v4/webpack:/pulumi-github-action/node_modules/@pulumi/pulumi/automation/cmd.js:19:1)
at processTicksAndRejections (node:internal/process/task_queues:96:5)

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

commented

@dkezri I was talking to someone else about this yesterday. If you use PULUMI_BACKEND_URL and set that to be whatever value you're setting cloud-url to be, this should work.

I think this is still an issue so it should be left open for now, but that's at least a workaround for now

commented

I wonder if this is just an actions issue? It seems to work fine locally on the command line.
Will have a check of the actions code.

commented

Moved to the GH action repo

commented

@dkezri I was talking to someone else about this yesterday. If you use PULUMI_BACKEND_URL and set that to be whatever value you're setting cloud-url to be, this should work.

I think this is still an issue so it should be left open for now, but that's at least a workaround for now

I am using this configuration in my git hub action:

  - name: setup azure pulumi state resources
    uses: azure/powershell@v1
    with:
      inlineScript: |
       ./pulumi_azure_blob_setup.ps1
      azPSVersion: "latest"  
    env:
      Environment: dev
      ARM_CLIENT_ID: ${{ secrets.ARM_CLIENT_ID }}
      ARM_TENANT_ID: ${{ secrets.ARM_TENANT_ID }}
      ARM_SUBSCRIPTION_ID: ${{ secrets.ARM_SUBSCRIPTION_ID }}
     
  - uses: pulumi/actions@v4
    with:
      command: up
      github-token: ${{ secrets.GIT_ACCESS_TOKEN }}
      stack-name: ${{ env.PULUMI_STACK_NAME }}
      work-dir: ${{ env.PULUMI_WORK_DIR }}
      cloud-url: azblob://${{ env.AZURE_CONTAINER_NAME }}
      upsert: true
    env:
      ARM_USE_OIDC: true
      ARM_CLIENT_ID: ${{ secrets.ARM_CLIENT_ID }}
      ARM_TENANT_ID: ${{ secrets.ARM_TENANT_ID }}
      ARM_SUBSCRIPTION_ID: ${{ secrets.ARM_SUBSCRIPTION_ID }}
commented

If you updated that and added PULUMI_BACKEND_URL to the env section in the Pulumi action:

- uses: pulumi/actions@v4
    with:
      command: up
      github-token: ${{ secrets.GIT_ACCESS_TOKEN }}
      stack-name: ${{ env.PULUMI_STACK_NAME }}
      work-dir: ${{ env.PULUMI_WORK_DIR }}
      cloud-url: azblob://${{ env.AZURE_CONTAINER_NAME }}
      upsert: true
    env:
      ARM_USE_OIDC: true
      ARM_CLIENT_ID: ${{ secrets.ARM_CLIENT_ID }}
      ARM_TENANT_ID: ${{ secrets.ARM_TENANT_ID }}
      ARM_SUBSCRIPTION_ID: ${{ secrets.ARM_SUBSCRIPTION_ID }}
      PULUMI_BACKEND_URL: azblob://${{ env.AZURE_CONTAINER_NAME }}

That should work

commented

@pierskarsenbarg
i did exactly what you suggested above, but now it gives another error:

Logging into azblob://pulumistatecontainerdev

/home/runner/work/_actions/pulumi/actions/v4/webpack:/pulumi-github-action/node_modules/@pulumi/pulumi/automation/errors.js:77
: new CommandError(result);
^
CommandError: code: -2
stdout:
stderr: Command failed with exit code 255: pulumi stack select --stack dev --non-interactive
error: read ".pulumi/meta.yaml": blob (key ".pulumi/meta.yaml") (code=Unknown): ===== RESPONSE ERROR (ErrorCode=InvalidAuthenticationInfo) =====
Description=Authentication information is not given in the correct format. Check the value of Authorization header.
RequestId:08d50ab4-601e-0047-04ce-ca147f000000
Time:2023-08-09T14:36:57.0675933Z, Details: (none)

err?: Error: Command failed with exit code 255: pulumi stack select --stack dev --non-interactive
error: read ".pulumi/meta.yaml": blob (key ".pulumi/meta.yaml") (code=Unknown): ===== RESPONSE ERROR (ErrorCode=InvalidAuthenticationInfo) =====
Description=Authentication information is not given in the correct format. Check the value of Authorization header.
RequestId:08d50ab4-601e-0047-04ce-ca147f000000

commented

Have you managed to get this working locally? This looks like incorrect auth tokens, but that should be simpler to verify locally than running through GHA.

commented

@Frassle this is only an github action issue, it works when i am using #PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }} then it create state in pulumi backend and also creates new resources in Azure, my point is to used Azure blob as backend

commented

Yes, but what I'm asking is if you've managed to get it creating stacks in Azure on your local machine?

That is if you try and pulumi login azblob://pulumistatecontainerdev with the full envvars above:

AZURE_STORAGE_ACCOUNT: pulumistateaccountdev
AZURE_CONTAINER_NAME: pulumistatecontainerdev
AZURE_STORAGE_KEY: rI/otrVwLuo0WrV+GUC7V5azr23RVc56AewKt5wv10SLDnXqpH4I**********
AZURE_STORAGE_SAS_TOKEN: se=.******************

Does that login successfully?

commented

I think your access keys are invalid. Double check with the azure CLI as well, as something like az storage blob list --container-name pulumistatecontainerdev might give better errors than the raw http of the gocloud library.

commented

@Frassle i get information about container:
PS C:\Users\dkezri> az storage blob list --container-name pulumistatecontainerdev --account-name pulumistateaccountdev

There are no credentials provided in your command and environment, we will query for account key for your storage account.
It is recommended to provide --connection-string, --account-key or --sas-token in your command as credentials.

You also can add --auth-mode login in your command to use Azure Active Directory (Azure AD) for authorization if your login account is assigned required RBAC roles.
For more information about RBAC roles in storage, visit https://docs.microsoft.com/azure/storage/common/storage-auth-aad-rbac-cli.

In addition, setting the corresponding environment variables can avoid inputting credentials in your command. Please use --help to get more information about environment variable usage.
[
{
"container": "pulumistatecontainerdev",
"content": "",
"deleted": null,
"encryptedMetadata": null,
"encryptionKeySha256": null,
"encryptionScope": null,
"hasLegalHold": null,
"hasVersionsOnly": null,
"immutabilityPolicy": {
"expiryTime": null,
"policyMode": null
},
"isAppendBlobSealed": null,
"isCurrentVersion": null,
"lastAccessedOn": null,
"metadata": {},

commented

Hitting the limits of what I can help with here, azure auth isn't something I've worked with much.
I'll see if I can get someone with more Azure knowledge to comment here, but overall I don't believe this to be a bug in pulumi, just a problem setting up the correct environment.

commented

Joining the discussion because I am hitting the same problem. I also suspect this is GA issue, because I have the same setup working locally with Pulumi.

Locally, I do not use storage key or SAS token. I only use CLI login with RBAC and it works flawlessly.

This is my workflow:

jobs:
  deploy:
    name: Deploy Infrastructure
    environment: dev
    steps:
      - uses: azure/login@v1
        with:
          client-id: ${{ secrets.AZURE_CLIENT_ID }}
          tenant-id: ${{ secrets.AZURE_TENANT_ID }}
          subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
      - name: Test Federated Credentials Login
        shell: bash
        run: |
          az account show -o table
          az storage account list -o table
          az keyvault list -o table
      - uses: actions/checkout@v3
      - uses: pulumi/actions@v4
        with:
          command: preview
          stack-name: dev
          work-dir: deployments/platform
          cloud-url: ${{ env.PULUMI_BACKEND_URL }}
          secrets-provider: ${{ env.PULUMI_SECRETS_PROVIDER }}

OIDC login with Azure CLI gives the correct permissions, identity is able to read all the necessary resources, but pulumi actions fails with error:

/usr/bin/tar xz --warning=no-unknown-keyword --overwrite -C /home/runner/.pulumi -f /runner/_work/_temp/cd9f01fb-0f4c-41d7-9edf-ad5dff61c39e
Logging into azblob://stacks

/runner/_work/_actions/pulumi/actions/v4/webpack:/pulumi-github-action/node_modules/@pulumi/pulumi/automation/errors.js:77
                    : new CommandError(result);
^
CommandError: code: -2
 stdout: 
 stderr: Command failed with exit code 255: pulumi stack select --stack dev --non-interactive
error: read ".pulumi/meta.yaml": blob (key ".pulumi/meta.yaml") (code=Unknown): ===== RESPONSE ERROR (ErrorCode=AuthorizationPermissionMismatch) =====
Description=This request is not authorized to perform this operation using this permission.
RequestId:c[36](***)ba214-a01e-0064-01d5-dcd9d1000000
Time:2023-09-01T13:08:22.22[38](***)768Z, Details: (none)

 err?: Error: Command failed with exit code 255: pulumi stack select --stack dev --non-interactive
error: read ".pulumi/meta.yaml": blob (key ".pulumi/meta.yaml") (code=Unknown): ===== RESPONSE ERROR (ErrorCode=AuthorizationPermissionMismatch) =====
Description=This request is not authorized to perform this operation using this permission.
RequestId:c36ba214-a01e-0064-01d5-dcd9d1000000
Time:2023-09-01T13:08:22.2238768Z, Details: (none)


    at Object.createCommandError (/runner/_work/_actions/pulumi/actions/v4/webpack:/pulumi-github-action/node_modules/@pulumi/pulumi/automation/errors.js:77:1)
    at Object.<anonymous> (/runner/_work/_actions/pulumi/actions/v4/webpack:/pulumi-github-action/node_modules/@pulumi/pulumi/automation/cmd.js:76:1)
    at Generator.throw (<anonymous>)
    at rejected (/runner/_work/_actions/pulumi/actions/v4/webpack:/pulumi-github-action/node_modules/@pulumi/pulumi/automation/cmd.js:19:1)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

Identity has the same roles as my account

commented

I tried using the pulumi CLI in run steps and it seems it has the same authorization error. It must be related to OIDC authentication (federated credentials), because pulumi won't be able to read the blob storage, regardless of the assigned roles.

For reference, the roles assigned to the identity are:

  • Contributor (subscription)
  • Storage Blob Data Contributor (storage account)
  • Storage Queue Data Contributor (storage account)
  • Storage File Data Privileged Contributor (storage account)
commented

Looking into this a bit today, the error message "PULUMI_ACCESS_TOKEN must be set for login during non-interactive CLI sessions" can only be hit if the CLI isn't in "filestate" mode.
It determines that by reading the current backend url from "~/.pulumi/credentials.json", which the action should be setting by calling pulumi login.

So somehow the backend information is getting lost.

commented

I had the same issue when used AWS S3 bucket as a backend, and I can confirm that setting $PULUMI_BACKEND_URL env variable resolved the problem for me.

commented

I fixed it by removing the step of automation using "./pulumi_azure_blob_setup.ps1" script from the github action.
So i ended up to create the azure storage account and container manually using Azure portal, also added my oidc application to the keyvault policy in order to read and decrypt the key. now everything is working fine. here is my github action for deploying to Azure dev environment:

name: Deploy to dev environment on Azure
on:
  workflow_dispatch:
  push:
    branches: [ "development" ]

permissions:
  id-token: write
  contents: read

env:
  AZCLI_VERSION: "latest"
  DOTNET_VERSION: 7.0.x
  PULUMI_WORK_DIR: "./PulumiAzure"
  PULUMI_STACK_NAME: dev
  AZURE_RESOURCE_GROUP: pulumi-state-dev-rg
  AZURE_STORAGE_ACCOUNT: pulumidevst
  AZURE_CONTAINER_NAME: dashboard-state-dev-blobc

jobs:
  CI:
     name: CI
     uses: ./.github/workflows/ci-base.yaml
     with:
       runs-on: '["ubuntu-latest"]'
       dotnet-version: 7.0.x

  Dev:
    name: Deploy to dev
    needs: CI
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
        with:
          token: ${{ secrets.GIT_ACCESS_TOKEN }}
      
      - name: Setup DotNet
        uses: actions/setup-dotnet@v3
        with:
          dotnet-version: ${{ env.DOTNET_VERSION }}

      - name: Azure CLI Login
        uses: azure/login@v1
        with:
          client-id: ${{ vars.AZURE_CLIENT_ID_DEV }}
          tenant-id: ${{ vars.AZURE_TENANT_ID }}
          subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID_DEV }} 

      - name: Azure CLI script
        uses: azure/CLI@v1
        with:
          azcliversion: ${{ env.AZCLI_VERSION }}
          inlineScript: |
            AZURE_STORAGE_KEY=$(az storage account keys list -n ${{ env.AZURE_STORAGE_ACCOUNT }} -g ${{ env.AZURE_RESOURCE_GROUP }} --query '[0].value' -o tsv)
            echo "AZURE_STORAGE_KEY=$AZURE_STORAGE_KEY" >> $GITHUB_ENV

      - name: pulumi up step
        uses: pulumi/actions@v4
        with:
          command: up
          stack-name: ${{ env.PULUMI_STACK_NAME }}
          cloud-url: azblob://${{ env.AZURE_CONTAINER_NAME }}
          work-dir: ${{ env.PULUMI_WORK_DIR }}
        env:
          ARM_USE_OIDC: true
          ARM_CLIENT_ID: ${{ vars.AZURE_CLIENT_ID_DEV }}
          ARM_TENANT_ID: ${{ vars.AZURE_TENANT_ID }}
          ARM_SUBSCRIPTION_ID: ${{ vars.AZURE_SUBSCRIPTION_ID_DEV }}
          AZURE_STORAGE_ACCOUNT: ${{ env.AZURE_STORAGE_ACCOUNT }}
          AZURE_STORAGE_KEY: ${{ env.AZURE_STORAGE_KEY }}
          AZURE_KEYVAULT_AUTH_VIA_CLI: true
      - run: echo "Azure instructure is now deployed to dev environment"
      - run: az logout