Error: NoCredentialProviders when using AWS X-Ray Exporter
smarza opened this issue · comments
Describe the bug
I am encountering an issue with the AWS OpenTelemetry Collector when trying to export traces to AWS X-Ray. The collector fails with a NoCredentialProviders: no valid providers in chain
error. Below are the details of my setup and the error logs.
Steps to reproduce
Docker Compose Configuration
version: '3.9'
services:
aws-otel-collector:
image: public.ecr.aws/aws-observability/aws-otel-collector:latest
command: ["--config=/etc/otel-agent-config.yaml"]
environment:
- AWS_PROFILE=erm-development
- AWS_REGION=eu-west-1
volumes:
- ./config.yaml:/etc/otel-agent-config.yaml
- ${USERPROFILE}/.aws:/home/aoc/.aws
ports:
- "4317:4317"
- "13133:13133"
OpenTelemetry Collector Configuration (config.yaml)
extensions:
health_check:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
processors:
batch/traces:
timeout: 1s
send_batch_size: 50
batch/metrics:
timeout: 60s
exporters:
logging:
loglevel: debug
awsxray:
region: 'eu-west-1'
awsemf:
region: 'eu-west-1'
log_group_name: 'log_group_name'
log_stream_name: 'log_stream_name'
namespace: 'namespace'
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch/traces]
exporters: [awsxray]
metrics:
receivers: [otlp]
processors: [batch/metrics]
exporters: [awsemf]
extensions: [health_check]
telemetry:
logs:
level: debug
Error Logs
Here are the relevant logs from the AWS OpenTelemetry Collector:
2024-05-16T11:02:15.002Z error exporterhelper/common.go:292 Exporting failed. Rejecting data. {"kind": "exporter", "data_type": "traces", "name": "awsxray", "error": "NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors", "rejected_items": 1}
AWS ~/.aws/config:
[profile erm-development]
sso_start_url = https://xxxxxxx/start
sso_region = eu-west-1
sso_account_id = xxxxxxxxxxxx
sso_role_name = xxxdev-Dev
AWS ~/.aws/credentials:
[iam-user]
aws_access_key_id=XXXXXXXXXXXXXXXX
aws_secret_access_key=xxxxxx
aws_session_token=xxxxx
region=eu-west-1
[erm-dev]
region=eu-west-1
role_arn=arn:aws:iam::xxxxxxxxxxxx:role/DeveloperRole
source_profile=iam-user
toolkit_artifact_guid=xxxxxxxx
Make sure you have logged with aws sso login using the correct profile.
Start the Collector with Docker Compose:
podman-compose down
podman-compose up -d
Send a Trace:
You can use otel-cli or a similar tool to send a trace to the collector. Below is an example using otel-cli:
otel-cli span --service-name test-service --endpoint http://localhost:4317 --name "test-span"
Check Logs
podman-compose logs -f aws-otel-collector
What did you expect to see?
I expected the AWS OpenTelemetry Collector to successfully authenticate using the provided AWS profile (erm-development
) and to send trace data to AWS X-Ray without encountering any credential errors.
What did you see instead?
Instead of the expected behavior, I encountered the following error in the logs:
2024-05-16T11:02:15.002Z error exporterhelper/common.go:292 Exporting failed. Rejecting data. {"kind": "exporter", "data_type": "traces", "name": "awsxray", "error": "NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors", "rejected_items": 1}
Environment
AWS OpenTelemetry Collector version: latest
Operating System: Windows 11
Client: Podman Engine
Version: 5.0.1
API Version: 5.0.1
Go Version: go1.22.1
Git Commit: 946d055df324e4ed6c1e806b561af4740db4fea9
Built: Mon Apr 1 16:15:15 2024
OS/Arch: windows/amd64
Server: Podman Engine
Version: 5.1.0-dev-4817811cb
API Version: 5.1.0-dev-4817811cb
Go Version: go1.21.9
Built: Wed Apr 17 02:00:00 2024
OS/Arch: linux/amd64
I managed to get it working by adding some environment variables in the collector container, specifying the locations for AWS_CONFIG_FILE and AWS_SHARED_CREDENTIALS_FILE.
The Docker Compose file now looks like this:
version: '3.9'
services:
aws-otel-collector:
image: public.ecr.aws/aws-observability/aws-otel-collector:latest
command: ["--config=/etc/otel-agent-config.yaml"]
environment:
- AWS_PROFILE=erm-dev
- AWS_REGION=eu-west-1
- AWS_SDK_LOAD_CONFIG=1
- AWS_CONFIG_FILE=/home/aoc/.aws/config
- AWS_SHARED_CREDENTIALS_FILE=/home/aoc/.aws/credentials
volumes:
- ./config.yaml:/etc/otel-agent-config.yaml
- ${USERPROFILE}/.aws/config:/home/aoc/.aws/config:ro
- ${USERPROFILE}/.aws/credentials:/home/aoc/.aws/credentials:ro
ports:
- "4317:4317" # Port for OTLP gRPC
- "13133:13133" # Port for health check
I noticed that the getting started documentation does not mention the need to specify these variables.
Does anyone know what is the reason for requiring this extra configuration?
This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.
This issue was closed because it has been marked as stale for 30 days with no activity.