Giters

open-telemetry / opentelemetry-collector

OpenTelemetry Collector

Home Page:https://opentelemetry.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[processor/attributes] unable to add span custom attribute from custom header

amoscatelli opened this issue · comments

commented

Describe the bug
By enabling include_metadata and using attributes processor I don't see http headers from the POST request on the otlp http endpoint added as span attributes.

Steps to reproduce
I have set include_metadata to true in my http otlp receiver
I try to upsert a span custom attribute using attributes processor just like in this readme:

https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/confighttp/README.md

In my case the header is CloudFront-Viewer-Longitude but I don't think that's relevant :

          - key: http.client_latitude
            from_context: CloudFront-Viewer-Latitude
            action: upsert
          - key: http.client_longitude
            from_context: CloudFront-Viewer-Longitude
            action: upsert

Anyway below you'll find my full configuration.

What did you expect to see?
CloudFront-Viewer-Longitude added as http.client_longitude span attribute.

What did you see instead?
http.client_longitude is not added among span attributes.

What version did you use?
otel/opentelemetry-collector-contrib:0.54.0

What config did you use?

receivers:
    otlp:
        protocols:
            http:
                include_metadata: true
                cors:
                    allowed_origins: [
                        ${RECEIVERS_OTLP_PROTOCOLS_HTTP_CORS_ALLOWED_ORIGINS}
                    ]
                    
    # Dummy receiver that's never used, because a pipeline is required to have one.
    otlp/spanmetrics:
        protocols:
            grpc:
                endpoint: "localhost:12345"

    prometheus/spanmetrics:
      config:
        scrape_configs:
          - job_name: 'ratelimiter'
            scrape_interval: 6m
            static_configs:
              - targets: ['localhost:8889']

processors:
    attributes/traces:
        actions:
          - key: http.client_latitude_2
            from_context: metadata.CloudFront-Viewer-Latitude
            action: upsert
          - key: http.client_longitude_2
            from_context: metadata.CloudFront-Viewer-Longitude
            action: upsert
          - key: http.client_latitude
            from_context: CloudFront-Viewer-Latitude
            action: upsert
          - key: http.client_longitude
            from_context: CloudFront-Viewer-Longitude
            action: upsert
          - key: test
            value: 123
            action: upsert
          - key: origin2
            from_context: metadata.origin
            action: upsert
          - key: client_latitude
            from_context: CloudFront-Viewer-Latitude
            action: upsert
          - key: client_longitude
            from_context: CloudFront-Viewer-Longitude
            action: upsert
          - key: client_latitude2
            from_context: cloudfront-viewer-latitude
            action: upsert
          - key: client_longitude2
            from_context: cloudfront-viewer-longitude
            action: upsert
          - key: client_latitude3
            from_context: metadata.cloudfront-viewer-latitude
            action: upsert
          - key: client_longitude3
            from_context: metadata.cloudfront-viewer-longitude
            action: upsert
          - key: client_latitude4
            from_context: metadata.CloudFront-Viewer-Latitude
            action: upsert
          - key: client_longitude4
            from_context: metadata.CloudFront-Viewer-Longitude
            action: upsert
    memory_limiter:
        check_interval: 5s
        limit_mib: 448
        spike_limit_mib: 64
    batch:
        send_batch_size: 48
        send_batch_max_size: 48
        timeout: 15s
    transform:
        logs:
            queries:
                - set(attributes["severity_text"], severity_text)
    filter:
        logs:
            exclude:
                match_type: regexp
                record_attributes:
                    - key: severity_text
                      value: "(TRACE|DEBUG)"
    spanmetrics:
        metrics_exporter: prometheus/spanmetrics
        dimensions:
            - name: enduser.id
            - name: http.status_code
            - name: http.client_ip
            - name: http.client_latitude
            - name: http.client_longitude
            - name: http.method
        aggregation_temporality: "AGGREGATION_TEMPORALITY_DELTA"

exporters:
    logging:
        loglevel: ${EXPORTERS_LOGGING_LOGLEVEL}
    otlp:
        endpoint: tempo-eu-west-0.grafana.net:443
        headers:
            authorization: ##########
            
    prometheusremotewrite:
        endpoint: https://prometheus-prod-01-eu-west-0.grafana.net/api/prom/push
        headers:
            authorization: ##########
            
    loki:
        endpoint: https://logs-prod-eu-west-0.grafana.net/loki/api/v1/push
        headers:
            authorization: ##########
        format: json
        labels:
            attributes:
                container_name: ""
                source: ""
            resource:
                host.name: "hostname"
                
    prometheus/spanmetrics:
        endpoint: "localhost:8889"

extensions:
    health_check:
service:
    telemetry:
        logs:
            level: ${SERVICE_TELEMETRY_LOGS_LEVEL}
    extensions: [health_check]
    pipelines:
        traces:
            receivers: [otlp]
            processors: [attributes/traces, spanmetrics, memory_limiter, batch]
            exporters: [otlp, logging]
        metrics/spanmetrics:
            # This receiver is just a dummy and never used.
            # Added to pass validation requiring at least one receiver in a pipeline.
            receivers: [otlp/spanmetrics]
            # The metrics_exporter must be present in this list.
            exporters: [prometheus/spanmetrics]
        metrics:
            receivers: [otlp, prometheus/spanmetrics]
            processors: [memory_limiter, batch]
            exporters: [prometheusremotewrite]
        logs:
            receivers: [otlp]
            processors: [transform, filter, memory_limiter, batch]
            exporters: [loki]

Environment
AWS Beanstalk with Docker Linux V2

Additional context

I need this to inject AWS Cloudfront geolocation headers into my spans :
https://aws.amazon.com/it/about-aws/whats-new/2020/07/cloudfront-geolocation-headers/

I have put my otel collector behind a non caching Cloudfront, and it is properly receiving POST http requests on v1/traces with those headers containing latitude and logitude.

I am sure everything is received properly because I am running tcpdump on 4318 port on otel collector and I DO see those headers.

Now I need to propagate those values from the http request, containing the spans in its body, to the span themself.

This is what include_metadata is supposed to enable, right ? Did I understand correctly ?

I also tried different combinations, trying to read from metadata context (inspired by https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/attributesprocessor/testdata/config.yaml) but none worked.

The only attribute I see properly set is "test" to value 123, but that's the only key not using "from_context".

Please help.

commented

@jpkrohling sorry to bother you
is the from_context: CloudFront-Viewer-Longitude supposed to work ?
am I doing something wrong or this is an issue indeed ?

commented

+1
The same issue for me. Version 0.54.0

Can't get any header from the request, but hardcoded key as in example above works
this is my configuration

receivers:
  otlp:
    protocols:
      http:
        include_metadata: true
        endpoint: 0.0.0.0:4318
      grpc: 
        endpoint: 0.0.0.0:4317

processors:
  attributes/from_context:
    actions:
      - key: http.x_request_id
        from_context: x_request_id
        action: upsert
exporters:
  logging:
    loglevel: info
  awsxray:
    region: eu-west-1
  awsemf:
    region: eu-west-1

service:
  telemetry:
    logs:
      level: "debug"
  pipelines:
    traces:
      receivers:
        - otlp
      exporters:
        - logging
        - awsxray
      processors: [attributes/from_context]
    metrics:
      receivers:
        - otlp
      exporters:
        - logging
        - awsemf
commented

I can't understand why it's not working, it seemed to be ok just a bunch of monthes ago :
#4901

commented

@amoscatelli I saw you have two kinds of setting,
one is with "metadata." prefix, for example

          - key: http.client_latitude_2
            from_context: metadata.CloudFront-Viewer-Latitude
            action: upsert

another is not with any prefix

          - key: http.client_latitude
            from_context: CloudFront-Viewer-Latitude
            action: upsert

Do these two settings both not work?

commented

@fatsheep9146 by sending traces to logging exporter I could verify that neither http.client_latitude or http.client_latitude_2 were added.

Only 'test' key was present :

          - key: test
            value: 123
            action: upsert

Which one is supposed to work ? With or without metadata prefix ?

commented

@amoscatelli
I check about the doc, but not very carefully. So I'm not really sure. For now, I think the one with prefix should work. But apparently it doesn't. I will try to reproduce this on my mac and see if I can find some clues.

commented

@fatsheep9146 give me a minute, I'll give it another try

commented

@fatsheep9146

Using this setting :

processors:
    attributes/traces:
        actions:
          - key: http.client_latitude
            from_context: CloudFront-Viewer-Latitude
            action: upsert
          - key: http.client_longitude
            from_context: CloudFront-Viewer-Longitude
            action: upsert
          - key: http.client_latitude2
            from_context: metadata.CloudFront-Viewer-Latitude
            action: upsert
          - key: http.client_longitude2
            from_context: metadata.CloudFront-Viewer-Longitude
            action: upsert
          - key: http.client_latitude3
            value: 123
            action: upsert
          - key: http.client_longitude3
            value: 123
            action: upsert

As you can see my OTEL receives requests with CloudFront-Viewer-Latitude header :

image

Looking inside my logs :

[ec2-user@ip-172-31-36-36 ~]$ docker logs 3c336b99b2e1 2>&1 | grep latitude
     -> http.client_latitude3: INT(123)
     -> http.client_latitude3: INT(123)
     -> http.client_latitude3: INT(123)
     -> http.client_latitude3: INT(123)
     -> http.client_latitude3: INT(123)
     -> http.client_latitude3: INT(123)

This confirms that every from_context key attribute are not propagated/properly set, while the hardcoded/valued ones work properly.

Please help !

commented

If you have include_metadata: true in the receiver, I would expect metadata.CloudFront-Viewer-Latitude to work. Are you able to prepare a curl call that would reproduce the problem? I'll then debug and see where the problem is.

commented

This is what I have :

receivers:
    otlp:
        protocols:
            http:
                include_metadata: true
                cors:
                    allowed_origins: [
                        "http://localhost:4200",
                        ###########
                    ]

I'll try to prepare a curl
Thank you @jpkrohling

commented

@jpkrohling here you are :
https://justpaste.it/4nzjd

I just added -H "CloudFront-Viewer-Latitude: 44.44444" and sending some spans generated by my application (copied directly from the payload of a traces request in browser DevTools).

Was this what you needed ?

commented

@jpkrohling this is the output of tcpdump for a real request coming from my application :

POST /v1/traces HTTP/1.1
X-Forwarded-For: 2.118.34.194, 130.176.221.45
X-Forwarded-Proto: http
X-Forwarded-Port: 80
Host: otel.visiontech.cloud
X-Amzn-Trace-Id: Root=1-62c6964e-7fe418027d9aa9667040a970
Content-Length: 2820
User-Agent: Amazon CloudFront
X-Amz-Cf-Id: xo3-xBbnzFR0YbHGDWW-3t7fLa0tvMcz-wjrwqcvG_LmtmxQpWtgTQ==
Via: 1.1 c2375420915a59a31c2575fcfd6cae7a.cloudfront.net (CloudFront)
Content-Type: application/json
CloudFront-Viewer-Latitude: 41.89040
CloudFront-Viewer-Longitude: 12.51260

As you can see CloudFront-Viewer-Latitude and CloudFront-Viewer-Longitude headers are present.
Should I test and try to read other headers with processor attributes ?

Maybe User-Agent ?

Is there any reason why metadata should not be filled with particular headers or in particular conditions ?

commented

New test :

          - key: http.client_latitude
            from_context: CloudFront-Viewer-Latitude
            action: upsert
          - key: http.client_longitude
            from_context: CloudFront-Viewer-Longitude
            action: upsert
          - key: http.client_latitude2
            from_context: metadata.CloudFront-Viewer-Latitude
            action: upsert
          - key: http.client_longitude2
            from_context: metadata.CloudFront-Viewer-Longitude
            action: upsert
          - key: http.client_latitude3
            value: 123
            action: upsert
          - key: http.client_longitude3
            value: 123
            action: upsert
          - key: http.client_latitude4
            from_context: metadata.CloudFront-Viewer-Latitude
            action: insert
          - key: http.client_longitude4
            from_context: metadata.CloudFront-Viewer-Longitude
            action: insert
          - key: http.client_latitude5
            from_context: CloudFront-Viewer-Latitude
            action: insert
          - key: http.client_longitude5
            from_context: CloudFront-Viewer-Longitude
            action: insert
          - key: test1
            from_context: metadata.X-Forwarded-For
            action: upsert
          - key: test2
            from_context: X-Forwarded-For
            action: upsert
          - key: test3
            from_context: metadata.X-Forwarded-For
            action: insert
          - key: test4
            from_context: X-Forwarded-For
            action: insert
          - key: test5
            from_context: metadata.X-Amzn-Trace-Id
            action: upsert
          - key: test6
            from_context: X-Amzn-Trace-Id
            action: upsert
          - key: test7
            from_context: metadata.X-Amzn-Trace-Id
            action: insert
          - key: test8
            from_context: X-Amzn-Trace-Id
            action: insert
          - key: test11
            from_context: metadata.Via
            action: upsert
          - key: test12
            from_context: Via
            action: upsert
          - key: test13
            from_context: metadata.Via
            action: insert
          - key: test14
            from_context: Via
            action: insert

Then this is logged :

Span #5
    Trace ID       : 0275913834ce9a99b5cc0047ded211c9
    Parent ID      : 
    ID             : 28827a6522de56bb
    Name           : click
    Kind           : SPAN_KIND_INTERNAL
    Start time     : 2022-07-07 09:14:05.8470003 +0000 UTC
    End time       : 2022-07-07 09:14:05.8471002 +0000 UTC
    Status code    : STATUS_CODE_UNSET
    Status message : 
Attributes:
     -> component: STRING(user-interaction)
     -> event_type: STRING(click)
     -> target_element: STRING(SPAN)
     -> target_xpath: STRING(//html/body/div[3]/div[2]/div/div/div/mat-option[10]/span)
     -> http.url: STRING(https://calc.optoplus.cloud/#/factory/compute)
     -> http.user_agent: STRING(Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.66 Safari/537.36 Edg/103.0.1264.44)
     -> http.client_latitude3: INT(123)
     -> http.client_longitude3: INT(123)
     -> test1: STRING(2.118.34.194, 130.176.221.40)
     -> test2: STRING(2.118.34.194, 130.176.221.40)
     -> test3: STRING(2.118.34.194, 130.176.221.40)
     -> test4: STRING(2.118.34.194, 130.176.221.40)
     -> test5: STRING(Root=1-62c6a3e0-7a2f843d26dac2c1205716cd)
     -> test6: STRING(Root=1-62c6a3e0-7a2f843d26dac2c1205716cd)
     -> test7: STRING(Root=1-62c6a3e0-7a2f843d26dac2c1205716cd)
     -> test8: STRING(Root=1-62c6a3e0-7a2f843d26dac2c1205716cd)
     -> test11: STRING(2.0 ec2e016357b2a4b61d6fc1a2e7c0826a.cloudfront.net (CloudFront))
     -> test12: STRING(2.0 ec2e016357b2a4b61d6fc1a2e7c0826a.cloudfront.net (CloudFront))
     -> test13: STRING(2.0 ec2e016357b2a4b61d6fc1a2e7c0826a.cloudfront.net (CloudFront))
     -> test14: STRING(2.0 ec2e016357b2a4b61d6fc1a2e7c0826a.cloudfront.net (CloudFront))
Resource SchemaURL: 
Resource labels:
     -> service.name: STRING(optoplus-calc-prod)
     -> telemetry.sdk.language: STRING(webjs)
     -> telemetry.sdk.name: STRING(opentelemetry)
     -> telemetry.sdk.version: STRING(1.2.0)

So the problem is related ONLY to the specific CloudFront-Viewer-Latitude and CloudFront-Viewer-Longitude headers ...
How can this be even possible 😢 ?

image
image

commented

@jpkrohling this is the output of tcpdump for a real request coming from my application :

POST /v1/traces HTTP/1.1
X-Forwarded-For: 2.118.34.194, 130.176.221.45
X-Forwarded-Proto: http
X-Forwarded-Port: 80
Host: otel.visiontech.cloud
X-Amzn-Trace-Id: Root=1-62c6964e-7fe418027d9aa9667040a970
Content-Length: 2820
User-Agent: Amazon CloudFront
X-Amz-Cf-Id: xo3-xBbnzFR0YbHGDWW-3t7fLa0tvMcz-wjrwqcvG_LmtmxQpWtgTQ==
Via: 1.1 c2375420915a59a31c2575fcfd6cae7a.cloudfront.net (CloudFront)
Content-Type: application/json
CloudFront-Viewer-Latitude: 41.89040
CloudFront-Viewer-Longitude: 12.51260

As you can see CloudFront-Viewer-Latitude and CloudFront-Viewer-Longitude headers are present. Should I test and try to read other headers with processor attributes ?

Maybe User-Agent ?

Is there any reason why metadata should not be filled with particular headers or in particular conditions ?

I guess maybe http header value should not be float ? Can you set CloudFront-Viewer-Latitude value to 41 to give it a try?

commented

Config :

          - key: http.client_latitude
            from_context: CloudFront-Viewer-Latitude
            action: upsert
          - key: http.client_longitude
            from_context: CloudFront-Viewer-Longitude
            action: upsert
          - key: http.client_latitude2
            from_context: metadata.CloudFront-Viewer-Latitude
            action: upsert
          - key: http.client_longitude2
            from_context: metadata.CloudFront-Viewer-Longitude
            action: upsert
          - key: http.client_latitude3
            from_context: metadata.X-CloudFront-Viewer-Latitude
            action: upsert
          - key: http.client_longitude3
            from_context: metadata.X-CloudFront-Viewer-Longitude
            action: upsert
          - key: http.client_latitude4
            from_context: X-CloudFront-Viewer-Latitude
            action: upsert
          - key: http.client_longitude4
            from_context: X-CloudFront-Viewer-Longitude
            action: upsert
          - key: test
            from_context: Test
            action: upsert

Curl :

curl 'http://172.31.36.36:4318/v1/traces' \
  -H 'sec-ch-ua: " Not;A Brand";v="99", "Microsoft Edge";v="103", "Chromium";v="103"' \
  -H 'Referer: https://calc.optoplus.cloud/' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.66 Safari/537.36 Edg/103.0.1264.44' \
  -H 'sec-ch-ua-platform: "Windows"' \
  -H 'Content-Type: application/json' \
  -H 'X-CloudFront-Viewer-Latitude: 44.44444' \
  -H 'X-CloudFront-Viewer-Longitude: pippo' \
  -H 'Test: 44.44444' \
  --data-raw '{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"dummy"}},{"key":"telemetry.sdk.language","value":{"stringValue":"webjs"}},{"key":"telemetry.sdk.name","value":{"stringValue":"opentelemetry"}},{"key":"telemetry.sdk.version","value":{"stringValue":"1.2.0"}}],"droppedAttributesCount":0},"instrumentationLibrarySpans":[{"spans":[{"traceId":"6e8bdcda0d523802ad34beab84f8f623","spanId":"265476d1a6cfa81f","name":"longtask","kind":1,"startTimeUnixNano":1657191217063300000,"endTimeUnixNano":1657191217245300200,"attributes":[{"key":"component","value":{"stringValue":"long-task"}},{"key":"longtask.name","value":{"stringValue":"self"}},{"key":"longtask.entry_type","value":{"stringValue":"longtask"}},{"key":"longtask.duration","value":{"intValue":182}},{"key":"longtask.attribution.name","value":{"stringValue":"unknown"}},{"key":"longtask.attribution.entry_type","value":{"stringValue":"taskattribution"}},{"key":"longtask.attribution.start_time","value":{"intValue":0}},{"key":"longtask.attribution.duration","value":{"intValue":0}},{"key":"longtask.attribution.container_type","value":{"stringValue":"window"}},{"key":"longtask.attribution.container_src","value":{"stringValue":""}},{"key":"longtask.attribution.container_id","value":{"stringValue":""}},{"key":"longtask.attribution.container_name","value":{"stringValue":""}}],"droppedAttributesCount":0,"events":[],"droppedEventsCount":0,"status":{"code":0},"links":[],"droppedLinksCount":0},{"traceId":"532be1a850f11ba96980c49a75f921f8","spanId":"5553289df6522b13","name":"longtask","kind":1,"startTimeUnixNano":1657191217255300000,"endTimeUnixNano":1657191217518300200,"attributes":[{"key":"component","value":{"stringValue":"long-task"}},{"key":"longtask.name","value":{"stringValue":"self"}},{"key":"longtask.entry_type","value":{"stringValue":"longtask"}},{"key":"longtask.duration","value":{"intValue":263}},{"key":"longtask.attribution.name","value":{"stringValue":"unknown"}},{"key":"longtask.attribution.entry_type","value":{"stringValue":"taskattribution"}},{"key":"longtask.attribution.start_time","value":{"intValue":0}},{"key":"longtask.attribution.duration","value":{"intValue":0}},{"key":"longtask.attribution.container_type","value":{"stringValue":"window"}},{"key":"longtask.attribution.container_src","value":{"stringValue":""}},{"key":"longtask.attribution.container_id","value":{"stringValue":""}},{"key":"longtask.attribution.container_name","value":{"stringValue":""}}],"droppedAttributesCount":0,"events":[],"droppedEventsCount":0,"status":{"code":0},"links":[],"droppedLinksCount":0}],"instrumentationLibrary":{"name":"@opentelemetry/instrumentation-long-task","version":"0.30.0"}},{"spans":[{"traceId":"9c62ddae5d5f1961bb919c80a8ccc1c2","spanId":"c3798a50ddbf1f89","name":"HTTP POST","kind":3,"startTimeUnixNano":1657191210572800300,"endTimeUnixNano":1657191217243000300,"attributes":[{"key":"http.method","value":{"stringValue":"POST"}},{"key":"http.url","value":{"stringValue":"https://services.optoplus.cloud/optoplus-services-web/api/lens/compute"}},{"key":"http.response_content_length","value":{"intValue":0}},{"key":"http.status_code","value":{"intValue":200}},{"key":"http.status_text","value":{"stringValue":""}},{"key":"http.host","value":{"stringValue":"services.optoplus.cloud"}},{"key":"http.scheme","value":{"stringValue":"https"}},{"key":"http.user_agent","value":{"stringValue":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.66 Safari/537.36 Edg/103.0.1264.44"}}],"droppedAttributesCount":0,"events":[{"timeUnixNano":1657191210572800300,"name":"open","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657191210573500200,"name":"send","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657191210574900200,"name":"fetchStart","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657181768969600300,"name":"domainLookupStart","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657181768969600300,"name":"domainLookupEnd","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657181768969600300,"name":"connectStart","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657181768969600300,"name":"secureConnectionStart","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657181768969600300,"name":"connectEnd","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657181768969600300,"name":"requestStart","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657181768969600300,"name":"responseStart","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657191217063300000,"name":"responseEnd","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657191217243000300,"name":"loaded","attributes":[],"droppedAttributesCount":0}],"droppedEventsCount":0,"status":{"code":0},"links":[],"droppedLinksCount":0},{"traceId":"8c36567b254a0b87462c16829cf52d28","spanId":"d6e3fb36397a117d","name":"HTTP POST","kind":3,"startTimeUnixNano":1657191210577000200,"endTimeUnixNano":1657191217514400300,"attributes":[{"key":"http.method","value":{"stringValue":"POST"}},{"key":"http.url","value":{"stringValue":"https://services.optoplus.cloud/optoplus-services-web/api/lens/compute"}},{"key":"http.response_content_length","value":{"intValue":0}},{"key":"http.status_code","value":{"intValue":200}},{"key":"http.status_text","value":{"stringValue":""}},{"key":"http.host","value":{"stringValue":"services.optoplus.cloud"}},{"key":"http.scheme","value":{"stringValue":"https"}},{"key":"http.user_agent","value":{"stringValue":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.66 Safari/537.36 Edg/103.0.1264.44"}}],"droppedAttributesCount":0,"events":[{"timeUnixNano":1657191210577000200,"name":"open","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657191210577800200,"name":"send","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657191210579900200,"name":"fetchStart","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657181768969600300,"name":"domainLookupStart","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657181768969600300,"name":"domainLookupEnd","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657181768969600300,"name":"connectStart","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657181768969600300,"name":"secureConnectionStart","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657181768969600300,"name":"connectEnd","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657181768969600300,"name":"requestStart","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657181768969600300,"name":"responseStart","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657191217136300300,"name":"responseEnd","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657191217514400300,"name":"loaded","attributes":[],"droppedAttributesCount":0}],"droppedEventsCount":0,"status":{"code":0},"links":[],"droppedLinksCount":0}],"instrumentationLibrary":{"name":"@opentelemetry/instrumentation-xml-http-request","version":"0.29.2"}}]}]}' \
  --compressed

Log :

Span #0
    Trace ID       : 6c8804c5066fa1207302e906b6637c25
    Parent ID      : 
    ID             : b4d6d5d41ebb75b6
    Name           : click
    Kind           : SPAN_KIND_INTERNAL
    Start time     : 2022-07-07 10:48:47.0784003 +0000 UTC
    End time       : 2022-07-07 10:48:47.0787003 +0000 UTC
    Status code    : STATUS_CODE_UNSET
    Status message : 
Attributes:
     -> component: STRING(user-interaction)
     -> event_type: STRING(click)
     -> target_element: STRING(MAT-ICON)
     -> target_xpath: STRING(//html/body/app-root/vt-mat-app-bar/mat-drawer-container/mat-drawer-content/div/app-compute/div/div/form/mat-card/mat-card-actions/button/span/mat-icon)
     -> http.url: STRING(https://calc.optoplus.cloud/#/factory/compute)
     -> http.user_agent: STRING(Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.66 Safari/537.36 Edg/103.0.1264.44)
     -> test: STRING(44.44444)

For god sake it seems a matter of NAME of the header ??? 😠

commented

Yea, I confirm, the issue is related to the name of the header.

'Test' is ok,
'X-CloudFront-Viewer-Latitude' is not ok,
'CloudFront-Viewer-Latitude' is not ok.

@jpkrohling
How can I help on fixing this ?
I don't think I can change the header name from Cloudfront ...

commented

I think I have enough information here to debug the actual code. I'll check and get back to you soon.

commented

Looks like I solved the mystery: the attribute is stored as X-Cloudfront-Viewer-Latitude within the context. Note the lowercase "front" in "CloudFront". Not sure yet why this is so, but you can get it working with a configuration like this:

receivers:
  otlp:
    protocols:
      http:
        include_metadata: true

exporters:
  logging:
    loglevel: debug

processors:
  attributes:
    actions:
      - key: http.client_latitude2
        from_context: metadata.X-Cloudfront-Viewer-Latitude
        action: upsert
extensions:

service:
  extensions:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [attributes]
      exporters: [logging]

Resulting in:

Attributes:
     -> http.method: STRING(POST)
     -> http.url: STRING(https://services.optoplus.cloud/optoplus-services-web/api/lens/compute)
     -> http.response_content_length: INT(0)
     -> http.status_code: INT(200)
     -> http.status_text: STRING()
     -> http.host: STRING(services.optoplus.cloud)
     -> http.scheme: STRING(https)
     -> http.user_agent: STRING(Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.66 Safari/537.36 Edg/103.0.1264.44)
     -> http.client_latitude2: STRING(44.44444)
commented

THANK YOU very much @jpkrohling , going to test it right now ...

EDIT : Yea, this is working properly !

Anyway this should be fixed, the match with the header should be ignore case !!!

X-Cloudfront-Viewer-Latitude and x-cloudfront-viewer-latitude are the same http header, doing otherwise could lead to issues with libraries and sdk IMHO.

commented

I never worked with GO language, I don't know where to start, but if that's the case I can help of course

commented

We had a similar problem when it comes to the "authorization" header, and I thought about normalizing the keys either in lower or upper case: open-telemetry/opentelemetry-collector-contrib#8994 (comment)

Having an accessor somewhere in the path would make it easy to access the normalized version from the context.

commented

I just submitted a new PR that would fix this:

processors:
  attributes:
    actions:
      - key: http.client_latitude
        from_context: metadata.X-CloudFront-Viewer-Latitude
        action: upsert
      - key: http.client_latitude2
        from_context: metadata.X-Cloudfront-Viewer-Latitude
        action: upsert
Attributes:
     -> http.method: STRING(POST)
     -> http.url: STRING(https://services.optoplus.cloud/optoplus-services-web/api/lens/compute)
     -> http.response_content_length: INT(0)
     -> http.status_code: INT(200)
     -> http.status_text: STRING()
     -> http.host: STRING(services.optoplus.cloud)
     -> http.scheme: STRING(https)
     -> http.user_agent: STRING(Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.66 Safari/537.36 Edg/103.0.1264.44)
     -> http.client_latitude: STRING(44.44444)
     -> http.client_latitude2: STRING(44.44444)
commented

Great !