custom segments are not getting flushed

[Jim Vergin]

we had the following setup where custom segments were not getting flushed:

Api Gateway invokes lambda A.

lambda A had roughly the following code:

const AWSXRay = require('aws-xray-sdk-core');
const AWS = AWSXRay.captureAWS(require('aws-sdk'));

exports.handler = async (event, context) => {

    let segment = AWSXRay.getSegment(); //returns the facade segment
    console.log('add subsegment');
    let subsegment = segment.addNewSubsegment('userinformation');
    subsegment.addMetadata('cognitouseridsub', sub);
    subsegment.addAnnotation('cognitouserid', sub);
    subsegment.addAnnotation('emailaddress', email);
    console.log('subsegment', JSON.stringify(subsegment));
    subsegment.close();
    
    synchronously invoke lambda B
}

lambda B had roughly the following code:

const AWSXRay = require('aws-xray-sdk-core');
const AWS = AWSXRay.captureAWS(require('aws-sdk'));

exports.handler = async (event, context) => {

  invoke an npm module

}

the npm module had roughly the following code

const AWSXRay = require('aws-xray-sdk-core');
const AWS = AWSXRay.captureAWS(require('aws-sdk'));

async function npmFunction(args) {
// implementation of the actual npm function call
}

With this setup/code when the segment was added to lambda A, it was not flushed and x-ray debug reported:

"Ignoring flush on subsegment 17e9f81d3f6c283d. Associated segment is marked as not sampled."

If I do not include the code:

const AWSXRay = require('aws-xray-sdk-core');
const AWS = AWSXRay.captureAWS(require('aws-sdk'));

in lambda B and the called npm modules, the added user segment is successfully added. Is this expected behavior? Is the best way to allow the creation of the custom segment to only initialize the x-ray sdk in the top level lambda? If we start using RUM does this change how we should use the x-ray sdk in the lambdas?

[Carol Abadeer]

Hi @jmv12345,
Is it possible to share your Lambda function logs so we can get a better understanding of the behavior here? What is the sampling decision on the parent segments in Lambda A and in Lambda B, and are you creating all of the subsegments as sampled subsegments (using the addNewSubsegment API)?

[Jim Vergin]

Lambda A is invoked via API Gateway. API Gateway is set to enable xray tracing. API Gateway is using an xray sampling rule (currently the default rule) which specifies that 100% of the requests should be handled.

Our lambdas were calling npm modules that had dependencies on the aws-xray-sdk-core specified in the module package.json and package-lock.json files. Due to the way that the lambdas and npm modules were being built, multiple versions of the aws-xray-sdk-core module were being included when a lambda was deployed.

I discovered that having multiple versions of aws-xray-sdk-core in a deployed lambda resulted in many subsegments not getting flushed. We eliminated all dependencies on aws-xray-sdk-core in package.json and package-lock.json files for the npm modules themselves. Now only the lambda package.json and package-lock.json files include dependencies on aws-xray-sdk-core. This ensures that only a single version of aws-xray-sdk-core is included in a deployed lambda. With this approach all subsegments are properly flushed, including the custom segment.

In lambda B the xray specific code being included now is:

const AWSXRay = require('aws-xray-sdk-core');
AWSXRay.captureHTTPsGlobal(require('https'));

Only lambda A specifies the capturing of aws sdk calls, using the code:

const AWSXRay = require('aws-xray-sdk-core');
const AWS = AWSXRay.captureAWS(require('aws-sdk'));

(Lambda A does not make https calls.)

None of the npm modules specify that the aws-sdk calls should be captured.

With the above changes, a trace for lambda A includes all aws sdk calls made by lambda A, plus aws sdk and https calls made in lambda B. And the custom segment added in lambda A is visible in the trace. The custom segment is added using the code:

segment = AWSXRay.getSegment(); //returns the facade segment
subsegment = segment.addNewSubsegment('userinformation');
subsegment.addAnnotation('cognitouserid', sub);
subsegment.addAnnotation('emailaddress', email);
subsegment.close();

Our web client is now instrumented with RUM. With the use of RUM, a trace for lambda A starts with the web client, proceeds through API Gateway, includes all aws sdk calls made by lambda A, and then all aws sdk and https calls made by lambda B.

After enabling cloudwatch logs on API Gateway, viewing traces in cloudwatch includes the logs for API Gateway, the logs for lambda A and the logs for lambda B.

With these configurations, the tracing is working as I expected. Please comment if there are any other adjustments that I should make. Otherwise I think we can close this issue.

[Carol Abadeer]

Great to hear the issue has been resolved! Generally, using different versions of the aws-xray-sdk packages can sometimes cause issues because some functionality differs from one version to the next, so ensuring that the same version is being used across microservices (and even lambda layers) can help eliminate any of these issues!

Thanks for sharing your findings and the resolution; I'll close out the issue since it has been resolved. Please feel free to reach out with any other questions you may have.