Seeing `runtime error: invalid memory address or nil pointer dereference` with extension v39 and v40

[Rey Abolofia]

Problem

Runtimes: java, dotnet, go
Extension: v39 and v40
Universal instrumentation: enabled, either by default (java/dotnet) or via DD_UNIVERSAL_INSTRUMENTATION=true (go)
Appsec: disabled (ie DD_SERVERLESS_APPSEC_ENABLED=false)

After the lambda function has returned its response, the Datadog Lambda Extension panics with the following error.

2023-03-14 18:06:21 UTC | DD_EXTENSION | DEBUG | [lifecycle] onInvokeEnd --------
2023-03-14 18:06:21 UTC | DD_EXTENSION | DEBUG | [lifecycle] Invocation has finished at: 2023-03-14 18:06:21.701256499 +0000 UTC m=+5.080367241
2023-03-14 18:06:21 UTC | DD_EXTENSION | DEBUG | [lifecycle] Invocation isError is: false
2023-03-14 18:06:21 UTC | DD_EXTENSION | DEBUG | [lifecycle] ---------------------------------------
2023-03-14 18:06:21 UTC | DD_EXTENSION | DEBUG | [lifecycle] No http status code found in the response payload
2023/03/14 18:06:21 http: panic serving 127.0.0.1:38412: runtime error: invalid memory address or nil pointer dereference
goroutine 263 [running]:
net/http.(*conn).serve.func1()
/usr/local/go/src/net/http/server.go:1850 +0xbf
panic({0x163b3a0, 0x26d6fa0})
/usr/local/go/src/runtime/panic.go:890 +0x262
github.com/DataDog/datadog-agent/pkg/serverless/appsec/httpsec.(*InvocationSubProcessor).OnInvokeEnd(0x0, 0xc0000a17c0, 0xc000489bc0)
/tmp/dd/datadog-agent/pkg/serverless/appsec/httpsec/http.go:136 +0x778
github.com/DataDog/datadog-agent/pkg/serverless/invocationlifecycle.(*LifecycleProcessor).OnInvokeEnd(0xc00048acd0, 0xc0000a17c0)
/tmp/dd/datadog-agent/pkg/serverless/invocationlifecycle/lifecycle.go:206 +0x2d7
github.com/DataDog/datadog-agent/pkg/serverless/daemon.(*EndInvocation).ServeHTTP(0xc0000bf700, {0x1c1d8a0, 0xc00001c540}, 0xc00012f300)
/tmp/dd/datadog-agent/pkg/serverless/daemon/routes.go:111 +0x34c
net/http.(*ServeMux).ServeHTTP(0x0?, {0x1c1d8a0, 0xc00001c540}, 0xc00012f300)
/usr/local/go/src/net/http/server.go:2487 +0x149
net/http.serverHandler.ServeHTTP({0x1c16fd0?}, {0x1c1d8a0, 0xc00001c540}, 0xc00012f300)
/usr/local/go/src/net/http/server.go:2947 +0x30c
net/http.(*conn).serve(0xc0001439a0, {0x1c1e8c8, 0xc000662b70})
/usr/local/go/src/net/http/server.go:1991 +0x607
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:3102 +0x4db

Impact

The customer function completes as expected. However, the following data may not be collected and sent by the extension.

• aws.lambda.enhanced.errors metric
• aws.lambda invocation span
• Any inferred spans normally created by the extension

Mitigation

At this point, if you are needing the above missing data, the best solution is to roll back your extension version to v38.

A fix is currently in review to address the underlying issue (see DataDog/datadog-agent#16054). However, due to the incident the week of March 7, we have chosen to hold off releasing a fix at this time as we are prioritizing product stability.

[Rey Abolofia]

This issue has now been fixed and included in Datadog Lambda Extension v41. If you are seeing the errors described in this issue, upgrading your extension to v41 or greater will solve this.