Monitoring Docs: Datadog

[Dan Van Brunt]

sst/www/docs/advanced/monitoring.md

Line 2 in be3d409

 title: Monitoring 

I think these docs might need an update? Currently, here are the issues that we found with the docs implementation:

Issue 1: Waiting for app.finish() breaks things

If you wait for app finish, all functions end up deployed with only placeholder code:
export function placeholder() {}

Issue 2: using the NodeJSProps "install" makes for HUGE 22MB functions

It seems like the docs say that using nodejs: { install: [...stuff...] } will...

"Packages that will be excluded from the bundle and installed into node_modules instead. Useful for dependencies that cannot be bundled, like those with binary dependencies"
...so while they will NOT be included in the bundle file... they WILL be included needlessly into the lambda and that can make the difference between a 22MB Lambda and a 236kB one.

Our Helper function

we went ahead and just created a simple helper that we plan to use that seems to work....

sst.config.ts

import { utils as constructUtils } from '@klick-bx/cdk-constructs'
import { utils as coreUtils } from '@klick-bx/core'
import { SSTConfig } from 'sst'
import { FunctionProps } from 'sst/constructs'

import MainStack from './stacks/MainStack.js'

import { name } from './package.json'

export default {
  config(_input) {
    return {
      name,
      region: 'us-east-1',
    }
  },
  stacks(app) {
    if (app.stage !== 'prod') {
      app.setDefaultRemovalPolicy('destroy')
    }
    const defaultFunctionProps = {
      runtime: 'nodejs18.x',
    } as FunctionProps
    constructUtils.setupAppWithDatadog({
      app,
      defaultFunctionProps,
      stackConfigs: [
        {
          stackClass: MainStack,
          props: {
            id: 'main',
            tags: coreUtils.getStackTags(),
          },
        },
      ],
    })
  },
} satisfies SSTConfig

index.ts

import { Effect, PolicyStatement } from 'aws-cdk-lib/aws-iam'
import { Datadog } from 'datadog-cdk-constructs-v2'
import { App, FunctionProps, Stack, StackContext, StackProps } from 'sst/constructs'

const DATADOG_API_KEY_SECRET_ARN =
  'NOT_TELLING'

export async function setupAppWithDatadog({
  app,
  stackConfigs,
  defaultFunctionProps,
}: {
  app: App
  defaultFunctionProps: FunctionProps
  stackConfigs: {
    stackClass: ({ stack }: StackContext) => void
    props?: (StackProps & { id?: string | undefined }) | undefined
  }[]
}) {
  const enableDatadog = !app.local

  if (enableDatadog) {
    // Allow functions to access secret
    app.addDefaultFunctionPermissions([
      new PolicyStatement({
        effect: Effect.ALLOW,
        resources: [DATADOG_API_KEY_SECRET_ARN],
        actions: ['secretsmanager:GetSecretValue'],
      }),
    ])
    // Exclude from the function bundle
    // since they'll be loaded from the Layer
    const propsExternal = defaultFunctionProps?.nodejs?.esbuild?.external || []
    app.setDefaultFunctionProps({
      ...defaultFunctionProps,
      nodejs: {
        ...defaultFunctionProps?.nodejs,
        esbuild: {
          ...defaultFunctionProps.nodejs?.esbuild,
          external: [...propsExternal, 'dd-trace', 'datadog-lambda-js'],
        },
      },
    })
  }

  // Initialize stacks
  stackConfigs.forEach(({ stackClass, props }) => {
    app.stack(stackClass, props)
  })

  if (enableDatadog) {
    // Attach the Datadog contruct to each stack
    app.node.children.forEach((stack) => {
      if (stack instanceof Stack) {
        console.warn('stack instanceof Stack', stack)
        const datadog = new Datadog(stack, 'datadog', {
          // Get the latest version from
          // https://github.com/Datadog/datadog-lambda-js/releases
          nodeLayerVersion: 105,
          // Get the latest version from
          // https://github.com/Datadog/datadog-lambda-extension/releases
          extensionLayerVersion: 55,
          site: 'datadoghq.com',
          apiKeySecretArn: datadogApiKeySecretArn,
          env: app.stage,
          service: app.name,
          // Just a recommendation, feel free to change the version per your CI/CD
          version: process.env.SEED_BUILD_SERVICE_SHA || process.env.GITHUB_SHA || undefined,
        })

        datadog.addLambdaFunctions(stack.getAllFunctions())
      }
    })
  }
}

[Apostolos Daniel Apostolidis]

I was looking at this exact issue this week. I've reached a similar conclusion: setting the Datadog libraries as external drops the size of the Lambda Functions from 22MB to 1.7KB.

The reason I was looking at this was unnecessary latency we noticed of approx. 1.2-1.5s for cold starts for Node18 hello world Lambda Functions. Are you seeing similar latency issues and what were your gains when reducing the lambda size?

[Dan Van Brunt]

@Apostolos-Daniel to be honest, we never let it get to the point where it could be perf'd. We decided not to take it to prod if we could not fix the size issue. I do believe that lambda file size can have a significant effect on perf.

[Jay]

@iDVB thanks for taking a look. Yeah these docs are community contributed.

Just scanning through your solution, are the big changes?

1. Removing app.finish
2. Using esbuild.external instead of install

If so, then I can edit the doc.

[Dan Van Brunt]

@jayair Yes that seems to be the major diff.
Thanks again!

[Jay]

Done!