Giters

aws / aws-cdk

The AWS Cloud Development Kit is a framework for defining cloud infrastructure in code

Home Page:https://aws.amazon.com/cdk

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

aws-mwaa: CfnEnvironment logging configuration required parameters set to optional

simond opened this issue · comments

commented

Describe the bug

When trying to instantiate a new CfnEnvironment with logging configuration set to the following:

loggingConfiguration: {
    taskLogs: {enabled: false},
    webserverLogs: {enabled: false},
    schedulerLogs: {enabled: false},
    workerLogs: {enabled: false},
    dagProcessingLogs: {enabled: false}
}

I get an error when running CDK deploy:

4:23:16 pm | CREATE_FAILED        | AWS::MWAA::Environment      | MwaaEnvironment
Resource handler returned message: "Invalid request provided: 5 validation errors detected: Value null at 'loggingConfiguration.schedulerLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'loggingConfigur
ation.dagProcessingLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'loggingConfiguration.webserverLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'loggingConfigurati
on.workerLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'loggingConfiguration.taskLogs.logLevel' failed to satisfy constraint: Member must not be null (Service: Mwaa, Status Code: 400, Request ID: 916

The error is due to the logLevel attribute not being passed into the logging options. The following fixes the issue:

loggingConfiguration: {
    taskLogs: {enabled: false, logLevel: 'INFO'},
    webserverLogs: {enabled: false, logLevel: 'INFO'},
    schedulerLogs: {enabled: false, logLevel: 'INFO'},
    workerLogs: {enabled: false, logLevel: 'INFO'},
    dagProcessingLogs: {enabled: false, logLevel: 'INFO'},
}

The issue is that that logLevel is not a required parameter according to the TypeScript interface:

/**
 * Defines the Apache Airflow logs to send for the log type (e.g. `DagProcessingLogs` ) to CloudWatch Logs. Valid values: `CRITICAL` , `ERROR` , `WARNING` , `INFO` .
 *
 * @see http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-mwaa-environment-moduleloggingconfiguration.html#cfn-mwaa-environment-moduleloggingconfiguration-loglevel
 */
readonly logLevel?: string;

Expected Behavior

CDK should either:

  • Allow me to exclude a logLevel (since I've disabled logging anyway) and deploy the instance of MWAA
    or
  • Force me to provide a logLevel

Current Behavior

Error message asking me to provide the logLevel

Reproduction Steps

        const mwaaEnvironment = new mwaa.CfnEnvironment(this, "MwaaEnvironment", {
            name: 'MyMwaaEnvironment',
            airflowVersion: '2.0.2',
            environmentClass: 'mw1.small',
            maxWorkers: 1,
            minWorkers: 1,
            dagS3Path: 'dags',
            sourceBucketArn: {some bucket arn},
            executionRoleArn: {some role arn},
            requirementsS3Path: 'requirements.txt',
            pluginsS3Path: "plugins.zip",
            networkConfiguration: {
                securityGroupIds: [securityGroup.securityGroupId],
                subnetIds: vpc.selectSubnets({subnetType: ec2.SubnetType.PUBLIC}).subnetIds.slice(0, 2)
            },
            webserverAccessMode: 'PUBLIC_ONLY',
            loggingConfiguration: {
                taskLogs: {enabled: false, logLevel: 'INFO'},
                webserverLogs: {enabled: false},
                schedulerLogs: {enabled: false},
                workerLogs: {enabled: false},
                dagProcessingLogs: {enabled: false},
            }
        });

Possible Solution

Provide a logLevel

Additional Information/Context

No response

CDK CLI Version

2.145.0 (build fdf53ba)

Framework Version

No response

Node.js Version

v22.0.0

OS

osx

Language

TypeScript

Language Version

v10.9.2

Other information

No response

commented

Findings:

  • Per AWS::MWAA::Environment LoggingConfiguration, the properties DagProcessingLogs, SchedulerLogs, TaskLogs, WebserverLogs and WorkerLogs are of type ModuleLoggingConfiguration.
  • ModuleLoggingConfiguration, LogLevel is optional. 
    LogLevel
Defines the Apache Airflow logs to send for the log type (e.g. DagProcessingLogs) to CloudWatch Logs. Valid values: CRITICAL, ERROR, WARNING, INFO.

Required: No

Type: String

Allowed values: CRITICAL | ERROR | WARNING | INFO | DEBUG

Update requires: [No interruption](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-update-behaviors.html#update-no-interrupt)
    The CloudFormation resource spec here also states LogLevel as optional.

However, deploying the following CDK stack:

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as mwaa from 'aws-cdk-lib/aws-mwaa';

export class Issue30645Stack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const mwaaEnvironment = new mwaa.CfnEnvironment(this, "MwaaEnvironment", {
      name: 'MyMwaaEnvironment',
      loggingConfiguration: {
          taskLogs: {enabled: false, logLevel: 'INFO'},
          webserverLogs: {enabled: false},
          schedulerLogs: {enabled: false},
          workerLogs: {enabled: false},
          dagProcessingLogs: {enabled: false},
          }
        });
  }
}

gives the below validation error from CloudFormation:

11:23:22 AM | CREATE_FAILED        | AWS::MWAA::Environment | MwaaEnvironment
Resource handler returned message: "Invalid request provided: 8 validation errors detected: Value null at 'dagS3Path' failed to satisfy constraint: Member must not be null; Value nu
ll at 'loggingConfiguration.schedulerLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'loggingConfiguration.dagProcessingLogs.logLevel' failed to
satisfy constraint: Member must not be null; Value null at 'loggingConfiguration.webserverLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'loggin
gConfiguration.workerLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'networkConfiguration' failed to satisfy constraint: Member must not be null
; Value null at 'sourceBucketArn' failed to satisfy constraint: Member must not be null; Value null at 'executionRoleArn' failed to satisfy constraint: Member must not be null (Serv
ice: Mwaa, Status Code: 400, Request ID: 8319e3d0-4320-4748-8ff1-c873cb413650)" (RequestToken: b6c7274d-c040-c02e-b1d6-f43c071438d3, HandlerErrorCode: InvalidRequest)


 ❌  Issue30645Stack failed: Error: The stack named Issue30645Stack failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Resource handler returned message: "Invalid request provided: 8 validation errors detected: Value null at 'dagS3Path' failed to satisfy constraint: Member must not be null; Value null at 'loggingConfiguration.schedulerLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'loggingConfiguration.dagProcessingLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'loggingConfiguration.webserverLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'loggingConfiguration.workerLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'networkConfiguration' failed to satisfy constraint: Member must not be null; Value null at 'sourceBucketArn' failed to satisfy constraint: Member must not be null; Value null at 'executionRoleArn' failed to satisfy constraint: Member must not be null (Service: Mwaa, Status Code: 400, Request ID: 8319e3d0-4320-4748-8ff1-c873cb413650)" (RequestToken: b6c7274d-c040-c02e-b1d6-f43c071438d3, HandlerErrorCode: InvalidRequest)
    at FullCloudFormationDeployment.monitorDeployment (/usr/local/lib/node_modules/aws-cdk/lib/index.js:451:10568)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack2 [as deployStack] (/usr/local/lib/node_modules/aws-cdk/lib/index.js:454:199716)
    at async /usr/local/lib/node_modules/aws-cdk/lib/index.js:454:181438

 ❌ Deployment failed: Error: The stack named Issue30645Stack failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Resource handler returned message: "Invalid request provided: 8 validation errors detected: Value null at 'dagS3Path' failed to satisfy constraint: Member must not be null; Value null at 'loggingConfiguration.schedulerLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'loggingConfiguration.dagProcessingLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'loggingConfiguration.webserverLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'loggingConfiguration.workerLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'networkConfiguration' failed to satisfy constraint: Member must not be null; Value null at 'sourceBucketArn' failed to satisfy constraint: Member must not be null; Value null at 'executionRoleArn' failed to satisfy constraint: Member must not be null (Service: Mwaa, Status Code: 400, Request ID: 8319e3d0-4320-4748-8ff1-c873cb413650)" (RequestToken: b6c7274d-c040-c02e-b1d6-f43c071438d3, HandlerErrorCode: InvalidRequest)
    at FullCloudFormationDeployment.monitorDeployment (/usr/local/lib/node_modules/aws-cdk/lib/index.js:451:10568)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack2 [as deployStack] (/usr/local/lib/node_modules/aws-cdk/lib/index.js:454:199716)
    at async /usr/local/lib/node_modules/aws-cdk/lib/index.js:454:181438

The stack named Issue30645Stack failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Resource handler returned message: "Invalid request provided: 8 validation errors detected: Value null at 'dagS3Path' failed to satisfy constraint: Member must not be null; Value null at 'loggingConfiguration.schedulerLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'loggingConfiguration.dagProcessingLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'loggingConfiguration.webserverLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'loggingConfiguration.workerLogs.logLevel' failed to satisfy constraint: Member must not be null; Value null at 'networkConfiguration' failed to satisfy constraint: Member must not be null; Value null at 'sourceBucketArn' failed to satisfy constraint: Member must not be null; Value null at 'executionRoleArn' failed to satisfy constraint: Member must not be null (Service: Mwaa, Status Code: 400, Request ID: 8319e3d0-4320-4748-8ff1-c873cb413650)" (RequestToken: b6c7274d-c040-c02e-b1d6-f43c071438d3, HandlerErrorCode: InvalidRequest)

Same goes for SourceBucketArn which is specified as not required at AWS::MWAA::Environment.

Most likely an issue with CloudFormation resource specification.

commented

Internal tracking ticket: P137161118

Another internal ticket (from above ticket): P137427298