Programmatic access of AWS CDK CLI

Question

Programmatic access of AWS CDK CLI

hassankhan opened this issue 6 years ago · comments

Description

Hi there, thanks for creating this package! I was just wondering if there's any way of triggering CDK CLI commands programatically? This would really help with respect to integrating with existing tools and workflows.

~~RFC 300: Programmatic access of AWS CDK CLI~~

Roles

Role	User
Proposed by	@hassankhan, @mrgrain
Author(s)	@mrgrain
API Bar Raiser	@rix0rrr
Stakeholders

See RFC Process for details

Workflow

Author is responsible to progress the RFC according to this checklist, and
apply the relevant labels to this issue so that the RFC table in README gets
updated.

Rico Hermans · Answer 1 · Mon Aug 20 2018 16:06:15 GMT+0800 (China Standard Time)

Hi @hassankhan,

The aws-cdk NodeJS package does expose some library functions that can be used from JavaScript/TypeScript projects directly. However, it's not a polished experience at the moment, because our attention has been focused more towards the construct library so far. You're very welcome to try it and tell us if you run into any issues, though. As a last resort, you can always shell out to the command-line tool.

I'm also curious: could you tell us something about the kinds of integration that you're thinking of?

Hassan Khan · Answer 2 · Tue Aug 21 2018 03:24:49 GMT+0800 (China Standard Time)

Hi @rix0rrr,

Thanks for the info! In an ideal world, I'd love to see something like:

import CDK from '@aws-cdk/cdk';

import MyApp from './MyApp';

const cdk = new CDK({ app: '', versionReporting: false });

cdk.deploy(new MyApp())
  .then(() => {
    return remove(new App());
  })

With regards to integration, essentially, our aim was to define resources such as Lambdas and DynamoDB tables once in our application code, and generate valid snippets from them and insert them into a template for deployment. For example, a theoretical DynamoDB model would look something like this (using dynamodb-data-mapper):

@table('pets')
class Pet {
    @hashKey()
    id: string;

    @rangeKey({defaultProvider: () => new Date()})
    createdAt: Date;

    @attribute()
    isCat?: boolean;
}

The table schema would then be read from this class as part of the deployment process, converted into its constituent Construct and added to the App/Stack. We could use the same approach for other resource types such as SNS Topics and Lambda functions.

Rico Hermans · Answer 3 · Tue Aug 21 2018 16:41:32 GMT+0800 (China Standard Time)

Okay, thanks. I definitely see the value of the first one, especially in defining one-off tasks like EMR jobs or integ tests.

For the second one: you can already do this today, even without tool integration.

If you define you data model in a library, you can use this library both in your runtime code as well as introspect it in your infrastructure code. Your build step would produce both a CloudFormation template as well as a JAR, and during deployment you deploy both together.

Rui Cheng · Answer 4 · Wed Jan 02 2019 21:04:02 GMT+0800 (China Standard Time)

+1
I mentioned this in a duplicate ticket as well.

Brian Quinn · Answer 5 · Tue Jan 08 2019 19:28:43 GMT+0800 (China Standard Time)

I'd like this too, in my case I'd like to wrap a deploy of an ECS cluster so I can scale it manually after the deploy is done (I create it with desiredCount: 0 so the CloudFormation creating is quicker).

Daniel Silva · Answer 6 · Tue Jan 08 2019 19:40:56 GMT+0800 (China Standard Time)

Additional use cases from aws/aws-cdk#1133 - synthesizing stacks from an IDE debugger and embedding into JVM build systems (like Gradle) without having to shell out to a CLI that depends on node.js.

Mike Moulton · Answer 7 · Tue Mar 26 2019 12:49:29 GMT+0800 (China Standard Time)

My team is currently programmatically invoking CDK to support dynamic composition of constructs to create stacks based on the answers to question we ask in a custom CLI tool we wrote. We are currently primarily using it to spin up either SAM apps, or static sites using codepipeline, codebuild, rds, vpc, ssm params, s3 buckets, cloudfront, certs, zones, etc.

I'd very much like an officially supported way of accomplishing this. Currently we are doing some very dirty things to mimic the nature of the CDK CLI and the fact it executes the cdk app multiple times to meet dependencies.

If the CDK team does not have plans to officially support this, I'd very much appreciate not making it harder to achieve this as the notes in aws/aws-cdk#2044 and aws/aws-cdk#2016 suggest.

Karthick S · Answer 8 · Wed May 29 2019 19:20:58 GMT+0800 (China Standard Time)

Is there anyway of triggering AWS CDK ClI (CDK synth , CDK deploy) commands pro-grammatically in typescript ?

Ari · Answer 9 · Fri Jun 07 2019 03:35:01 GMT+0800 (China Standard Time)

I found a way to handle this. I'll create an example repo (hopefully) to demonstrate how I accomplished this process, but the short version looks like this:

Create an AppStack instance and set the synthesizer to the output of app.run():

import { AppStacks } from 'aws-cdk/lib/api/cxapp/stacks'
import { Configuration } from 'aws-cdk/lib/settings';
// ...
const out = this._app.run();        
const argv = {
    '$0': this._argv['$0'],
    '_': this._argv['_']
}
const configuration = new Configuration(argv)
await configuration.load()

const appStacks = new AppStacks({
    aws: this.sdk,
    configuration: configuration,
    synthesizer: async () => out
})

Using the AppStack instance, create an instance of the CdkToolkit:

const appStacks = await this.getAppStacks();

const provisioner = new CloudFormationDeploymentTarget({
    aws: this.sdk,
})
const cli = new CdkToolkit({ appStacks, provisioner });

In order to get the credentials, we'll need to muck around at a low-level with the AWS-SDK:

import { debug } from '@aws-rocket/core';
import AWS = require('aws-sdk')
import { SDK } from 'aws-cdk/lib/api/util/sdk';
import { Mode } from 'aws-cdk/lib/api/aws-auth/credentials'

async function getCredentialsConfig(sdk: SDK): Promise<any> {
    const region = await sdk.defaultRegion()
    const defaultAccount = await sdk.defaultAccount()
    const credentials = await (sdk as any).credentialsCache.get(defaultAccount, Mode.ForReading)
    
    return {
      region,
      credentials
    }
  }

export const awsCredentialsMiddleware = async (argv: any) => {
    debug(`Setting profile to: ${argv.profile}`)
    if (argv.profile) {
        const sdk = new SDK({
            profile: argv.profile
        })
        const credentials = await getCredentialsConfig(sdk)
        AWS.config.update(credentials)
        AWS.config.setPromisesDependency(Promise);
        argv.AWS = AWS
        argv.SDK = sdk
    }
    return argv
}

Finally, this all comes together using the actual deploy like so:

const appStacks = await this.getAppStacks();
const allStacks = await appStacks.listStacks()
const allStackNames = allStacks
    .map((s: cxapi.CloudFormationStackArtifact) => s.name)

const cli = await this.getCdkToolkit()
try {
    const res = await cli.deploy({
        stackNames: allStackNames,
        // `requireApproval` is so that the user never needs to 
        // accept anything for the deploy to actually occur
        requireApproval: RequireApproval.Never,
    })
    console.log('deploy result ->', res);
} catch (e) {
    console.error(`Error deploying`, e);
    process.exit(-1)
}

Hope this helps and hopfully this becomes a part of the actual library. I'll try to submit a PR if I can, but no promises. If anyone else has any better way of handling this, I'd love suggestions.

Joseph Carroll · Answer 10 · Fri Aug 23 2019 11:49:14 GMT+0800 (China Standard Time)

This is all very interesting and hopeful that it will get implemented 🤞

@auser - Were you ever able to put together a repo / gist with all of this information? I follow the logic but I'm not sure how I follow how to assemble all of the pieces. Is each code snippet supposed to be in the same file and if so does each code section follow one an other? I can guess, but I also don't know where the this._app and this._argv come from.

@fulghum / @shivlaks - It looks like there is some movement towards an implementation. Do feel programmatic access might be limited in some areas or is the goal to have a full API access the same you would have on command line?

More or less, do you think something like the following would be supported:

import AWS from 'aws-sdk';
import { deploy } from 'aws-cdk';
import MyStack from './my-stack';

const cfnOutput = await deploy(new MyStack());
console.log(`Successfully deployed stack:\n${cfnOutput}`);

const s3FileAndDetails = { /* ... */ };
const s3 = new AWS.S3();
await s3.putObject(s3FileAndDetails).promise();

Also, instead of needing to run the configuration like @auser has above:

const argv = {
    '$0': this._argv['$0'],
    '_': this._argv['_']
}
const configuration = new Configuration(argv)
await configuration.load()

It would be great if we could skip the command line altogether and just pass in a json object with relevant configuration. (Thereby eliminating the need for a cdk.json, but one could exist with overrides)

// Use factory function to allow reading config overrides in async
const configuration = await Configuration.create({ /* json from cdk.json */ })

Ryan Mierzejewski · Answer 11 · Mon Sep 09 2019 10:23:20 GMT+0800 (China Standard Time)

If you want to synth programmatically to the directory of your choice...

// Let's say you want it synthed to ./build
const outdir = path.join(process.cwd(), 'build') 
// App's constructor let's you specify the outdir 
const app = new cdk.App({outdir})
new ApiLambdaCrudDynamoDBStack(app, 'ApiLambdaCrudDynamoDBExample')
app.synth()

If you want to inspect what was produced app.synth() returns a CloudAssembly which can be interrogated with getStack('your stack name')

I haven't tried deploying programmatically yet.

Joseph Carroll · Answer 12 · Thu Sep 26 2019 05:03:53 GMT+0800 (China Standard Time)

When implementing this feature, would it make sense to do it in the context of aws-sdk-js-v3?
https://github.com/aws/aws-sdk-js-v3

Raphael Campardou · Answer 13 · Sat Oct 19 2019 20:24:52 GMT+0800 (China Standard Time)

(One of) the premise of CDK being "programmatic CloudFormation", it opens the way for fully programmable infrastructure, potentially even starting with the account creation
(and that's awesome, we (building an "Ops as a Service" product) have been hoping for this for quite some time, even looking into developing our own solution).
Creating a full, real-life infrastructure requires more features than CloudFormation can offers, even considering CustomResources.
Some of those will require the SDK ; some others need to talk to 3rd party services ; fetch data before ; output/notify after.

Being able to use CDK not so much as a toolkit/cli, not as a complete solution, but as a lib, as a way to integrate the infrastructure-control layer into a programm would be great. As great as CDK promise to be, it's not a complete solution, probably can't, and shouldn't even try to be one.
Of course, one can always shell out, wrap around CDK CLI, but that's not ideal.

It ties in with a few other GitHub issues:

jfaixo · Answer 14 · Sun Dec 15 2019 17:04:38 GMT+0800 (China Standard Time)

Another use case : when your CDK stack creates Cognito UserPool Client Apps, it generates client id and secrets that need to be injected in your clients. If your client is a webapp in S3, that means you have to do steps in this order:

create userpool and clients, get id values
build webapp with updated id values
deploy webapp to S3

If it was possible to deploy programmatically, it could be possible to have the whole process written in the same tool (for example nodejs). Otherwise (as of now I do this way), you have to rely to some other external tool to call the CLI multiple times and schedule these actions (like a bash script, or other nodejs script through process exec..).

Allen · Answer 15 · Thu Mar 26 2020 03:44:43 GMT+0800 (China Standard Time)

I just found a nice little helper that wraps some of this here: https://github.com/dmitrii-t/cdk-util May be worth a look for people that want to do this type of thing. Definitely can't wait to see it in the core API with first class support.

Ryan Mierzejewski · Answer 16 · Thu Mar 26 2020 05:59:07 GMT+0800 (China Standard Time)

@jfaixo

If it was possible to deploy programmatically

It is.

import * as s3 from "@aws-cdk/aws-s3";
import * as s3deploy from "@aws-cdk/aws-s3-deployment"; // <<--- this right here
import * as cdk from "@aws-cdk/core";

export class WebStack extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const websiteBucket = new s3.Bucket(this, "WebsiteBucket", {
      websiteIndexDocument: "index.html",
      publicReadAccess: true,
      versioned: true
    });

    new s3deploy.BucketDeployment(this, "DeployWebsite", {
      sources: [s3deploy.Source.asset("./build")], // <<--- folder to upload
      destinationBucket: websiteBucket
    });
  }
}

Allen · Answer 17 · Fri Mar 27 2020 02:26:22 GMT+0800 (China Standard Time)

Unfortunately I have found that the code and helpers I can find online are all out of date with the latest version of the CDK. How are people using the CDK in the case where you have a command line tool that needs to dynamically allocate and manage resources on AWS?

Sukhwant Prafullit · Answer 18 · Mon Mar 30 2020 20:02:53 GMT+0800 (China Standard Time)

I am looking to use it from JAVA.

Abilio Henrique · Answer 19 · Tue Jun 02 2020 09:44:43 GMT+0800 (China Standard Time)

I've been able to sort of get this working programatically but have discovered that the cli.deploy etc functions return void, undefined, boolean and 0/1 values instead of useful values like cloudformation stack arns, outputs etc.

Shiv Lakshminarayan · Answer 20 · Wed Jun 10 2020 10:17:14 GMT+0800 (China Standard Time)

from @benwainwright

Currently, I can get a list of stacks that will be generated by my project in the order they should be deployed (this is important; see below) by running the command cdk list
It would be useful if the output from this command could be exposed as an api in a way that doesn't require full synthesis of the templates under the hood.

full use case in aws/aws-cdk#8436

Karl Trumstedt · Answer 21 · Fri Jun 12 2020 04:22:25 GMT+0800 (China Standard Time)

This would be wonderful. Now it feels like CI integration is a pain.. You have to parse output for changeset names and stack ids after deploying to do any more serious CI flow. Using the synthesizing & deploy parts of the tool programmatically and also having the CLI able to output in a JSON format instead of just console logs with necessary information would be perfect

moshir · Answer 22 · Tue Aug 11 2020 15:24:19 GMT+0800 (China Standard Time)

Hi,
any update on this one ?
I'm trying to automate the creation of AWS resources from a lambda function, and would be great to have a programmatic API in addition to the cdk CLI.
Alternatively, is there any way to simply get the cloudformation template string generated by synth(), and then pass it to boto3 cloudformation API ?

Ross Gardiner · Answer 23 · Sat Dec 05 2020 07:13:27 GMT+0800 (China Standard Time)

This would be very useful for my use case: dynamically deploying MediaLive broadcast pipelines. I don't really see shelling out to the CLI as a viable option.

Ignas Kiela · Answer 24 · Wed Jan 27 2021 19:31:58 GMT+0800 (China Standard Time)

This would also be very useful for us for running integration tests. We currently shell out to CDK CLI for this, but that means we have to do some annoying things like having a separate app in test folder that adds more outputs, etc. Ability to deploy stacks with programmatic usage would help greatly for our use case.

Nicola Peduzzi · Answer 25 · Sat Mar 13 2021 08:00:44 GMT+0800 (China Standard Time)

I managed to do this following some @ryan-mars suggestions.

yarn add aws-cdk
yarn add @aws-cdk/core
# plus all the @aws-cdk/... you need

import { App, Stack } from '@aws-cdk/core';
import { SdkProvider } from 'aws-cdk/lib/api/aws-auth';
import { CloudFormationDeployments } from 'aws-cdk/lib/api/cloudformation-deployments';

class ExampleStack extends Stack { /* your stack here */ }

const app = new App();
const stack = new ExampleStack(app, 'StackName', {});

const stackArtifact = app.synth().getStackByName(stack.stackName);

const sdkProvider = await SdkProvider.withAwsCliCompatibleDefaults({
  profile: 'your ~/.aws/config profile name here',
});
const cloudFormation = new CloudFormationDeployments({ sdkProvider });
const deployResultPromise = cloudFormation.deployStack({
  stack: stackArtifact,
});

Dan Van Brunt · Answer 26 · Sat Mar 13 2021 08:35:40 GMT+0800 (China Standard Time)

I managed to do this following some @ryan-mars suggestions.


yarn add aws-cdk

yarn add @aws-cdk/core

# plus all the @aws-cdk/... you need

import { App, Stack } from '@aws-cdk/core';

import { SdkProvider } from 'aws-cdk/lib/api/aws-auth';

import { CloudFormationDeployments } from 'aws-cdk/lib/api/cloudformation-deployments';



class ExampleStack extends Stack { /* your stack here */ }



const app = new App();

const stack = new ExampleStack(app, 'StackName', {});



const stackArtifact = app.synth().getStackByName(stack.stackName);



const sdkProvider = await SdkProvider.withAwsCliCompatibleDefaults({

  profile: 'your ~/.aws/config profile name here',

});

const cloudFormation = new CloudFormationDeployments({ sdkProvider });

const deployResultPromise = cloudFormation.deployStack({

  stack: stackArtifact,

});

What does the deployResultPromise response look like? Is there a way at that point to get the stack resolved outputs?

Nicola Peduzzi · Answer 27 · Sat Mar 13 2021 18:51:38 GMT+0800 (China Standard Time)

What does the deployResultPromise response look like? Is there a way at that point to get the stack resolved outputs?

It's an (experimental) DeployStackResult when resolved.

I haven't tried to parse it yet, not sure how to use it. Yet I'm guessing the outputs map contains everything. Worst case the stackArn could be used with the regular SDK to get the stack status

Dan Van Brunt · Answer 28 · Sun Mar 14 2021 03:50:03 GMT+0800 (China Standard Time)

@thenikso noice! Thanks! I've gonna give something a try now.

Ross Gardiner · Answer 29 · Tue Apr 13 2021 23:59:06 GMT+0800 (China Standard Time)

This workaround has been really useful for me. We are now using it to dynamically deploy our MediaLive+MediaPackage+CloudFront stacks at runtime.

To keep things relatively reliable and avoid request timeouts, I don't await the result of deployment. Instead I:

Add an SNS notification topic to each CloudFormation stack and listen for it to complete/fail creation (slightly complicated by CloudFormation's very underspecified event messages!)
Keep track of deployment jobs that have been started (but not finished) and poll the stack every few minutes to check its status.

I'd love to see this become an officially supported API, since it's much better than the alternatives of using the SDK manually or using traditional CloudFormation configuration files. Hopefully this example code helps anyone else who's trying to do something similar 🙂

Surya Teja · Answer 30 · Fri Jun 11 2021 15:20:53 GMT+0800 (China Standard Time)

import { SdkProvider } from 'aws-cdk/lib/api/aws-auth';
import { CloudFormationDeployments } from 'aws-cdk/lib/api/cloudformation-deployments';
...........................
.............................
const cloudFormation = new CloudFormationDeployments({ sdkProvider });
const deployResultPromise = cloudFormation.deployStack({
  stack: stackArtifact,
});

@thenikso I understand that CDK Toolkit is written in TypeScript. But is there some way I could use python to do a similar deployment? As most of our codebase is in python. It would be helpful to us if you could point me in the right direction, I could take it from there.

Something like :

from aws_cdk import ( aws_cdk.lib as cloud_formation_deployment )
 .....
cloud_formation_deployment.deploy_stack(stack)

Andrew Philips · Answer 31 · Fri Jun 11 2021 23:37:04 GMT+0800 (China Standard Time)

See @thenikso comment of 12 March 2021 for context.

I had to formulate the deploy call as an async call at the module level. This compiled and ran. My particular application requires that I pass in a variable stack name. Although I'm happy to run from a shell, I need to parameterize the name. A programmatic API would be one way to solve this problem.

(async () => {
  const synth = app.synth().getStackByName(stack.stackName)
  const sdkProvider = await SdkProvider.withAwsCliCompatibleDefaults({ profile: '<aws profile>' });
  const cloudFormation = new CloudFormationDeployments({ sdkProvider: sdkProvider});
  const deployResultPromise = cloudFormation.deployStack({ stack: synth });
})();

I wound up going a different way. Inside my ./bin/platform.ts invocation code, I look for an environment variable and use that instead. Even so, the above code worked well for me at the time.

Dan Van Brunt · Answer 32 · Wed Jun 16 2021 03:47:10 GMT+0800 (China Standard Time)

@thenikso, how did you get top-level await working in cdk?

Also, I'm reading through this other issue and wondering the same thing about how to get programmtic use of CDK to use the same settings and defaults as the CLI.
For starters how to pass in context or get it to use the context file?

Nicola Peduzzi · Answer 33 · Tue Jul 20 2021 22:00:18 GMT+0800 (China Standard Time)

as @codeedog noted, you might want to wrap it all in an async function

Florian Fuß · Answer 34 · Thu Sep 02 2021 21:07:46 GMT+0800 (China Standard Time)

are there any plans to officially support using only the api functions without the cli ? This issue is open for 3 years now ...

Eric Riddoch · Answer 35 · Tue Jan 18 2022 15:14:57 GMT+0800 (China Standard Time)

+1 for this. My team has been using troposphere for a while, and I'm seriously envious of the CDK community. Here are some use cases our team has been using the programmatic API of troposphere (a Python IaC library) to solve:

A custom CLI tool to abstract the hard parts of creating REST APIs. Our data scientists are able to place a app-config.yml file in a repository. They then write a FastAPI app compatible with Lambda to serve their models. Then they run our-custom-cli-tool deploy serverless-rest-api --config app-config.yml. This is great, because the data scientists don't have to know anything about troposphere, aws-cdk, or any other IaC tool, but they can easily deploy things.
Integration testing: with troposphere, we can create a stack--for example, a serverless REST API--and then run tests against it. That could mean sending several requests to a REST API and watching for expected responses. If the tests fail, we tear down the stack. Sure, you can do this with a sequence of CLI calls, or maybe by calling CDK as a subprocess, but it feels clunky.
Using stack outputs as inputs to other stacks at runtime. Suppose I want to create an IAM user and put those user credentials into the user-data.sh script of an AWS Lightsail instance. First, I would create an IAM user stack and save the credentials in AWS Secrets Manager. Then, I would wait for that creation to complete and fetch the credentials. Finally, I would create a stack with an AWS Lightsail instance, rendering the credentials into the user-data.sh body.

Conclusion:

I saw that pulumi has an "Automation API" in all the same languages as CDK (they're probably using the JSII inspired by AWS CDK, haha). That said, I've really enjoyed the abstraction of CDK and the community support. The other day I created an S3 bucket with the CDK construct. I set the "expire" policy so objects would auto-delete. Then I saw that to achieve that, an entire lambda function had been created to react to "expiry" events and delete the associated objects. That abstraction is so much nicer than anything we've had with troposphere!

I would dearly love to re-write these apps using AWS CDK. I have no idea the complexity it would take to implement this, since it goes against a fundamental assumption of CDK as of now, but I'd sure be grateful.

Florian Fuß · Answer 36 · Tue Jan 18 2022 21:03:52 GMT+0800 (China Standard Time)

Yeah pulumi is on my try-out list.

I made my experience with CDK and it had a benefit for simple small serverless workloads.
For all other things it was just a pain to work with.
In the end it`s just a wrapper for cloudformation and that is the root cause of all nasty IaC problems.

Will use my old good terraform or try out cdktf for real usecases.

Matthieu Napoli · Answer 37 · Tue Mar 22 2022 23:30:29 GMT+0800 (China Standard Time)

Watch out, programmatic usage of the CDK was strongly impacted in the latest releases: aws/aws-cdk#18667

If you used that approach this is now broken.

Ross Gardiner · Answer 38 · Tue Mar 22 2022 23:40:49 GMT+0800 (China Standard Time)

Now that the CDK is bundled into a single package this all seems like a bit of a lost cause - at least for us it's impractically big to include in our production bundles. I think it's probably better to use CDK to generate CloudFormation templates ahead of time.

Antonio Angelino · Answer 39 · Tue Mar 22 2022 23:47:41 GMT+0800 (China Standard Time)

We widely use AWS CDK programmatically, so we will be forced to stick with aws-cdk 2.13.0.
I really hope we won't be forced to fork CDK in future 🤦‍♂️

Christian Heike · Answer 40 · Tue Mar 22 2022 23:59:23 GMT+0800 (China Standard Time)

Please see: aws/aws-cdk#18667 (comment)

Fabrice Delhoste · Answer 41 · Wed Mar 23 2022 01:02:18 GMT+0800 (China Standard Time)

+1.
We also use AWS CDK intensively and programmatically.

Sepehr Laal · Answer 42 · Wed Mar 23 2022 10:38:35 GMT+0800 (China Standard Time)

programmatic access to the CLI (or at least its functionality) has been a major pain point at two enterprises I worked for and personally I find it unacceptable to just shell out to the CDK CLI to do anything. shelling out is not a secure practice in any environment outside of dev, specifically when people want to integrate with CDK in existing pipelines.

Two doable workarounds exist as of today (that I know of):

using the stack artifact directly, like I do in cdk-web: https://github.com/3p3r/cdk-web/blob/main/cdk-web-cli.js
using cxapi and reloading the artifacts from disk/memory as mentioned here

Unfortunately both approaches rely on CloudFormationDeployments class which is an export out of the CLI package. it seems like offering this API directly as an official API is a great place to start working on resolving this RFC.

It is worth noting that using the AWS SDK to "just deploy" the artifacts does not work at all. Complicated stacks require assets, context lookups, and other metadata related tasks to be done normally via the CLI. so that route is almost like duplication of the CLI's source.

Fortunately, my work has been in Node, JS, and TS and I am lucky to have require and deep imports in Node. other languages and environments are not that lucky.

Matthieu Napoli · Answer 43 · Wed Mar 23 2022 16:10:18 GMT+0800 (China Standard Time)

Unfortunately both approaches rely on CloudFormationDeployments class which is an export out of the CLI package.

That's the problem, this class is no longer usable since aws/aws-cdk#18667

Sepehr Laal · Answer 44 · Wed Mar 23 2022 23:18:09 GMT+0800 (China Standard Time)

@mnapoli it is usable in Node still. You need to direct require it outside its deep import. That’s a node core feature. It can’t be taken away. Only import syntax won’t work directly. It will always be possible to require anything until node core stops supporting that.

Eric Riddoch · Answer 45 · Thu Mar 31 2022 16:28:26 GMT+0800 (China Standard Time)

@mnapoli Sorry if this is a naive question. What would it take for this feature to move forward?

Is this something the community can help with?
Are we waiting for a comment on this issue to propose a viable approach?
Does it have to do with the prioritization of the internal AWS CDK team?
Do you suspect that it may never happen due to a fundamental conflict with how CDK is architected?

I feel like this is potentially such an amazing feature that would close a large gap between what CDK offers and what Pulumi offers (thinking of their "Automation API"). As the comments in this thread point out, it would open up a world of totally relevant use cases that many of us IaC users have to meet.

Matthieu Napoli · Answer 46 · Thu Mar 31 2022 16:44:03 GMT+0800 (China Standard Time)

For clarity I don't work on the CDK, I'm just reporting a discussion that happened in an issue.

But from what I discussed with some of the CDK maintainers it seems that it's mostly about the effort and they are not ready to invest into it yet, and also that they think they would need to provide a cross-language API for all languages CDK supports (instead of just Node).

My personal wish is that they'd focus on Node only first, and start by not breaking the public API that's already available (i.e. it works already, just don't make it worse).

Antonio Angelino · Answer 47 · Thu Mar 31 2022 20:47:39 GMT+0800 (China Standard Time)

My personal wish is that they'd focus on Node only first, and start by not breaking the public API that's already available (i.e. it works already, just don't make it worse).

Totally agree!

Le Stephane · Answer 48 · Thu May 26 2022 13:35:21 GMT+0800 (China Standard Time)

I'm also curious: could you tell us something about the kinds of integration that you're thinking of?

I'm writing integration tests in Kotlin (because the project's most important parts are lambda functions written in Kotlin, in the same monorepo), while keeping the cdk definitions in typescript, and would like to deploy the infrastructure in --hotswap mode without spawning a system command, to give a better error report in the IDE's test results panel when something goes wrong.

Arne De Herdt · Answer 49 · Mon Jun 27 2022 16:59:08 GMT+0800 (China Standard Time)

Three years and still no suitable solution, or even feedback about this...
We are in the same boat as everyone else, with having a custom CLI tool for deploying to AWS and so far the only option is to redo this all through the AWS CLI, which doesn't even allow clean parameter passing and what not unless you use --context.

The this thread started off promising with the ability to invoke the synth and deploy through code, but with that being broken now as well this option is out....

Jan · Answer 50 · Mon Jan 02 2023 20:45:19 GMT+0800 (China Standard Time)

Is there any progress or other tickets we can track for this feature? :)

Dax · Answer 51 · Tue Jan 03 2023 02:48:05 GMT+0800 (China Standard Time)

We actually are accessing CDK programmatically for SST.

It definitely is not what CDK intends and could probably break at any point but you can look at how we deploy stacks here:

https://github.com/serverless-stack/sst/blob/sst2/packages/sst/src/stacks/deploy.ts#L76

Momo Kornher · Answer 52 · Thu Jan 19 2023 17:01:48 GMT+0800 (China Standard Time)

Thanks all for the feedback so far. We have now released a first iteration of this: @aws-cdk/cli-lib-alpha

Please let me know what you're thinking and keep posting feature ideas. We are well aware this isn't a complete and ready solution yet (hence the experimental state), but it's an important stepping stone to gather interest and feedback.

louisdussarps · Answer 53 · Wed Feb 01 2023 23:15:59 GMT+0800 (China Standard Time)

We may be a little late following previous AWS Annoucement.
Still, please allow me to present how we programmatically manage the CDK via Orbits (an openSource flow for DevOps).
It seems to me that Orbits' approch is still relevant if you have complex interactions to manage within your stacks or through a multi - steps deployment.

https://github.com/LaWebcapsule/orbits
https://github.com/LaWebcapsule/orbits/tree/main/src/examples/git-cdk-s3

Vinicius Henrique · Answer 54 · Sun Jun 04 2023 12:43:32 GMT+0800 (China Standard Time)

Hello @mrgrain, I'm trying to implement this using Python 3.11, but actually facing issue related below when running synth:

cli.synth()
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/vinicius/.cache/pypoetry/virtualenvs/auk-I98UnL2N-py3.11/lib/python3.11/site-packages/aws_cdk/cli_lib_alpha/__init__.py", line 2276, in synth
    return typing.cast(None, jsii.ainvoke(self, "synth", [options]))
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vinicius/.cache/pypoetry/virtualenvs/auk-I98UnL2N-py3.11/lib/python3.11/site-packages/jsii/_kernel/__init__.py", line 149, in wrapped
    return _recursize_dereference(kernel, fn(kernel, *args, **kwargs))
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vinicius/.cache/pypoetry/virtualenvs/auk-I98UnL2N-py3.11/lib/python3.11/site-packages/jsii/_kernel/__init__.py", line 457, in ainvoke
    promise = self.provider.begin(
              ^^^^^^^^^^^^^^^^^^^^
  File "/home/vinicius/.cache/pypoetry/virtualenvs/auk-I98UnL2N-py3.11/lib/python3.11/site-packages/jsii/_kernel/providers/process.py", line 387, in begin
    return self._process.send(request, BeginResponse)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vinicius/.cache/pypoetry/virtualenvs/auk-I98UnL2N-py3.11/lib/python3.11/site-packages/jsii/_kernel/providers/process.py", line 327, in send
    self._process.stdin.flush()
BrokenPipeError: [Errno 32] Broken pipe

My code is:

Running my click CLI command:

auk cloud aws instance create

Click method:

@aws.command('instance')
@click.pass_context
@click.argument('create', nargs=-1)
def instance(ctx, create: bool = False):
    print('create instances')
    if create:
        producer = Producer()
        cli = AwsCdkCli.from_cloud_assembly_directory_producer(producer)
        cli.synth(
            stacks=["EC2Stack"]
        )

aws.py file:

import os.path

from aws_cdk import Stack, App
from aws_cdk import aws_ec2 as ec2
from aws_cdk import aws_iam as iam
from aws_cdk.aws_s3_assets import Asset
from constructs import Construct
from aws_cdk.cli_lib_alpha import ICloudAssemblyDirectoryProducer
import jsii

dirname = os.path.dirname(__file__)


class EC2InstanceStack(Stack):

    def __init__(self, scope: Construct, id: str, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        vpc = ec2.Vpc(self, "VPC",
                      nat_gateways=0,
                      subnet_configuration=[ec2.SubnetConfiguration(
                          name="public", subnet_type=ec2.SubnetType.PUBLIC)]
                      )

        amzn_linux = ec2.MachineImage.latest_amazon_linux(
            generation=ec2.AmazonLinuxGeneration.AMAZON_LINUX_2,
            edition=ec2.AmazonLinuxEdition.STANDARD,
            virtualization=ec2.AmazonLinuxVirt.HVM,
            storage=ec2.AmazonLinuxStorage.GENERAL_PURPOSE
        )

        role = iam.Role(self, "InstanceSSM",
                        assumed_by=iam.ServicePrincipal("ec2.amazonaws.com"))

        role.add_managed_policy(iam.ManagedPolicy.from_aws_managed_policy_name(
            "AmazonSSMManagedInstanceCore"))

        instance = ec2.Instance(self, "Instance",
                                instance_type=ec2.InstanceType("t3.nano"),
                                machine_image=amzn_linux,
                                vpc=vpc,
                                role=role
                                )

        # # Script in S3 as Asset
        # asset = Asset(self, "Asset", path=os.path.join(
        #     dirname, "configure.sh"))
        # local_path = instance.user_data.add_s3_download_command(
        #     bucket=asset.bucket,
        #     bucket_key=asset.s3_object_key
        # )

        # # Userdata executes script from S3
        # instance.user_data.add_execute_file_command(
        #     file_path=local_path
        # )
        # asset.grant_read(instance.role)


@jsii.implements(ICloudAssemblyDirectoryProducer)
class Producer:
    def produce(self, context):
        app = App(context=context)
        # stack = Stack(app)
        stack = EC2InstanceStack(app, "EC2Stack")
        return app.synth().directory

Could you explain better how can I use this?

Vinicius Henrique · Answer 55 · Sun Jun 04 2023 12:44:33 GMT+0800 (China Standard Time)

Following doc: https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.cli_lib_alpha/README.html

Versions:

[tool.poetry.dependencies]
python = "^3.11"
google-cloud-compute = "^1.11.0"
click = "^8.1.3"
aws-cdk-lib = "^2.82.0"
aws-cdk-cli-lib-alpha = "^2.82.0a0"
jsii = "^1.82.0"

Momo Kornher · Answer 56 · Mon Jun 05 2023 17:54:32 GMT+0800 (China Standard Time)

Hey @bastosvinicius Apologies for this, but unfortunately Python is currently not supported by this alpha module. While there are some scenarios where it seems to work, most don't. This is due to an issue with jsii and how async code is handled. =/

Vinicius Henrique · Answer 57 · Wed Jun 14 2023 23:27:34 GMT+0800 (China Standard Time)

Thanks for reply @mrgrain. Is there a forecast for definitive python support?

Momo Kornher · Answer 58 · Wed Jun 14 2023 23:29:44 GMT+0800 (China Standard Time)

It's pretty much this PR: aws/jsii#3919
I was busy with other project work, hoping to get back to it in July.

Vinicius Henrique · Answer 59 · Wed Jun 14 2023 23:35:20 GMT+0800 (China Standard Time)

I'll follow the PR and wait for the new version of jsii to be released to test it. Thanks for the feedback.

Evgeny Karasik · Answer 60 · Wed Feb 21 2024 15:11:00 GMT+0800 (China Standard Time)

Should include other CLI commands, like cdk diff in the scope - aws/aws-cdk#679