Centralized Logging on AWS has been superseded by the Centralized Logging with OpenSearch Centralized Logging with OpenSearch solution. All existing deployments will continue to work but the solution will no longer be supported and maintained.
Centralized Logging on AWS is a reference implementation that provides a foundation for logging to a centralized account. Customers can leverage the solution to index CloudTrail Logs, CW Logs, VPC Flow Logs on an Amazon OpenSearch Service domain. The logs can then be searched on different fields.
This solution gives you a turnkey environment to begin logging and analyzing your AWS environment and applications. Additionally, if you are looking to
- collect logs from multiple AWS accounts and organizations
- collect logs from multiple regions
- a single pane view for log analysis and visualization
then you can get all this with this 1-click deployment solution.
This solution uses Amazon OpenSearch Service (successor to Amazon Elasticsearch Service) and Kibana, an analytics and visualization platform that is integrated with Amazon OpenSearch Service, that results in a unified view of all the log events.
The Centralized Logging on AWS solution contains the following components: log ingestion, log indexing, and visualization. You must deploy the AWS CloudFormation template in the AWS account where you intend to store your log data.
- Prerequisite: Node.js>=16 | npm >= 8
Clone the repository and run the following commands to install dependencies, format and lint as per the project standards
npm ci
npm run prettier-format
npm run lint
You may make any needed change as per your requirement. If you want to customize the Centralized Logging on AWS opinionated defaults, you can modify the solution manifest file. You can also control sending solution usage metrics to aws-solutions, from the manifest file.
"solutionVersion": "%%VERSION%%", #provide a valid value eg. v1.0
"sendMetric": "Yes",
You can run unit tests with the following command from the root of the project
npm run test
You can build lambda binaries with the following command from the root of the project
npm run build
Run the following command from the root of the project. Deploys all the primary solution components needed for Centralized Logging on AWS. Deploy in Primary Account
cd source/resources
npm ci
npm run cdk-bootstrap -- --profile <PROFILE_NAME>
npm run cdk-synth
npm run cdk-deploy -- CL-PrimaryStack --parameters AdminEmail=<EMAIL> --parameters SpokeAccounts=<ACCOUNT-ID-1,ACCOUNT-ID-2...> --parameters JumpboxKey=<EC2_KEY_PAIR> --parameters JumpboxDeploy='Yes' --profile <PROFILE_NAME>
Note: for PROFILE_NAME, substitute the name of an AWS CLI profile that contains appropriate credentials for deploying in your preferred region.
The default deployment uses opinionated values as setup in solution manifest file. In this scenario let's say we want to enable CloudWatch logging for ES domain.
You would need to update the ESDomain resource in cl-primary-stack.ts as below:
logging: {
slowSearchLogEnabled: true,
appLogEnabled: true,
slowIndexLogEnabled: true,
},
Centralized Logging on AWS solution consists of:
- cdk constructs to generate needed resources
- helper for bootstrapping purposes like creating CloudWatch Logs Destinations
- transformer to translate kinesis data stream records into Elasticsearch documents
|-config_files [ config files for prettier, eslint etc. ]
|-architecture.png [ solution architecture diagram ]
|-source/
|dashboard.ndjson [ sample dashboard for demo ]
|run-unit-test.sh [ script to run unit tests ]
|-resources
|-bin/
|-app.ts [ entry point for CDK app ]
|-__tests__/ [ unit tests for CDK constructs ]
|-lib/
|-cl-demo-ec2-construct.ts [ CDK construct for demo web server resource ]
|-cl-demo-stack.ts [ CDK construct for demo stack]
|-cl-jumpbox-construct.ts [ CDK construct for windows jumpbox resource ]
|-cl-primary-stack.ts [ CDK construct for primary stack and related resources ]
|-utils.ts [ utilities for generic functionalities across CDK constructs ]
|-manifest.json [ manifest file for CDK resources ]
|-config_files [ tsconfig, jest.config.js, package.json etc. ]
|-services/
|-@aws-solutions/utils/ [ library with generic utility functions for microservice ]
|-helper/ [ lambda backed helper custom resource to help with solution launch/update/delete ]
|-transformer/ [ microservice to translate kinesis records into es documents ]
See license here
Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
Licensed under the Apache License Version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at
http://www.apache.org/licenses/LICENSE-2.0
or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions and limitations under the License.