Microservice_Observability_with_AWS_OpenSearch

Microservice Observability with AWS OpenSearch Servise

Amazon OpenSearch Service

Amazon OpenSearch is a managed AWS service that makes it easy to deploy, operate, and scale OpenSearch clusters in the AWS Cloud. Amazon OpenSearch Service supports OpenSearch and legacy Elasticsearch OSS.

OpenSearch

OpenSearch is a distributed, open-source and analytics suite used for a broad set of use cases like real-time application monitoring, log analytics, and website search that includes OpenSearch Dashboards.

OpenTelemetry

OpenTelemetry is a collection of tools, APIs, and SDKs to instrument, generate, collect, and export telemetry data(metrics, logs, and traces) to help you analyze your application's behavior.

Docker

Docker is OS-level virtualization software to run software in containers.

Amazon EKS

Amazon Elastic Kubernetes Service is a managed service that you can use to run Kubernetes on AWS without needing to install, operate, and maintain your own Kubernetes control plane or nodes. Kubernetes is an open-source system for automating the deployment, scaling, and management of containe.

Amazon ECR

Amazon Elastic Container Registry is a managed container image registry service that is secure, scalable and reliable.

AWS Distro for OpenTelemetry

AWS Distro for OpenTelemetry is a secure, production-ready, AWS-supported distribution of the OpenTelemetry project.

Trace Analytics

Trace Analytics is a plugin which you can use to analyze trace data from distributed application. The default installation of OpenSearch Dashboard for Amazon OpenSearch Service includes this plugin.

Data Prepper

Data Prepper is a open source utility service with ability to filter, enrich, transform, normalize and aggregate data to enable an end-to-end analysis life cycle from gathering raw logs to facilitating sophisticated and actionable interactive ad-hoc analyses on the data.

FluentBit

FluentBit is an open source processor and forwarder which collects, enriches and sends metrics and logs to various destinations.

What is Observability?

Observability is the capability to continuously generate and discover actionable insights based on signals from the system under observation with the goal to influence the system.

Source Code GitHub repo

GET _cat/indices?v

curl -sSL https://raw.githubusercontent.com/aws-samples/observability-with-amazon-opensearch/main/00-setup.sh | bash -s stable
source ~/.bash_profile

cd ~/environment/observability-with-amazon-opensearch/scripts/
bash 01-build-push.sh

...
raw-pipeline #Line 25
...
 sink: #Line 31
        - opensearch:
            hosts: [ "https://__AOSDomainEndpoint__" ]
            username: "__AOSDomainPassword__"
            password: "__AOSDomainUserName__"
            trace_analytics_raw: true
...
service-map-pipeline #Line 37
...
 sink: #Line 44
        - opensearch:
            hosts: [ "https://__AOSDomainEndpoint__" ]
            username: "__AOSDomainPassword__"
            password: "__AOSDomainUserName__"
            trace_analytics_service_map: true
...
log-pipeline #Line 50
...
sink: #Line 62
        - opensearch:
            hosts: [ "https://__AOSDomainEndpoint__" ]
            username: "__AOSDomainPassword__"
            password: "__AOSDomainUserName__"
            index: sample_app_logs

cd ~/environment/observability-with-amazon-opensearch/scripts/
bash 02-apply-k8s-manifests.sh

watch -n 10 kubectl get pods --all-namespaces

kubectl get svc -nclient-service | awk '{print $4}' | tail -n1

cd /environment/observability-with-amazon-opensearch/scripts/
kubectl get svc -nclient-service | awk '{print $4}' | tail -n1

GET _cat/indices/sample_app_logs*?v

source = sample_app_logs

aws ecr get-login-password --region ${AWS_REGION} | docker login --username AWS --password-stdin ${ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com

export repo_name=payment-service
cd $HOME/environment/observability-with-amazon-opensearch/sample-apps/08-paymentService/
docker build -t ${repo_name}:fixed .
docker tag ${repo_name}:fixed ${ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${repo_name}:fixed
docker push ${ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${repo_name}:fixed

sed -i -e "s|amazonaws\.com\/payment\-service|amazonaws\.com\/payment\-service\:fixed|g" ${HOME}/environment/observability-with-amazon-opensearch/sample-apps/08-paymentService/kubernetes/01-deployment.yaml
kubectl apply -f ${HOME}/environment/observability-with-amazon-opensearch/sample-apps/08-paymentService/kubernetes/

cd ~/environment/observability-with-amazon-opensearch/scripts/
bash 03-delete-ecr-repo.sh

darjidhruv26 / Microservice_Observability_with_AWS_OpenSearch

Microservice_Observability_with_AWS_OpenSearch

Source Code GitHub repo

Resources

About