Aiven Metadata Parser

Want to get all the juicy metadata out of your Aiven services? Check the Aiven Metadata Parser!

Based on the aiven-client python library, the Aiven Metadata Parser exports the metadata from your services in order to build a connected graph showing all interconnections.

Currently the Aiven Metadata Parser is work in progress and extracts only a limited amount of information and connections

Create a set of test services

You can use the create_services.sh and delete_services.sh to create and destroy a pre-set of services. The create_services.sh uses:

jq to parse JSON output
kcat for Apache Kafka®
psql for PostgreSQL®
The Aiven CLI to connect and create services
The Python packages in the requirements.txt file to push data to Grafana

Before starting it, please install the required dependences, login with the Aiven CLI and execute

./scripts/create_services.sh <PROJECT_NAME>

Where <PROJECT_NAME> is the name of your Aiven project

To delete the services you can call

./scripts/delete_services.sh <PROJECT_NAME>

If <PROJECT_NAME> is not passed, the default project will be used

Start with the Aiven Metadata Parser

You need to be on Python 3.7, install the required libraries with

pip install -r requirements.txt

Copy the conf.env.sample file to conf.env in the conf folder and edit the token parameter and the project name from which you want to extract parameters. If you don't have a project with services already running you can create a sample set of services with the create_services.sh file, which requires the aiven-client to be installed and the user to be logged.

Once conf.env is set, you can start the metadata extraction with

python main.py

This will generate:

A file nx.html containing the complete interactive graph
A file filtered.html containing the complete interactive graph filtered on the node with id pg~demo-pg~schema~public~table~pasta (this might error out if you don't have such node)
A file graph_data.dot containing the information in DOT format
A file graph_data.gml containing the information in GML format

Furthermore if, after executing the main.py you also execute the following

python app.py

The app.py reads the graph_data.gml file generated at step 1 and creates a Reactive Web Applications with Plotly and Dash (code taken from here)

Warning The code is a bare minimum product, doesn't cover all services and options and is quite chaotic! it demonstrates the idea and how we could start implementing it.

Possible issues and solutions

If you run python app.py and see an error saying No such file or directory: 'neato', you will have to install graphviz on your machine (the package version from pip doesn't seem to work) - check how to install here

Contributing

Wanna contribute? Check the CONSTRIBUTING file

License

Aiven Metadata Extractor is licensed under the Apache license, version 2.0. Full license text is available in the LICENSE file.

Please note that the project explicitly does not require a CLA (Contributor License Agreement) from its contributors.

Contact

Bug reports and patches are very welcome, please post them as GitHub issues and pull requests at https://github.com/aiven/metadata-parser . To report any possible vulnerabilities or other serious issues please see our security policy.

Apache, Apache Kafka, Kafka, Apache Flink, Flink, Apache Cassandra, and Cassandra are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. ClickHouse is a registered trademark of ClickHouse, Inc. https://clickhouse.com. M3, M3 Aggregator, M3 Coordinator, OpenSearch, PostgreSQL, MySQL, InfluxDB, Grafana, Terraform, and Kubernetes are trademarks and property of their respective owners. *Redis is a trademark of Redis Ltd. and the Redis box logo is a mark of Redis Ltd. Any rights therein are reserved to Redis Ltd. Any use by Aiven is for referential purposes only and does not indicate any sponsorship, endorsement or affiliation between Redis and Aiven. All product and service names used in this website are for identification purposes only and do not imply endorsement.

Fxztam / metadata-parser