Aluno: Marcel Pinheiro Caraciolo (mpc3@cesar.school)
Este projeto consiste no uso das tecnologias S3, Athena e Glue provisionadas por meio da tecnologia CDK Toolkit em Python na AWS.
- Ao provisionar o ambiente em CDK já realizamos o upload dos 2 arquivos de base de dados da empresa fictícia 5GFlix na zona landing em um bucket no S3.
- Temos um Job Glue provisionado para realizar as tarefas de pré-processamento dos dados (limpeza, extração de colunas, e armazenamento em formato parquet).
- Os dados processados são armazenados em parquet na zona transformada em um bucket no S3.
- Temos um job Crawler Glue que realiza o mapeamento e inferência das base de dados e armazena os dados em um database para consultas futuras em Athena.
- Temos um workspace no Athena com consultas em SQL mapeadas para respostas de análise das perguntas solicitadas pela gestão do 5GFlix (objetivo do exercício).
-
Ao provisionar o ambiente em CDK já realizamos o upload dos 2 arquivos de base de dados da empresa fictícia 5GFlix na zona landing em um bucket no S3.
-
Temos um Job Glue provisionado para realizar as tarefas de pré-processamento dos dados (limpeza, extração de colunas, e armazenamento em formato parquet).
-
Os dados processados são armazenados em parquet na zona transformada em um bucket no S3.
-
Temos um job Crawler Glue que realiza o mapeamento e inferência das base de dados e armazena os dados em um database para consultas futuras em Athena.
![Captura de Tela 2024-03-25 às 15 21 05](https://private-user-images.githubusercontent.com/275084/316610567-4855da2c-9aca-4576-a05f-900fd6730b09.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTQzMzYyMzYsIm5iZiI6MTcxNDMzNTkzNiwicGF0aCI6Ii8yNzUwODQvMzE2NjEwNTY3LTQ4NTVkYTJjLTlhY2EtNDU3Ni1hMDVmLTkwMGZkNjczMGIwOS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNDI4JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDQyOFQyMDI1MzZaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT00YmQxNDUwOGQxMDgxODMwMmIwZjJmNWVjYTk1MTExZDU1OWQ2NWY1MTVjOGE0NDVkNWJmM2Q2MDRjOWJmNDNjJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.VJG5dtTiLAN5_BeWL6ziyUbo7uy7OVwGeSudnAVgujo)
- Temos um workspace no Athena com consultas em SQL mapeadas para respostas de análise das perguntas solicitadas pela gestão do 5GFlix (objetivo do exercício).
![Captura de Tela 2024-03-25 às 15 22 16](https://private-user-images.githubusercontent.com/275084/316610863-cbcf1d10-ac6d-4dfc-83c6-073701426444.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTQzMzYyMzYsIm5iZiI6MTcxNDMzNTkzNiwicGF0aCI6Ii8yNzUwODQvMzE2NjEwODYzLWNiY2YxZDEwLWFjNmQtNGRmYy04M2M2LTA3MzcwMTQyNjQ0NC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNDI4JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDQyOFQyMDI1MzZaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0zMzY0NDYxMmU2MDY4ZWIyMzVkZjllN2YyYjY5MGYyMTc3OWU2YzFhNzdlZjU3MDMyYmIxMjdiYmFlNGFlZDE0JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.86mQJ_T3-tpmwcS2NhXNpm8Gg1H8DexzPwO_GpBusr0)
![Captura de Tela 2024-03-25 às 15 27 19](https://private-user-images.githubusercontent.com/275084/316612185-19d611da-5a78-4d5b-8d1f-d240420a173d.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTQzMzYyMzYsIm5iZiI6MTcxNDMzNTkzNiwicGF0aCI6Ii8yNzUwODQvMzE2NjEyMTg1LTE5ZDYxMWRhLTVhNzgtNGQ1Yi04ZDFmLWQyNDA0MjBhMTczZC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNDI4JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDQyOFQyMDI1MzZaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT01MzVkNGIxYmIwYzNiM2ExZGNmNTA2NGRmMDJjZGVmZTI1YmYwNDU5OTEyOWU1OGYwMTBmNzAyYTZmZjNhYTEyJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.5zb8HhPqkMl25sLJ_jP4OzHF_YoHjV_ccUx2fnUo2A0)
![Captura de Tela 2024-03-25 às 15 28 47](https://private-user-images.githubusercontent.com/275084/316612564-464a5e35-495d-45aa-8a96-3ae73b2d538e.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTQzMzYyMzYsIm5iZiI6MTcxNDMzNTkzNiwicGF0aCI6Ii8yNzUwODQvMzE2NjEyNTY0LTQ2NGE1ZTM1LTQ5NWQtNDVhYS04YTk2LTNhZTczYjJkNTM4ZS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNDI4JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDQyOFQyMDI1MzZaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1hNGQ4M2E1OTAyMWI0YjI5OWVhYjZjM2RmMzFmYzQzOGJhODY5N2MwOTNjM2VmZDE0NTBkZDk5NjY0YTQ4MTBiJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.TMZQPSCMfdu3a4QlFKx1IaF0B_eDWunszjX5VXmTYeI)
1 Lost: Season 1, 2004 4.6709891019450955 2 Ghost in the Shell: Stand Alone Complex: 2nd Gig, 2005 4.586363636363636 3 The Simpsons: Season 6, 1994 4.581295988606693 4 Inu-Yasha, 2000 4.554434413170473 5 Lord of the Rings: The Return of the King: Extended Edition: Bonus Material, 2003 4.552
year total_launches 1 1922 1 2 1926 1 3 1917 1 4 1915 1 5 1924 2 6 1918 2 7 1916 2 8 1929 2 9 1931 2
Quantos filmes que possuem avaliação maior ou igual a 4.7, considerando apenas os filmes avaliados na última data de avaliação do dataset?
![Captura de Tela 2024-03-25 às 15 32 55](https://private-user-images.githubusercontent.com/275084/316613639-9653db28-dfe8-4c19-a8bf-a655e72ad007.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTQzMzYyMzYsIm5iZiI6MTcxNDMzNTkzNiwicGF0aCI6Ii8yNzUwODQvMzE2NjEzNjM5LTk2NTNkYjI4LWRmZTgtNGMxOS1hOGJmLWE2NTVlNzJhZDAwNy5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNDI4JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDQyOFQyMDI1MzZaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0zMDI5OGJhY2E4MzRhZTMwNzNjMmM2ZDRmYzFiM2RjMTVlMWMzYjhhZDE2YjcyNmM2ZGJkMWU5MDQwYWU3NmI2JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.WozYzJ7zxJ5m3JtaMTohcNKv25WC9YiMR206TS4j5w8)
780 filmes
![Captura de Tela 2024-03-25 às 15 34 37](https://private-user-images.githubusercontent.com/275084/316614079-5fc95f3e-17ef-4741-bf5c-5ada72815b74.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTQzMzYyMzYsIm5iZiI6MTcxNDMzNTkzNiwicGF0aCI6Ii8yNzUwODQvMzE2NjE0MDc5LTVmYzk1ZjNlLTE3ZWYtNDc0MS1iZjVjLTVhZGE3MjgxNWI3NC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNDI4JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDQyOFQyMDI1MzZaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT00OTA2ODFjMjQ4MzM0NzY5MTQ2ZmQyMzhhZWE5ZDViZTRkMTk3ODYwODNkOTczNzJhOGUzZGYwNGJmNjEyM2I0JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.Fm6-p0cnYpuxvadJkRs4ztwkVn6Gcd9w_gYbfEDR888)
cust_id total_eval 1 305344 4467 2 387418 4422 3 2439493 4195 4 1664010 4019 5 2118461 3769
The cdk.json
file tells the CDK Toolkit how to execute your app.
This project is set up like a standard Python project. The initialization
process also creates a virtualenv within this project, stored under the .venv
directory. To create the virtualenv it assumes that there is a python3
(or python
for Windows) executable in your path with access to the venv
package. If for any reason the automatic creation of the virtualenv fails,
you can create the virtualenv manually.
To manually create a virtualenv on MacOS and Linux:
$ python3 -m venv .venv
After the init process completes and the virtualenv is created, you can use the following step to activate your virtualenv.
$ source .venv/bin/activate
If you are a Windows platform, you would activate the virtualenv like this:
% .venv\Scripts\activate.bat
Once the virtualenv is activated, you can install the required dependencies.
$ pip install -r requirements.txt
At this point you can now synthesize the CloudFormation template for this code.
$ cdk synth
To add additional dependencies, for example other CDK libraries, just add
them to your setup.py
file and rerun the pip install -r requirements.txt
command.
cdk ls
list all stacks in the appcdk synth
emits the synthesized CloudFormation templatecdk deploy
deploy this stack to your default AWS account/regioncdk diff
compare deployed stack with current statecdk docs
open CDK documentation
Enjoy!