EmpregoLigado / de-code-challenge-jr

Data Engineer Code Challenge - Levee

Through this challenge, we want to know your development skills and especially how you solve problems.

No time to take the challenge?

You can send a pull request link with your contribution to any Open Source project or any project of your own that you believe demonstrates your technical level and the quality of your code. Remember that the more code we view, the better it will be for us to evaluate you. :)

About the challenge

We have 2 data subsets:

1. jobs.txt: Contains the jobs openings separated by category (file "jobs.txt")
2. category.txt: Contains the category name and code (file "category.txt")

See the structure below:

Jobs:

Partner ID	Job Title	Category ID	Expires At	Open Position Amount
1123	Seller	3	1/21/2018	1

Categories:

Category ID	Title
1	Technology

Based on this data, we want you to create a solution that shows the following calculations:

1. The number of Open Positions per Category Name;
2. The last three Jobs that have expired.

Make sure:

• Your code should allow further executions. That is, when a new file arrives (with the same columns) the code must runs successfully;
• Your code should assure the quality of the data.

It is not mandatory, but it is important to observe the recommendations below.

• It would be great if you load this data to any Database Systems of your choice;
• It would be great if you use Python in some part of your solution;

Prerequisites

Your test must have a README with the necessary steps to:

• Install the dependencies;
• Run the project;
• See the output data;

Technology

• Python. Scala can be an alternative.
• SQL
• As a DB, we recommend PostgreSQL, MySQL or MongoDB, but you can choose any other.

We like:

• Workload Management;
• Data Visualization Tools;
• Spark;
• Documentation;
• Docker;
• Structured commits;
• Tests;
• CI.

What is being evaluated

Your ability to understand a problem, design a solution and apply it. We want to see how you solve the proposed problem and your skills with the proposed technologies.

Submission

To send us your code, you can:

• Fork this repository and send us a pull request.
• Give us access to your repository on github, bitbucket or gitlab. Add the user bonigauglitz.
• If you need to talk to us: [e-mail]