WorkShoft / python-developer-delectatech

Python Developer test

Python Developer - Mikel Losada

DOCS

Setting up the project

I have provided some Make commands to set up the Docker containers and prepare the MySQL database.

$ make build

Run the containers on a terminal:

$ make up

Set up MySQL like this from another terminal:

$ make setupmysql

MySQL / MariaDB import

Restoring the database

I have provided a dump created with mysqldump at dumps/mysql.gz.

This dump has been created using the Make dumpsql command (in the Makefile of this project).

Loading fixtures

Copy restaurants_input.json and segments_input.json to the root folder of this project (where manage.py is).

Run this command to load the data:

$ make loadsql

Generic queries

I have provided a custom django-admin command to perform generic queries from the command line.

The query command takes Python dictionaries such as

{'popularity_rate': {'gt': 5.5},'satisfaction_rate': {'ne': None},}

and displays the result of the query.

Here’s an example of using the command to retrieve the first restaurant with a popularity rate over 5.5:

$ docker-compose -f local.yml run django python manage.py query \
"{'popularity_rate': {'gt': 5.5},'satisfaction_rate': {'ne': None},}" \
--first

Venta los Cabezuelos

By default, queries are run against the SQL database.

You can query the MongoDB database by adding the mongo flag. The as_dict flag has no effect on Mongo queries.

Here’s an example of using the query command against Mongo to get restaurants with over 100 reviews that have a popularity rate between 8 and 8.2:

$ docker-compose -f local.yml run django python manage.py query \ 
"{'popularity_rate': {'gt': 8, 'lt': 8.2}, 'total_reviews': {'gt': 100}, \
'satisfaction_rate': {'ne': None},}" \ 
--mongo 

Here are the command options:

OPTION	INFO
first	displays first query result
as_dict	displays query result(s) in dict format
mongo	queries the MongoDB database

The source code for these queries can be found at:

• restaurants/repositories/sqlrepository.py
• restaurants/repositories/mongorepository.py
• restaurants/management/commands/query.py

If you wish to do queries with Python, you can do so by creating your own sqlrepository.SqlRestaurantRepo and mongorepository.MongoRestaurantRepo objects.

Here is an example with the Django shell:

$ make shell

from restaurants.repositories import sqlrepository

sqlrepo = sqlrepository.SqlRestaurantRepo()

params = {
    "popularity_rate": {"gt": 5.5},
    "satisfaction_rate": {"ne": None},
}

results = sqlrepo.query_restaurants(params)
first_result = sqlrepo.query_restaurants_first(params)

Exporting SQL to JSON

Exporting the SQL database to a JSON file is supported, via the dumpsqltojson django-admin custom command.

$ make dumpsqltojson

The format of the JSON file will be like this:

{
  "e97827dfe4714f6685af2bacfd9d2e65": {
    "name": "Small Segment",
    "size": 1000,
    "uidentifier": "e97827dfe4714f6685af2bacfd9d2e65",
    "restaurants": [
      {
	"name": "El Caf\u00e8 Blau",
	"street_address": "Calle de Casp",
	"latitude": 41.3984659,
	"longitude": 2.1830135,
	"city_name": "Barcelona",
	"popularity_rate": 6.57,
	"satisfaction_rate": 6.95,
	"total_reviews": 2,
	"uidentifier": "d907f70e0b9145b381778d5a20032a25",
	"average_price": 5
      },
      ......
    ]
  }
}

MongoDB import

Restoring the database

I have provided a dump created with mongodump at dumps/mongo.gz.

This dump has been created using the Make dumpmongo command (in the Makefile of this project).

Loading fixtures

Copy restaurants_input.json and segments_input.json to the root folder of this project (where manage.py is). Run this command to load the data:

$ make loadmongo

Exporting MongoDB to JSON

Exporting the SQL database to a JSON file is supported, via the dumpmongotojson django-admin custom command.

$ make dumpmongotojson

The format of the JSON file will be like this:

{
  "e97827dfe4714f6685af2bacfd9d2e65": {
    "name": "Small Segment",
    "size": 1000,
    "uidentifier": "e97827dfe4714f6685af2bacfd9d2e65",
    "restaurants": [
      {
	"name": "El Caf\u00e8 Blau",
	"street_address": "Calle de Casp",
	"latitude": 41.3984659,
	"longitude": 2.1830135,
	"city_name": "Barcelona",
	"popularity_rate": 6.57,
	"satisfaction_rate": 6.95,
	"total_reviews": 2,
	"uidentifier": "d907f70e0b9145b381778d5a20032a25",
	"average_price": 5
      },
      ......
    ]
  }
}

Data migration

MongoDB to SQL

restaurants/management/commands/mongotosql.py

$ make mongotosql

SQL to MongoDB

restaurants/management/commands/sqltomongo.py

$ make sqltomongo

CRUD app

The CRUD app uses Django and MySQL.

You need to create a Django superuser to access the app.

$ make createsuperuser

Run the containers:

$ make up

You can now access the app at http://0.0.0.0:8000/admin/ with your user credentials.

You will receive a confirmation message in the terminal where you are running the application.

The Restaurant and Segment views are at http://0.0.0.0:8000/admin/restaurants/restaurant/ and http://0.0.0.0:8000/admin/restaurants/segment/ respectively.

Running tests

I have provided a test Make command thats runs pytest on restaurants/tests and creates an HTML report at docs/coverage/index.html

$ make test

Appendix

Make commands

build

Builds Django, MariaDB and MongoDB containers.

setupmysql

Creates the python_developer SQL database.

This script creates the mikel user with password ‘mikel’ and grants it all privileges, flushing them afterwards.

You will need to run this command after you do a fresh build.

up

Runs Django, MariaDB and MongoDB containers.

Access Django at

0.0.0.0:8000 

createsuperuser

Creates a Django superuser.

Run this once to create a superuser and access the admin site at

0.0.0.0:8000/admin

loadmongo

Creates a Mongo collection in the python_developer_db database.

segments_input.json and restaurants_input.json must be in the root folder.

loadsql

Populates the python_developer MySQL database.

segments_input.json and restaurants_input.json must be in the root folder.

dumpmongo

Creates a gzip-compressed dump of the python_developer_db Mongo database using mongodump.

dumpsql

Creates a SQL dump of the python_developer MySQL database using mysqldump.

dumpsqltojson

Creates a JSON dump of the python_developer MySQL database.

dumpmongotojson

Creates a JSON dump of the segment_collection Mongo collection.

sqltomongo

Migrates the SQL database to a MongoDB collection.

mongotosql

Migrates the MongoDB segment collection to the Restaurant and Segment SQL tables.

shell

Enters the IPython shell in the Django container.

makemigrations

Creates Django migration files.

migrate

Migrates the database using migration files.

Get Docker

https://docs.docker.com/engine/install/

Running Docker as a non sudo user

https://docs.docker.com/engine/install/linux-postinstall/

What tool did you use to create this documentation?

Emacs and org-mode, with the Big Blow theme. The source is at docs/README.org.

How do you generate HTML on Org Mode?

By pressing C-c C-e h-h.

How did you set up this project?

I used Django Cookiecutter.