Advanced Python 3
This repository presents advanced usage of Python incl OOP (object-oriented programming) implementation, working with relational databases and NOSQL databases + fetching HTML data.
Projects
Projects covering Advanced Python using Python3.7
- classes in Python : association
- classes in Python : inheritance
- classes in Python : abstract classes and interfaces
- fetching html data: BeautifulSoup
- using sqlite database
- using ORM: SQLAlchemy
- working with MongoDB database
Classes in Python: association
Classes association
Project which simulates shuffling of card deck (52 cards). You should print sorted deck of cards, after shuffle it and print again.
Class Card:
- constructor
__init__(self, suit, rank)
sets the Card - overrided
__str__(self)
method prints card name, f.e. "jack of hearts"
Class Deck:
- contains two lists of ranks and suits
- constructor
__init__(self)
sets the Deck: creates list of 52 objects of class Card - method
shuffle(self)
shuffles cards in Deck using random - overrided
__str__(self)
method prints all cards in a deck
Classes in Python: inheritance
Classes inheritance
Project with class hierarchy: reusing methods of base classes. *In Python it is not necessary to declare virtual methods
Class Shape:
- 3 default fields: name of figure, area and perimiter
- method
ComputePerim(self)
is virtual-like to be overrided in derived classes - method
ComputeArea(self)
is virtual-like to be overrided in derived classes - overrided
__str__(self)
method prints figure name and its calculated area and perimiter
Class Circle:
- inherits from Shape
- constructor
__init__(self, radius)
sets radius of circle - overrided method
ComputePerim(self)
calculates the perimeter of circle - overrided method
ComputeArea(self)
calculates the area of circle
Class Rectangle:
- inherits from Shape
- constructor
__init__(self, width, height)
sets width and height of rectangle - overrided method
ComputePerim(self)
calculates the perimeter of rectangle - overrided method
ComputeArea(self)
calculates the area of rectangle
Class Square:
- inherits from Rectangle
- constructor
__init__(self, width)
sets side of shape and calls the cinstructor of base class rectangle (constructor chaining).
Abstract classes and interfaces in Python
Declaration of abstract class and interface in python3
This task and UML class diagram were taken from C# / Java project to draw an analogy between OOP style in Python and OOP in C#/Java. From this case it's clear that interfaces in Python are a little bit useless: Python has a multiple inheritance, C#/Java - single
ABC (abstract class) is a part of Python Standart Library.
As for interfaces, module zope can be used.
pip install zope
Class HotDrink (abstract):
from abc import ABC, abstractmethod
- 2 protected fields: milk = 3, sugar = 3 (default values for derived classes)
- 2 abstract properties: Milk, Sugar
- 3 abstract methods: AddMilk, AddSugar, Drink
- all class members which have
@abstractmethod
decorator MUST be overrided in child classes.
Interface ICup:
from zope.interface import Interface, Attribute
- 2 properties
- 2 methods.
Class CupOfCoffee: - implements ICup interface
- inherits abstract class HotDrink
- has its own property BeanType: arabica or robusta
@implementer(ICup)
class CupOfCoffee(HotDrink):
pass
Class CupOfTea:
- implements ICup interface
- inherits abstract class HotDrink
- has its own property LeafType: green or black
@implementer(ICup)
class CupOfTea(HotDrink):
pass
Python Text Web Scraping
BeautifulSoup + static content
Get Full Moon Calendar data from fullmoon.info web site for 150 years. Every year is presented in separate page:https://www.fullmoon.info/en/fullmoon-calendar/
X, where X is a year. create table and insert fetched data fetched into sqlite database.
- Install packages:
pip install beautifulsoup4
pip install requests
Beautiful Soup (bs4) = Python library for pulling data out of HTML and XML files.
requests = HTTP library
sqlite3 = Python standard library contains the SQLite module.
- Create function which takes list of years and scrapes every of 150 pages. Every html-page consists of such span-block, f.e. for
https://www.fullmoon.info/en/fullmoon-calendar/1901.html
:
<table class="table-calendar">
<tbody>
<tr><td class="weekday">Saturday</td><td class="date">5 January 1901</td><td class="time">1:13:24 am</td><td class="notes"> </td></tr>
<tr><td class="weekday">Sunday</td><td class="date">3 February 1901</td><td class="time">4:29:42 pm</td><td class="notes"> </td></tr>
<tr><td class="weekday">Tuesday</td><td class="date">5 March 1901</td><td class="time">9:04:06 am</td><td class="notes"> </td></tr>
<tr><td class="weekday">Thursday</td><td class="date">4 April 1901</td><td class="time">2:20:06 am</td><td class="notes"> </td></tr>
<tr><td class="weekday">Friday</td><td class="date">3 May 1901</td><td class="time">7:18:42 pm</td><td class="notes"> [/]</td></tr>
<tr><td class="weekday">Sunday</td><td class="date">2 June 1901</td><td class="time">10:52:42 am</td><td class="notes"> </td></tr>
<tr><td class="weekday">Tuesday</td><td class="date">2 July 1901</td><td class="time">12:17:36 am</td><td class="notes"> </td></tr>
<tr><td class="weekday">Wednesday</td><td class="date">31 July 1901</td><td class="time">11:33:42 am</td><td class="notes"> [+]</td></tr>
<tr><td class="weekday">Thursday</td><td class="date">29 August 1901</td><td class="time">9:20:48 pm</td><td class="notes"> </td></tr>
<tr><td class="weekday">Saturday</td><td class="date">28 September 1901</td><td class="time">6:35:42 am</td><td class="notes"> </td></tr>
<tr><td class="weekday">Sunday</td><td class="date">27 October 1901</td><td class="time">4:06:12 pm</td><td class="notes"> [*]</td></tr>
<tr><td class="weekday">Tuesday</td><td class="date">26 November 1901</td><td class="time">2:17:30 am</td><td class="notes"> </td></tr>
<tr><td class="weekday">Wednesday</td><td class="date">25 December 1901</td><td class="time">1:15:48 pm</td><td class="notes"> </td></tr>
</tbody>
</table>
We need to get class="table-calendar"
.
- You can find full csv dataset on my Kaggle page: Full Moon Calendar 1900-2050
- Before inserting data into database we need to convert date and time in one object: 5 January 1901 1:13:24 am -> 1901-01-05 01:13:24
Python Images Web Scraping
BeautifulSoup + static content
Fetch data from minerals.net: create folders with gemstones names and download images of them in each coresponding folder.
- Get the page with list of all gemstones and find HTML-element with them:
page = 'https://www.minerals.net/GemStoneMain.aspx'
resp = requests.get(page)
if resp.status_code == 200:
html = resp.text
soup = BeautifulSoup(html, 'html.parser')
table_gems=soup.find_all('table',{'id':'ctl00_ContentPlaceHolder1_DataList1'})
-
Parse links to the pages of each gemstone and create a dictionary {gemstone name : link }
-
Parse each page to get pictures of gemstones
table_images=soup.find_all('table',{'id':'ctl00_ContentPlaceHolder1_DataList1'})
You can find full gemstones images dataset from multiple sources on my Kaggle page: Gemstones Images
Gismeteo Web Scraping + SQLAlchemy
BeautifulSoup + SQLite
Fetch data from gismeteo.ru: city + period and put it into database using ORM sqlalchemy v2
- Install packages:
pip install beautifulsoup4
pip install sqlalchemy
- Get example page weather in Moscow 14 days and find HTML-element with class
widget-items
. - Get day of week, date, description, temperature (max), temperature (min), wind (m/s), wind direction, precipitation, humidity, radiation.
- Create class Weather with corresponding fields.
- Create SqlAlchemy Engine (connection), SqlAlchemy Table with 10 columns and mapper between class Weather and SqlAlchemy Table.
- Process data from site to be string, date, float etc and put it into database. Aware of duplicates.
Python + MongoDB
Working with nosql database using python
basic operations working with MongoDB using pymongo.
pip install pymongo
opt: download and install GUI Compass
The sample database is weather.json. Import data into MongoDB instance and check the collection: Schema -> Analyze.
Using Compass check data:
{"airTemperature.value": 3}
{"airTemperature.value": {$gt: 10}}
{"wind.type": {$ne: "N"}}
{"atmosphericPressureChange" : {$exists: true}}
Simple examples of CRUD operations in MongoDB using pymongo: insert_one (insert_many), find, update_one (update_many), delete_one (delete_many)
.
All these methods expect dictionary/dictionaries as a parameter(s).