Windseed
Extendable skeleton for developing web applications using [Tornado] (https://github.com/tornadoweb/tornado) and [peewee] (https://github.com/coleifer/peewee)
For its implementation using aiohttp refer to aiowing
Prerequisites
- Ubuntu 14.10
- PostgreSQL 9.3
- Python 3.5+
- nginx
- Supervisor
Structure
-
backups - folder for pg_dump to save dump using bash/backup.sh
-
bash - various bash scripts including project settings within environment variables:
- backup.sh - create pg_dump in backups folder
- env.sh - contains all environment variables for project
- superuser.sh - create superuser
- supervisor.sh - script for supervisor
- tables.sh - drop database, crate database and create db tables
- tests.sh - run tests
- windseed.sh - run application
-
conf - config files for nginx and supervisor for development and production
-
static - project static files, mainly Bootstrap 3, robots.txt, admin styles
-
templates - project templates
-
windseed - windseed package:
- apps - project's applications folder, each app within its folder has:
- handlers.py - app handlers
- models.py - app models
- tests.py - app tests
- urls.py - app URLs and routes
- base - base classes:
- handler.py - base handler class
- model.py - base model class
- test.py - base test class
- settings - project settings
- db.py - postgresql database pool
- env.py - environment variables from env.sh and paths
- urls.py - project routes
- utils - project utilities:
- superuser.py - create superuser
- tables.py - create tables
- apps - project's applications folder, each app within its folder has:
-
windseed.py - Windseed Tornado application
Install Python 3.5+
sudo add-apt-repository ppa:fkrull/deadsnakes
sudo apt-get update
sudo apt-get install python3.5 python3.5-venv python3.5-dev
Setup environment and packages
pyvenv-3.5 .env
source .env/bin/activate
pip install -r requirements.txt
Generate cookie secret
openssl rand -base64 40
and put it to env.sh
Run
bash scripts/tables.sh - create db and tables
bash scripts/superuser.sh - create superuser
bash scripts/tests.sh - run tests
bash scripts/windseed.sh - run project
Nginx
sudo apt-get install nginx
sudo cp conf/prod-nginx.conf /etc/nginx/nginx.conf
sudo service nginx restart
Ubuntu/nginx file limits
/etc/security/limits.conf:
soft nofile 16384
hard nofile 16384
/etc/sysctl.conf:
fs.file-max = 16384
/etc/pam.d/common-session:
session required pam_limits.so
SSL certificate
sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout windseed.key -out windseed.crt
Supervisor
sudo apt-get install supervisor
pgrep -fl supervisor
sudo service supervisor start
sudo cp conf/supervisord.conf /etc/supervisor/supervisord.conf
sudo supervisord -c conf/supervisord.conf
sudo service supervisor restart
sudo unlink /tmp/supervisor.sock
kill related processes
Performance
Hardware:
- Intel® Core™ i7-5500U CPU @ 2.40GHz × 4
- 16 Gb RAM
Testing with [wrk] (https://github.com/wg/wrk):
- 10 000, 100 000, 1 000 000 records in table
- 1 minute test with 1 minute timeout
- 1 and 10 threads
- From 10 to 100 connections with 10 connections step
- Pagination using SQL OFFSET and pagination using pagination table
- First and last page
- Sorted by name
- Compare Windseed and [Django] (https://github.com/embali/windseed-django) apps with similar functionality
Start wrk
wrk -t1 -c10 -d1m --timeout 1m "http://localhost:8000/?page=1"
Start Windseed
bash bash/windseed.sh
Start Django
uwsgi --module=djangotest.wsgi:application
--env DJANGO_SETTINGS_MODULE=djangotest.settings
--http=127.0.0.1:8000
--processes 1
10 000 records
Windseed
c10 | c20 | c30 | c40 | c50 | c60 | c70 | c80 | c90 | c100 | |
---|---|---|---|---|---|---|---|---|---|---|
t1, offset, first | 4.35 | 4.21 | 4.26 | 4.15 | 4.29 | 4.36 | 4.38 | 4.18 | 4.49 | 4.38 |
t1, offset, last | 21.17 | 20.40 | 21.15 | 21.17 | 21.25 | 20.54 | 20.48 | 21.03 | 21.37 | 21.22 |
t1, table, first | 7.47 | 6.78 | 6.94 | 6.82 | 7.13 | 7.34 | 7.28 | 7.39 | 7.11 | 7.02 |
t1, table, last | 6.97 | 6.75 | 6.91 | 6.64 | 6.59 | 7.10 | 7.22 | 7.08 | 7.02 | 7.07 |
t10, offset, first | 4.44 | 4.07 | 3.99 | 4.19 | 4.10 | 4.05 | 4.28 | 4.27 | 4.26 | 3.95 |
t10, offset, last | 20.17 | 19.97 | 20.01 | 20.23 | 20.55 | 20.77 | 20.74 | 20.45 | 20.55 | 20.94 |
t10, table, first | 6.84 | 6.39 | 7.09 | 7.27 | 7.12 | 7.18 | 7.24 | 7.35 | 7.27 | 7.60 |
t10, table, last | 7.51 | 7.04 | 7.24 | 7.40 | 7.34 | 7.31 | 7.17 | 7.60 | 7.73 | 7.73 |
Django
c10 | c20 | c30 | c40 | c50 | c60 | c70 | c80 | c90 | c100 | |
---|---|---|---|---|---|---|---|---|---|---|
t1, offset, first | 17.80 | 17.38 | 18.01 | 18.10 | 18.18 | 18.03 | 18.33 | 18.80 | 17.96 | 18.11 |
t1, offset, last | 37.15 | 36.36 | 36.32 | 36.52 | 37.31 | 37.62 | 37.55 | 37.66 | 38.24 | 38.46 |
t1, table, first | 18.36 | 17.16 | 17.26 | 17.62 | 17.30 | 17.25 | 17.40 | 17.41 | 17.27 | 17.46 |
t1, table, last | 17.64 | 17.01 | 17.04 | 17.21 | 17.31 | 17.35 | 17.31 | 17.07 | 16.27 | 15.81 |
t10, offset, first | 16.49 | 16.47 | 15.96 | 16.23 | 16.11 | 16.10 | 16.44 | 16.23 | 16.34 | 16.69 |
t10, offset, last | 33.44 | 32.70 | 32.91 | 33.58 | 33.73 | 33.61 | 33.52 | 33.98 | 33.98 | 34.29 |
t10, table, first | 16.04 | 15.16 | 15.89 | 15.43 | 15.58 | 15.77 | 16.03 | 18.46 | 18.46 | 18.52 |
t10, table, last | 18.54 | 18.29 | 18.30 | 15.43 | 16.89 | 17.40 | 17.52 | 17.49 | 17.79 | 17.65 |
![1 thread, offset pagination, first page](/charts/104/20160330115515.png?raw=true "1 thread, offset pagination, first page")
![1 thread, offset pagination, last page](/charts/104/20160330120051.png?raw=true "1 thread, offset pagination, last page")
![1 thread, table pagination, first page](/charts/104/20160330120639.png?raw=true "1 thread, table pagination, first page")
![1 thread, table pagination, last page](/charts/104/20160330121024.png?raw=true "1 thread, table pagination, last page")
![10 threads, offset pagination, first page](/charts/104/20160330124517.png?raw=true "10 threads, offset pagination, first page")
![10 threads, offset pagination, last page](/charts/104/20160330124836.png?raw=true "10 threads, offset pagination, last page")
![10 threads, table pagination, first page](/charts/104/20160330125228.png?raw=true "10 threads, table pagination, first page")
![10 threads, table pagination, last page](/charts/104/20160330125539.png?raw=true "10 threads, table pagination, last page")
100 000 records
Windseed
c10 | c20 | c30 | c40 | c50 | c60 | c70 | c80 | c90 | c100 | |
---|---|---|---|---|---|---|---|---|---|---|
t1, offset, first | 12.64 | 12.84 | 12.51 | 12.35 | 12.54 | 12.49 | 12.53 | 12.45 | 12.79 | 12.90 |
t1, offset, last | 340.91 | 348.84 | 379.75 | 400.00 | 447.76 | 483.87 | 495.87 | 545.45 | 588.24 | 600.00 |
t1, table, first | 33.78 | 13.59 | 13.87 | 13.82 | 13.69 | 14.00 | 13.73 | 14.08 | 14.11 | 13.99 |
t1, table, last | 14.55 | 14.34 | 14.11 | 14.87 | 14.19 | 15.03 | 15.80 | 14.49 | 15.67 | 15.97 |
t10, offset, first | 12.47 | 12.60 | 12.56 | 12.76 | 12.76 | 12.67 | 12.64 | 12.47 | 12.61 | 14.21 |
t10, offset, last | 355.03 | 365.85 | 387.10 | 413.79 | 483.87 | 512.82 | 560.75 | 606.06 | 666.67 | 833.33 |
t10, table, first | 36.56 | 14.10 | 14.32 | 14.49 | 14.40 | 15.54 | 14.34 | 14.45 | 14.26 | 14.69 |
t10, table, last | 14.97 | 14.23 | 14.26 | 14.29 | 14.30 | 14.31 | 14.45 | 14.65 | 14.50 | 14.12 |
Django
c10 | c20 | c30 | c40 | c50 | c60 | c70 | c80 | c90 | c100 | |
---|---|---|---|---|---|---|---|---|---|---|
t1, offset, first | 33.92 | 33.67 | 33.50 | 32.95 | 34.05 | 33.90 | 33.78 | 33.94 | 34.05 | 35.84 |
t1, offset, last | 402.68 | 392.16 | 416.67 | 441.18 | 480.00 | 508.47 | 555.56 | 618.56 | 740.74 | 800.00 |
t1, table, first | 60.98 | 23.93 | 24.86 | 24.84 | 23.97 | 24.92 | 24.58 | 25.06 | 26.64 | 24.74 |
t1, table, last | 25.02 | 23.73 | 24.35 | 24.59 | 24.59 | 24.93 | 25.13 | 24.39 | 24.36 | 24.52 |
t10, offset, first | 32.75 | 32.35 | 33.15 | 31.90 | 32.12 | 32.38 | 32.52 | 32.59 | 32.86 | 33.99 |
t10, offset, last | 372.67 | 372.67 | 394.74 | 428.57 | 458.02 | 491.80 | 540.54 | 600.00 | 689.66 | 810.81 |
t10, table, first | 55.05 | 23.10 | 23.32 | 23.36 | 23.45 | 23.56 | 23.68 | 23.81 | 23.83 | 24.09 |
t10, table, last | 24.10 | 23.32 | 23.42 | 23.95 | 25.21 | 25.17 | 24.42 | 24.69 | 24.27 | 24.40 |
![1 thread, offset pagination, first page](/charts/105/20160330033437.png?raw=true "1 thread, offset pagination, first page")
![1 thread, offset pagination, last page](/charts/105/20160330034042.png?raw=true "1 thread, offset pagination, last page")
![1 thread, table pagination, first page](/charts/105/20160330040018.png?raw=true "1 thread, table pagination, first page")
![1 thread, table pagination, last page](/charts/105/20160330040436.png?raw=true "1 thread, table pagination, last page")
![10 threads, offset pagination, first page](/charts/105/20160330040722.png?raw=true "10 threads, offset pagination, first page")
![10 threads, offset pagination, last page](/charts/105/20160330041002.png?raw=true "10 threads, offset pagination, last page")
![10 threads, table pagination, first page](/charts/105/20160330041221.png?raw=true "10 threads, table pagination, first page")
![10 threads, table pagination, last page](/charts/105/20160330041456.png?raw=true "10 threads, table pagination, last page")
1 000 000 records
Windseed
c10 | c20 | c30 | c40 | c50 | c60 | c70 | c80 | c90 | c100 | |
---|---|---|---|---|---|---|---|---|---|---|
t1, offset, first | 95.69 | 94.15 | 98.36 | 107.33 | 110.50 | 119.52 | 115.61 | 126.05 | 123.46 | 126.85 |
t1, offset, last | 405.41 | 348.84 | 357.14 | 375.00 | 465.12 | 413.79 | 483.87 | 545.45 | 526.32 | 600.00 |
t1, table, first | 212.01 | 102.21 | 109.29 | 106.95 | 109.49 | 105.63 | 108.89 | 110.91 | 113.21 | 115.38 |
t1, table, last | 116.73 | 100.50 | 101.52 | 103.63 | 105.45 | 106.95 | 108.11 | 110.50 | 113.64 | 115.38 |
t10, offset, first | 102.04 | 107.53 | 102.39 | 102.92 | 106.19 | 107.72 | 110.29 | 110.29 | 114.50 | 119.28 |
t10, offset, last | 408.16 | 320.86 | 340.91 | 357.14 | 382.17 | 410.96 | 476.19 | 521.74 | 555.56 | 582.52 |
t10, table, first | 214.29 | 105.45 | 110.29 | 101.18 | 102.39 | 106.01 | 106.19 | 107.72 | 109.69 | 113.21 |
t10, table, last | 114.72 | 99.01 | 98.85 | 101.35 | 103.99 | 103.45 | 106.38 | 111.11 | 112.36 | 128.76 |
Django
c10 | c20 | c30 | c40 | c50 | c60 | c70 | c80 | c90 | c100 | |
---|---|---|---|---|---|---|---|---|---|---|
t1, offset, first | 199.34 | 208.33 | 212.01 | 219.78 | 223.05 | 236.22 | 250.00 | 255.32 | 263.16 | 276.50 |
t1, offset, last | 606.06 | 437.96 | 476.19 | 512.82 | 560.75 | 606.06 | 689.66 | 800.00 | 882.35 | 1071.43 |
t1, table, first | 314.14 | 108.11 | 111.11 | 114.72 | 120.00 | 119.52 | 121.95 | 124.22 | 127.93 | 129.87 |
t1, table, last | 132.45 | 109.09 | 111.32 | 112.57 | 116.96 | 119.05 | 123.46 | 131.29 | 132.16 | 131.00 |
t10, offset, first | 198.02 | 204.78 | 212.01 | 219.78 | 230.77 | 243.90 | 247.93 | 262.01 | 271.49 | 289.86 |
t10, offset, last | 625.00 | 422.54 | 458.02 | 495.87 | 535.71 | 588.24 | 659.34 | 750.00 | 857.14 | 983.61 |
t10, table, first | 310.88 | 111.94 | 110.29 | 114.29 | 124.74 | 123.20 | 123.20 | 126.05 | 123.46 | 127.66 |
t10, table, last | 137.30 | 117.19 | 117.19 | 116.96 | 123.46 | 126.85 | 126.05 | 131.00 | 131.00 | 131.00 |
![1 thread, offset pagination, first page](/charts/106/20160330042211.png?raw=true "1 thread, offset pagination, first page")
![1 thread, offset pagination, last page](/charts/106/20160330042437.png?raw=true "1 thread, offset pagination, last page")
![1 thread, table pagination, first page](/charts/106/20160330042704.png?raw=true "1 thread, table pagination, first page")
![1 thread, table pagination, last page](/charts/106/20160330042920.png?raw=true "1 thread, table pagination, last page")
![10 threads, offset pagination, first page](/charts/106/20160330043142.png?raw=true "10 threads, offset pagination, first page")
![10 threads, offset pagination, last page](/charts/106/20160330043421.png?raw=true "10 threads, offset pagination, last page")
![10 threads, table pagination, first page](/charts/106/20160330043646.png?raw=true "10 threads, table pagination, first page")
![10 threads, table pagination, last page](/charts/106/20160330043929.png?raw=true "10 threads, table pagination, last page")