mikepqr / serverless-for-data-scientists

Code and notebooks for a talk given at PyBay, 2018-08-19

Serverless for data scientists

Code and notebooks for a talk given at PyBay, 2018-08-19.

• Slides and words
• Video

Contents

1. Deployment of a basic flask application with zappa
2. Pywren demo
3. Hyperparameter optimization with pywren
4. Model deployment on AWS Lambda with zappa

Projects and resources mentioned in the talk

• zappa -- tool for AWS Lambda deployment of Python applications
• dwx -- Twitter bot that runs on AWS Labmda.
• Serving 39 Million Requests for $370/Month -- AWS Lambda migration example
• Occupy the Cloud: Distributed Computing for the 99% -- the pywren paper
• pywren -- tool for embarrassingly parallel computation using AWS Lambda as the backend
• riscamp 2017 pywren tutorial -- more detailed look at pywren than my talk
• pywren web scraping -- blog post that inspired the webscraping example in the talk
• 305 Million Solutions to The Black-Scholes Equation in 16 Minutes with AWS Lambda -- large scale embarrassingly parallel numerical computation example
• nips.json -- code to scrape the NIPS website using pywren (and the data scraped)
• Encoding, Fast and Slow: Low-Latency Video Processing Using Thousands of Tiny Threads -- an example of AWS Lambda as part of a long-lived data processing pipeline (doesn't use pywren)