noahgift / spot_price_machine_learning

Machine Learning for Spot Prices

Home Page:https://www.amazon.com/Pragmatic-AI-Introduction-Cloud-based-Learning/dp/0134863860

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

CircleCI CircleCI Build Status

AWS Spot Price Machine Learning Tools

Machine Learning for Spot Prices Command-Line Tool and Jupyter Notebook. This material is covered in Chapter 9 of Pragmatic AI

Pragmatic AI Labs

alt text

You can continue learning about these topics by:

Contains two command-line tools

TO DO

  • Linear Optimization using Pyomo
  • Documentation

Assumptions, AWS account and/or profile with credentials.

[happygolucky]

aws_access_key_id =

aws_secret_access_key =

Running lint and tests

To make sure it works, do something like below:

make setup && make install

source <your virtualenv> (not required)

make lint

make test Runs Jupyter notebook test of project

Problems Solved: Recommended Spot Instances and Finding Deals

Describe Spot Instances by Median Price History and Price per ECU

(.pragia-aws) ➜  pragai-aws git:(master) ✗ ./spot-price-ml.py describe
              SpotPrice  price_ecu_spot  cluster  price_memory_spot
InstanceType                                                       
r4.xlarge         0.033           0.002        0              0.001
r4.large          0.017           0.002        0              0.001
r4.2xlarge        0.061           0.002        0              0.001
c3.2xlarge        0.098           0.003        1              0.007
r4.4xlarge        0.158           0.003        0              0.001
r3.xlarge         0.044           0.003        0              0.001
m4.4xlarge        0.185           0.003        0              0.003
m3.xlarge         0.043           0.003        0              0.003
m3.medium         0.009           0.003        0              0.002
c3.4xlarge        0.175           0.003        1              0.006
c3.8xlarge        0.368           0.003        1              0.006
cc2.8xlarge       0.344           0.004        1              0.006
r4.8xlarge        0.385           0.004        0              0.002
c3.large          0.027           0.004        1              0.007
c3.xlarge         0.053           0.004        1              0.007
r4.16xlarge       0.805           0.004        0              0.002
c4.2xlarge        0.125           0.004        1              0.008
c4.xlarge         0.057           0.004        1              0.008
m4.16xlarge       0.712           0.004        0              0.003
m4.10xlarge       0.440           0.004        0              0.003
c4.4xlarge        0.265           0.004        1              0.009
m3.2xlarge        0.094           0.004        0              0.003
c4.8xlarge        0.558           0.004        1              0.009
m4.2xlarge        0.133           0.005        0              0.004
m4.xlarge         0.062           0.005        0              0.004
g3.4xlarge        0.247           0.005        0              0.002
r3.4xlarge        0.237           0.005        0              0.002
r3.8xlarge        0.668           0.006        0              0.003
d2.4xlarge        0.319           0.006        0              0.003
x1.32xlarge       1.998           0.006        2              0.001
r3.2xlarge        0.195           0.007        0              0.003
i2.4xlarge        0.416           0.008        0              0.003
g3.8xlarge        0.762           0.008        0              0.003
i2.8xlarge        0.848           0.008        0              0.003
x1.16xlarge       1.394           0.008        2              0.001
d2.8xlarge        1.018           0.009        2              0.004
i2.2xlarge        0.335           0.012        1              0.005
g3.16xlarge       2.409           0.013        2              0.005
p2.8xlarge        1.800           0.019        2              0.004
p2.xlarge         0.267           0.022        2              0.004

Get Machine Learning Based Recommendations on Substitutes for Your Favorite Spot Instance

(.pragia-aws) ➜  pragai-aws git:(master) ✗ ./spot-price-ml.py recommend --instance c3.8xlarge 
              SpotPrice  price_ecu_spot  cluster  price_memory_spot
InstanceType                                                       
c3.2xlarge        0.098           0.003        0              0.007
c3.4xlarge        0.176           0.003        0              0.006
c3.8xlarge        0.370           0.003        0              0.006
c4.4xlarge        0.265           0.004        0              0.009
cc2.8xlarge       0.356           0.004        0              0.006
c3.large          0.027           0.004        0              0.007
c3.xlarge         0.053           0.004        0              0.007
c4.2xlarge        0.125           0.004        0              0.008
c4.8xlarge        0.557           0.004        0              0.009
c4.xlarge         0.060           0.004        0              0.008
hi1.4xlarge       0.370           0.011        0              0.006

Launch Self-Terminating Spot Instances

Note, the defaults will need to be set for your environment: security groups, keyname, etc.

(.spot-price-ml) ➜  spot_price_machine_learning git:(master) ./spot_launcher.py launch --help         
Usage: spot_launcher.py launch [OPTIONS]

  Request spot instance

Options:
  --instance TEXT       Instance Type
  --duration TEXT       Duration
  --keyname TEXT        Key Name
  --profile TEXT        IamInstanceProfile
  --securitygroup TEXT  Key Name
  --ami TEXT            Key Name
  --help                Show this message and exit.

To launch with a longer duration, say 1 hour and 55 minutes:

(.pragia-aws) ➜  pragai-aws git:(master) ✗ ./spot_launcher.py launch --duration 115


2017-09-20 06:46:53,046 - __main__ - INFO - SPOT REQUEST DATA: {'SpotPrice': '0.8', 'Type': 'one-time', 'InstanceCount': 1, 'LaunchSpecification': {'ImageId': 'ami-6df1e514', 'InstanceType': 'r4.large', 'KeyName': 'pragai', 'IamInstanceProfile': {'Arn': 'arn:aws:iam::561744971673:instance-profile/admin'}, 'UserData': 'CiAgICAgICAgI2Nsb3VkLWNvbmZpZwogICAgICAgIHJ1bmNtZDoKICAgICAgICAgLSBlY2hvICJoYWx0IiB8IGF0IG5vdyArIDExNSBtaW4KICAgIA==', 'BlockDeviceMappings': [{'DeviceName': '/dev/xvda', 'Ebs': {'DeleteOnTermination': True, 'VolumeType': 'gp2', 'VolumeSize': 8}}], 'SecurityGroupIds': ['sg-61706e07']}}

Seaborn Plots from Juypter

Spot Prices vs Spot Prices/ECU

Cluster Facets

Jupyter Notebook Version

https://github.com/noahgift/spot_price_machine_learning/blob/master/notebooks/spot_pricing_ml.ipynb

About

Machine Learning for Spot Prices

https://www.amazon.com/Pragmatic-AI-Introduction-Cloud-based-Learning/dp/0134863860

License:MIT License


Languages

Language:Jupyter Notebook 99.8%Language:Python 0.2%Language:Makefile 0.0%