This repository provides runtime for running R
on AWS Lambda. It also provides a workflow for using dependecies like R packages by creating additional layers.
AWS Lambda is a serverless solution for running scripts triggered by various events. It supports several runtimes by default (including Python
, Go
, Java
, C#
, Node.js
, Ruby
and PowerShell
), but it is possible to create custom runtime environments to use almost any other, non-supported language.
This approach uses AWS Lambda component, namely layers, as containers for R
environment and provided packages. In your lambda function, you can use pre-built layers we provide or you can create your own ones with specific content (different R
version, custom packages, etc.).
Using R scripts in AWS Lambda is just a few steps:
- In AWS Console create a new function (suggested region:
eu-central-1
). ChooseCustom runtime
. - After creating new function:
- In Function code create a new script with
.R
extension, for examplemy_script.R
. - Paste function code inside, for example
hello_world <- function() { "Hello world!" }
- Change
Handler
to[script file name].[function name]
. - Delete
bootstrap
andhello.sh
files created by AWS. We don't need them, because the R runtime provides its own version ofbootstrap
. - In Basic settings, change
Timeout
to 30 seconds or more. SinceR
is not a speed demon, the default value (3 seconds) is too low.
- In Function code create a new script with
- Add R layer:
- In Designer panel click Layers to open layers configuration for your Lambda. This is where we need to add layer containing the
R
runtime. - Go to
Currently available layers
below and choose the base layer in your region. Add a layer giving its ARN. For example, foreu-central-1
, the ARN isarn:aws:lambda:eu-central-1:599651763768:layer:basic-r:1
. - If your code needs any additonal packages, they should be added additional layers. Go to Creating custom layers below to learn how to create such layers. After you create a layers, simply add it by providing layer ARN. You can provide up to 5 layers. Lambda will look for required dependencies (e.g. packages) in them, in provided order. Always use R layer as the first one. Packages should be included as additional layers.
- In Designer panel click Layers to open layers configuration for your Lambda. This is where we need to add layer containing the
- That's it! Now you can save and test your function. Remember to provide proper input data in JSON format - in our example it should be empty.
In your Lambda you can use published pre-built layers.
layer arn | region | content |
---|---|---|
arn:aws:lambda:eu-central-1:599651763768:layer:basic-r:1 | eu-central-1 | R 3.5.1 |
arn:aws:lambda:eu-central-1:599651763768:layer:dplyr | eu-central-1 | dplyr (with dependencies) for R 3.5.1 |
A layer can be used only in the provided region! If there's no layer for your region, you can create one (for how to do that see Basic R layer below) or open an issue in this repo asking for one.
If you need packages that are not provided in the prebuilt layers, you neet to create additional layers containing them. See R packages Layer for instructions how to do that. You'll create an instance, install packages and extract them into a Lambda layer. We provide an AMI and scripts that make this straightforward.
If you need a different version of R than provided, you'll need to create a basic R
layer, follow the instruction provided in Basic R Layer section. Creating AMI (to use it later for R
packages layers) is described in Lambda AMI with R section.
To be able to use this workflow, you have to configure AWS services.
- Install/upgrade aws cli:
pip install awscli --upgrade --user
- Configure credentials:
aws configure
(provide: AWS Access Key ID, AWS Secret Access Key, Default region name) - You need a Key Pair to be able to connect to EC2 instances. If you do not have one, you can create it in the Amazon EC2 console or using
aws cli
:aws ec2 create-key-pair --key-name [key name] --query 'KeyMaterial' --output text >> [file name].pem
(Important: Key file name has to be the same as the key name!). You will have to provide the path to the private key as a script argument (-k
flag) in the next steps.
Note: the instance will use your default security group. Make sure that it is open for incoming traffic from your IP on port 22 so that the script can connect and install needed packages on the instance.
- Run
./setup_r_instance.py -k [path to private key]
. It will create an EC2 instance, install R and download the archiveR.zip
. Check./setup_r_instance.py --help
for options. You have to provide at least the path to the private key (-k
). Script by default terminates the instance. If you want to prevent it set-t=False
. - Run
./build_runtime.sh
script. It will create an archiveruntime.zip
with R runtime for AWS Lambda. - Create a new layer:
aws lambda publish-layer-version --layer-name [layer name] --zip-file fileb://runtime.zip
- Run
./setup_r_instance.py -k [path to private key] -a create_ami -n [new ami name]
. It will create EC2 instance, install R and create AMI. Check./setup_r_instance.py --help
for options. You have to provide at least three parameters:-k
path to the private key (e.g.~/.ssh/key.pem
);-a create_ami
action;-n
AMI name. Script by deafult terminates the instance. If you want to prevent it set-t=False
. Script will create AMI.
- Run
./r_package_layer.py -k [path to private key] -m [R Lambda AMI id] -p [packages to install]
. It will create an instance from AMI with R preinstalled, install required packages and download archivepackages.zip
. Check./r_package_layer.py --help
for options. You have to provide at least three parameters:-k
path to the private key (e.g.~/.ssh/key.pem
);-m
Lambda AMI with preinstalled R id;-p
packages to install (if more than one, pass in quotes e.g."glue, stringr"
). Script by default terminates the instance. If you want to prevent it set-t=False
. - Create a new layer:
aws lambda publish-layer-version --layer-name [layer name] --zip-file fileb://packages.zip
AMI name | AMI id | region | conent |
---|---|---|---|
r-lambda-ami_id | ami-0a1147e8e86aa6175 | eu-central-1 | R 3.5.1 |