hugo-lambda-function

AWS Lambda function to build a Hugo website. Inspired by Ryan Brown's hugo-lambda and Jeremy Olexa's hugo-lambda-function

Also see the blog article Dynamic GitHub Actions with AWS Lambda for detailed information about how to set up GitHub and Lambda.

Files

lambda_function.py - The actual function that gets ran

build-package.sh - Helper information to build the zip file

Ideology

The idea of this repo is to build a zip package that can be deployed to AWS Lambda. This is what happens when everything is in place:

A new commit is pushed to GitHub.
The GitHub webhook notifies Amazon SNS.
The Lambda function fires on the SNS event.
The Lambda function creates a 'pending' status for the commit.
It downloads the commit as zip file from GitHub. Then it uses Hugo to generate the static website and syncs it to S3.
After a successful deployment, Lambda sets the GitHub commit status to 'success'.

There is also an option for scheduled events to trigger the build instead of GitHub pushes.

Usage

Note: For the first website you need to follow all the steps below. For any additional website, only the webhook of the GitHub repo (step 3) and the new AWS S3 bucket (Step 9) must be created. And, if used, a new scheduled event.

Step 1: Create an SNS Topic

Go to the AWS SNS console.
Click "Create topic".
Fill in any topic name you like and create the topic.
Keep the topic ARN around. We'll use that later.

Step 2: Create an IAM User and a Role

IAM User

We need a user for GitHub to publish:

Go to the Amazon IAM console.
Click "Users" then "Create New Users".
Enter a name for the GitHub publisher user. Make sure "Generate an access key for each user" is activated.
Create the user.
Show the user security credentials and keep them around for later use.
Return to the main IAM console page.
Click "Users", then click the name of your newly created user to edit its properties.
Scroll down to "Permissions" and create a new "Inline Policy".
Select the "Custom Policy" radio button, then press "Select".
Type a name for your policy, then paste the following statements. They authorize publication to the SNS topic you created before. Here you'll use the topic ARN.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "sns:Publish"
      ],
      "Resource": [
        "<SNS topic ARN goes here>"
      ],
      "Effect": "Allow"
    }
  ]
}

This IAM user represents the GitHub publishing process. The policy ensures that this user is only able to publish to the topic we just made. We'll share this user’s credentials with GitHub in a later step. As a security best practice, you should create a unique user for each system that you provide access to, rather than sharing user credentials, and you should always scope access to the minimum set of resources required (in this case, the SNS topic).

IAM Role

We also need a role to grant permissions to the Lambda function we will create later.

Still in IAM, select "Roles".
Click "Create New Role" and give it a name.
Select the "AWS Lambda" service role.
Do not attach a policy, just click "Next Step", then "Create Role".
Open your newly created role.
Add an "Inline Policy" and select "Custom Policy".
Give it a name and paste the following policy as the policy document and save:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "lambda:InvokeFunction",
                "lambda:GetFunctionConfiguration",
                "lambda:GetFunction"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:*:*:*"
        },
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::*"
        },
        {
            "Effect": "Allow",
            "Action": "dynamodb:*",
            "Resource": "arn:aws:dynamodb:*:*:table/lambdaLocks"
        }
    ]
}

With this policy, your Lambda function will have access to AWS S3 and DynamoDB and also allow inspection of a Lambda function and logging.

Step 3: Set up the GitHub Webhook

Navigate to your GitHub repo.
Click on "Settings" in the sidebar.
Click on "Webhooks & Services".
Click the "Add service" dropdown, then click "AmazonSNS". Fill out the form (supplying the IAM user credentials you created in Step 2), then click "Add service". (Note that the label says "topic", but it requires the entire ARN, not just the topic name.)
Make sure that the GitHub user has write access to this repository.

Now GitHub actions will publish to your SNS topic.

Step 4: Build the Lambda Zip Package

Open build-package.sh and adjust the hugo version.
Run build-package.sh. This will download Hugo and s3cmd (for syncing with AWS S3) and bundle it up together with main.py.

Hint: It's easiest to run the build-package.sh script on a Linux machine. If you don't have one you could simple use an online IDE like Cloud9. Just upload the build-package.sh and main.py, open the build-package.sh script and run it.

Step 5: Create the Lambda Function

Open the AWS Lambda console.
Click on "Create a Lambda function".
Don't choose a blueprint and click on "Skip".
Edit function name.
Choose "Python 2.7" as runtime.
Use the "Upload a .ZIP file" version and select the previousely generated Zip package.
As "Role" select the role you created in Step 2.
Choose some higher value for "Memory", e.g. 1024.
The "Timeout" should be set to the maximum of 5 min.
Create the fuction.

Step 6: Add SNS Event Sources

Open your newly created Lambda function and go to "Event sources".
Select "Add an Event Source".
Select "SNS" and the SNS topic created in Step 1.

Step 7: Add GitHub Access Token

The Lambda function needs access to our GitHub repository to add commit statuses and commit messages.

Open your GitHub account's "Personal access tokens" settings: https://github.com/settings/tokens
Click on "Generate new token".
Give it a name.
Activate the "repo" scope.
Generate the token and copy it.
Go back to your Lambda function and open the "Configuration" tab.
In the "Description" field, add the following JSON including the token:

{"github_token": "Your GitHub Access Token goes here"}

Step 8: Create DynamoDB Table

To coordinate concurrently running Lambda functions, our function needs a simple table in DynamoDB for locking.

Open the AWS DynamoDB console.
Click on "Create table".
Enter "lambdaLocks" as table name (this name must be entered exactly like this).
Enter "id" as the primary key String.
Click "Create".

Step 9: Create the S3 Bucket

Opent the AWS S3 console.
Click "Create Bucket".
Choose a "Bucket Name".
Important: The bucket name must be the same as the GitHub repository name. And for S3 to work as a static website host the name must be the same as the domain.
The "Region" should preferably be the same as the location of the Lambda function (for better sync performance).

That's it. Now run some tests by commiting to your GitHub repository.

(Optional) Add Scheduled Event Source

If you want to run the Lambda function not just on a GitHub push but also schedule it periodically, you can add another "Event Source" as follows:

Open the AWS CloudWatch console.
Select "Events"
Click the "Create rule" button.
Select "Schedule" as "Event selector" and either choose a fixed rate or a cron expression.
As "Targets" select "Lambda function" and choose your function.
Expand "Configure input".
Select "Constant (JSON text)".
In the "Constant" field add the following JSON. Adjust "owner" and "repo". You can also change the "ref" if you want to use something else than the master branch:

{"event_type": "scheduled", "owner": "Your GitHub username goes here", "repo": "Your GitHub repo name goes here", "ref": "heads/master"}

Draft Bucket

The Lambda function has support for draft websites. The draft website lives under the draft. subdomain (www. is removed, everything else is used as is).

To enable draft websites we need to do the following:

Create a second Lambda function that matches the normal Lambda function descripbed above.
Add "draft":true to the JSON in the Lambda description (where the Github token is).
Go the SNS Topic and create a new subscription for the draft Lambda function.
Create a new S3 bucket (draft.example.com).
Add the following bucket policy:

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "Allow password.html for all requesters",
			"Effect": "Allow",
			"Principal": "*",
			"Action": "s3:GetObject",
			"Resource": "arn:aws:s3:::draft.example.com/password.html"
		},
		{
			"Sid": "Allow everything else if redirected to",
			"Effect": "Allow",
			"Principal": "*",
			"Action": "s3:GetObject",
			"Resource": "arn:aws:s3:::draft.example.com/*",
			"Condition": {
				"StringLike": {
					"aws:Referer": "*://draft.example.com/*"
				}
			}
		}
	]
}

This will only allow access to the website if the referer is our website. See also http://xingdig.com/blog/password-protecting-an-s3-website/

Set the "Error document" to password.html (instead of 404.html or similar).
Now, all we need is to provide a password.html file that prompts the user for a password. If it is correct, it redirects the user to the requested site. For an example of such an HTML file see password.html and password.txt.
Hugo has a variable .Site.BuildDrafts. With this variable we can detect if we're building the draft or the normal website. This enables us to add a warning to the user that he is on the draft website.

Future

It might be nice to package up the zip file generation in a CloudFormation Stack
It would be very nice to package up the entire function/SNS topic in a CloudFormation Stack as well

marcojakob / hugo-lambda-function