EListMan - Email List Manager

Mailing list system providing address validation and unsubscribe URIs.

Source: https://github.com/mbland/elistman

(Try force reloading the page to get the latest badges if this is a return visit. The browser cache may hide the latest results.)

Only serves one list at a time as defined by deployment parameters.

Implemented in Go using the following Amazon Web Services:

Uses CloudFormation and the AWS Serverless Application Model (SAM) for deploying the Lambda function, binding to the API Gateway, managing permissions, and other configuration parameters.

Originally implemented to support https://mike-bland.com/subscribe/.

The very earliest stages of the implementation were based on hints from victoriadrake/simple-subscribe, but all the code is original.

Open Source License

This software is made available as Open Source software under the Mozilla Public License 2.0. For the text of the license, see the LICENSE.txt file.

Setup

Install tools

Run the bin/check-tools.sh script to check that the required tools are installed.

This script will try to install some missing tools itself. If any are missing, the script will provide a link to installation instructions.
Note: The script does not check for the presence of make, as it comes in many flavors and aliases and already ships with many operating systems. The notable exception is Microsoft Windows.

However, if the winget command is available, you can install the GnuWin32.Make package:
```
winget install -e --id GnuWin32.Make
```
Then:
- Press Win-R and enter systempropertiesadvanced to open the System Properties > Advanced pane.
- Click the Environment Variables... button.
- Select Path in either the User variables or System variables pane, then click the corresponding Edit... button.
- Click the New button, then click the Browse... button.
- Navigate to This PC > Local Disk (C:) > Program Files (x86) > GnuWin32 > bin.
- Click the OK button, then keep clicking the OK button until all of the System Properties panes are closed.
Make should then be available as either make or make.exe.

Configure the AWS CLI

Configure your credentials for a region from the Email Receiving Endpoints section of Amazon Simple Email Service endpoints and quotas.

Follow the guidance on the AWS Command Line Interface: Quick Setup page if necessary.

Configure AWS Simple Email Service (SES)

Set up SES in the region selected in the above step. Make sure to enable DKIM and create a verified domain identity per Verifying your domain for Amazon SES email receiving.

Create a Receipt Rule Set and set it as Active. EListMan will add a Receipt Rule for an unsubscribe email address to this Receipt Rule Set.

Create a Receipt Rule to receive Email notifications for the postmaster and abuse accounts, along with any other accounts that you'd like.

You can add these recipient conditions manually, which would require creating a Simple Notification Service (SNS) topic manually as well.
Alternatively, consider using mbland/ses-forwarder to automate configuring the Receipt Rule Set with an appropriate Receipt Rule.

When you're ready for the system to go live, publish an MX record for Amazon SES email receiving.

It's also advisable to configure your account-level suppression list to automatically add addresses resulting in bounces and complaints.

Assuming you have your AWS CLI environment set up correctly, this should confirm that SES is properly configured (with your own identity listed, of course):

$ aws sesv2 list-email-identities

{
    "EmailIdentities": [
        {
            "IdentityType": "DOMAIN",
            "IdentityName": "mike-bland.com",
            "SendingEnabled": true,
            "VerificationStatus": "SUCCESS"
        }
    ]
}

You can also view other account attributes, such as account suppression list status, send quotas, and send rates, via:

$ aws sesv2 get-account

{
   ...
    "ProductionAccessEnabled": true,
    "SendQuota": {
        "Max24HourSend": ...,
        "MaxSendRate": ...,
        "SentLast24Hours": ...
    },
    "SendingEnabled": true,
    "SuppressionAttributes": {
        "SuppressedReasons": [
            "BOUNCE",
            "COMPLAINT"
        ]
    },
    "Details": {
        "MailType": "MARKETING",
        "WebsiteURL": "https://mike-bland.com/",
        "ContactLanguage": "EN",
        "UseCaseDescription": "This is for publishing blog posts to email subscribers.",
        "AdditionalContactEmailAddresses": [
            "mbland@acm.org"
        ],
        ...
    }
}

Configure AWS API Gateway

Set up a custom domain name in API Gateway in the region selected in the above step. Create a SSL certificate in Certificate Manager for it as well.

Setting up custom domain names for HTTP APIs

If done correctly, the following command should produce output resembling the example:

$ aws apigatewayv2 get-domain-names

{
    "Items": [
        {
            "ApiMappingSelectionExpression": "$request.basepath",
            "DomainName": "api.mike-bland.com",
            "DomainNameConfigurations": [
                {
                    "ApiGatewayDomainName": "<...>",
                    "CertificateArn": "<...>",
                    "DomainNameStatus": "AVAILABLE",
                    "EndpointType": "REGIONAL",
                    "HostedZoneId": "<...>",
                    "SecurityPolicy": "TLS_1_2"
                }
            ]
        }
    ]
}

Configure API Gateway to write CloudWatch logs

Next, set up an IAM role to allow the API to write CloudWatch logs. You only need to execute the steps in the Create an IAM role for logging to CloudWatch section. One possible name for the new IAM role would be ApiGatewayCloudWatchLogging.

The last step from the above instructions is to

$ ARN="arn:aws:iam::...:role/ApiGatewayCloudWatchLogging"
$ aws apigateway update-account --patch-operations \
    op='replace',path='/cloudwatchRoleArn',value='$ARN'

If successful, the output should resemble the following, where <ARN> is the value of $ARN from above:

{
    "cloudwatchRoleArn": "<ARN>",
    "throttleSettings": {
        "burstLimit": ...,
        "rateLimit": ...
    },
    "features": []
}

Note: Per the documentation for the AWS::ApiGateway::Account CloudFormation entity, "you should only have one AWS::ApiGateway::Account resource per region per account." This is why it's not included in template.yml in favor of the one-time-per-account instructions above.

However, if you want to try using SAM/CloudFormation to manage it, see:

Stack Overflow: Configuring logging of AWS API Gateway - Using a SAM template

Run tests

To make sure the local environment is in good shape, and your AWS services are properly configured, run the main test suite via make test. (Note that the example output below is slightly edited for clarity.)

$ make test

go vet -tags=all_tests ./...
go run honnef.co/go/tools/cmd/staticcheck -tags=all_tests ./...
go build -tags=all_tests ./...

go test -tags=small_tests ./...
ok      github.com/mbland/elistman/agent        0.110s
ok      github.com/mbland/elistman/db   0.392s
ok      github.com/mbland/elistman/email        0.187s
ok      github.com/mbland/elistman/handler      0.260s
ok      github.com/mbland/elistman/ops  0.461s
ok      github.com/mbland/elistman/types        0.523s

go test -tags=medium_tests -count=1 ./...
ok      github.com/mbland/elistman/db   2.970s
ok      github.com/mbland/elistman/email        1.150s

go test -tags=contract_tests -count=1 ./db -args -awsdb
ok      github.com/mbland/elistman/db   44.264s

If you're using Visual Studio Code, you can run all but the last test via the Test: Run All Tests command (testing.runAll). The default keyboard shortcut is ⌘; A.

The project VS Code configuration is in .vscode/settings.json.
For other helpful testing-related keyboard shortcuts, press ⌘K ⌘S, then search for testing.

Test sizes

The tests are divided into suites of varying test sizes, described below, using Go build constraints (a.k.a. "build tags"). These constraints are specified on the first line of every test file:

$ head -n1 */*_test.go

==> agent/agent_test.go <==
//go:build small_tests || all_tests

# ...snip...

# See "Test coverage" section below for an explanation of the
# dynamodb_contract_test build constraints.
==> db/dynamodb_contract_test.go <==
//go:build ((medium_tests || contract_tests) && !no_coverage_tests) || coverage_tests || all_tests

# ...snip...

==> email/mailer_contract_test.go <==
//go:build medium_tests || contract_tests || all_tests

# ...etc...

Small tests

The small_tests all run locally, with no external dependencies. These tests cover all fine details and error conditions.

Medium/contract tests

Each of the medium_tests exercises integration with specific dependencies. Most of these dependencies are actual, live AWS services that require a network connection.

These tests are designed to set up required state and clean up any side effects. Other than ensuring the network is available, and the required resources are running and accessible, no external intervention is necessary.

medium_tests validate high level use cases and fundamental assumptions, not exhaustive details and error conditions. That's what the small_tests are for, resulting in fewer, less complicated, faster, and more stable medium_tests.

Each of the contract_tests are also medium_tests. In fact, it's arguable that these tags are redundant, but I want the reader to contemplate both concepts and their equivalence.

The medium/contract tests in db/dynamodb_contract_test.go run against:

a local Docker container running the amazon/dynamodb-local image when run without the -awsdb flag
- e.g. When run via go test -tags=medium_tests -count=1 ./..., in VS Code via ⌘; A, or in CI via -tags=coverage_tests, described below.
the actual DynamoDB for your AWS account when run with the -awsdb flag
- e.g. When run via go test -tags=contract_tests -count=1 ./db -args -awsdb

Note: -count=1 is the Go idiom to ensure tests are run with caching disabled, per go help testflag.

Large tests and smoke tests

There are no end-to-end large_tests yet, outside of bin/smoke-tests.sh. The smoke tests are described below, as are the plans for adding end-to-end tests one day.

Test coverage

To check code coverage, you can run:

$ make coverage

go test -covermode=count -coverprofile=coverage.out \
          -tags=small_tests,coverage_tests ./...

ok  github.com/mbland/elistman/agent    0.351s  coverage: 100.0% of statements
ok  github.com/mbland/elistman/db       3.214s  coverage: 100.0% of statements
ok  github.com/mbland/elistman/email    0.539s  coverage: 100.0% of statements
ok  github.com/mbland/elistman/handler  0.613s  coverage: 100.0% of statements
ok  github.com/mbland/elistman/ops      0.457s  coverage: 100.0% of statements
ok  github.com/mbland/elistman/types    0.675s  coverage: 100.0% of statements

go tool cover -html=coverage.out
[ ...opens default browser with HTML coverage results... ]

You can also check coverage in VS Code by searching for the Go: Toggle Test Coverage in Current Package command via Show All Commands (⇧⌘P).

Note that db/dynamodb_contract_test.go is the one and only medium_test that we need for test coverage purposes. It contains the coverage_tests build constraint, enabling the CI pipeline to collect its coverage data without running other medium_tests.

Build the `elistman` CLI

Build the elistman command line interface program in the root directory via:

go build

Run the command and check the output to see if it was successful:

$ ./elistman -h

Mailing list system providing address validation and unsubscribe URIs

See the https://github.com/mbland/elistman README for details.

To create a table:
  elistman create-subscribers-table TABLE_NAME

To see an example of the message input JSON structure:
  elistman preview --help

To preview a raw message before sending, where `generate-email` is any
program that creates message input JSON:
  generate-email | elistman preview

To send an email to the list, given the STACK_NAME of the EListMan instance:
  generate-email | elistman send -s STACK_NAME

Usage:
  elistman [command]

Available Commands:
  [...commands snipped...]

Flags:
  -h, --help      help for elistman
  -v, --version   version for elistman

Use "elistman [command] --help" for more information about a command.

Create the DynamoDB table

Run elistman create-subscribers-table <TABLE_NAME> to create the DynamoDB table, replacing <TABLE_NAME> with a table name of your choice. Then run aws dynamodb list-tables to confirm that the new table is present.

Create the configuration file

Create the deploy.env configuration file in the root directory containing the following environment variables (replacing each value with your own as appropriate):

# This will be the name of the CloudFormation stack. The `--stack-name` flag of
# `elistman` CLI commands will require this value.
STACK_NAME="mike-blands-blog-example"

# This is the domain name configured in the "Configure AWS API Gateway" step.
API_DOMAIN_NAME="api.mike-bland.com"

# This will be the first component of the EListMan API endpoints after the
# hostname, e.g., api.mike-bland.com/email/subscribe.
API_MAPPING_KEY="email"

# The domain from which emails will be sent. This should likely match the
# website on which the subscription form appears.
EMAIL_DOMAIN_NAME="mike-bland.com"

# The proper name of the website from which emails will appear to be sent. It
# need not match to the site's <title> exactly, but should clearly describe what
# subscribers expect.
EMAIL_SITE_TITLE="Mike Bland's blog"

# The proper name of the email sender. It need not match EMAIL_SITE_TITLE, but
# again, should not surprise subscribers.
SENDER_NAME="Mike Bland's blog"

# The username of the email sender. The full address will be of the form:
# SENDER_USER_NAME@EMAIL_DOMAIN_NAME, e.g., posts@mike-bland.com.
SENDER_USER_NAME="posts"

# The username of the unsubscribe email recipient. The full address will be of
# the form: UNSUBSCRIBE_USER_NAME@EMAIL_DOMAIN_NAME, e.g.,
# unsubscribe@mike-bland.com.
UNSUBSCRIBE_USER_NAME="unsubscribe"

# The path to the unsubscribe form relative to EMAIL_DOMAIN_NAME. See the
# "Understand the {{UnsubscribeUrl}} template" and "Publish your HTML
# unsubscribe form" sections below.
UNSUBSCRIBE_FORM_PATH="/unsubscribe"

# The name of the Receipt Rule Set created in the "Configure AWS Simple Email
# Service (SES)" step.
RECEIPT_RULE_SET_NAME="mike-bland.com"

# The name of the DynamoDB table created via `elistman create-subscribers-table`
# in the "Create the DynamoDB table" step.
SUBSCRIBERS_TABLE_NAME="<TABLE_NAME>"

# Percentage of daily quota to consume before self-limiting bulk sends via
# `elistman send -s STACK_NAME`.  See the "Send rate throttling and send quota
# capacity limiting" step for a detailed description. (Does not apply when
# running `elistman send` with specific subscriber addresses specified on the
# command line.)
MAX_BULK_SEND_CAPACITY="0.8"

# EListMan will redirect API requests to the following URLs according to the 
# "Algorithms" described below.
INVALID_REQUEST_PATH="/subscribe/malformed.html"
ALREADY_SUBSCRIBED_PATH="/subscribe/already-subscribed.html"
VERIFY_LINK_SENT_PATH="/subscribe/confirm.html"
SUBSCRIBED_PATH="/subscribe/hello.html"
NOT_SUBSCRIBED_PATH="/unsubscribe/not-subscribed.html"
UNSUBSCRIBED_PATH="/unsubscribe/goodbye.html"

Run smoke tests locally

bin/smoke-test.sh invokes curl to send HTTP requests to the running Lambda, all of which expect an error response without any side effects (save for logging).

To check that your configuration works locally, you'll need two separate terminal windows to run bin/smoke-test.sh. In the first, run:

$ make run-local

[ ...validates template.yml, builds lambda, etc... ]
bin/sam-with-env.sh deploy.env local start-api --port 8080
[ ...more output... ]
You can now browse to the above endpoints to invoke your functions....
2023-05-29 16:08:04 WARNING: This is a development server....
 * Running on http://127.0.0.1:8080
2023-05-29 16:08:04 Press CTRL+C to quit

In the next terminal, run:

$ ./bin/smoke-test ./deploy.env --local

INFO: SUITE: Not found (403 locally, 404 in prod)
INFO: TEST: 1 — invalid endpoint not found
Expect 403 from: POST http://127.0.0.1:8080/foobar/mbland%40acm.org

curl -isS -X POST http://127.0.0.1:8080/foobar/mbland%40acm.org

HTTP/1.1 403 FORBIDDEN
Server: Werkzeug/2.3.4 Python/3.8.16
Date: Mon, 29 May 2023 20:19:57 GMT
Content-Type: application/json
Content-Length: 43
Connection: close

{"message":"Missing Authentication Token"}

PASSED: 1 — invalid endpoint not found:
    status: 403

INFO: TEST: 2 — /subscribe with trailing component not found
Expect 403 from: POST http://127.0.0.1:8080/subscribe/foobar

[ ...more test output/results... ]

PASSED: 6 — invalid UID for /unsubscribe:
    status: 400

PASSED: All 6 smoke tests passed!

Then enter CTRL-C in the first window to stop the local SAM Lambda server.

Understand the danger of spam bots and the need for a CAPTCHA

Before deploying to production, we need to talk about spam.

The EListMan system tries to validate email addresses through its own up front analysis and by sending validation links to subscribers. However, opportunistic spam bots can still—and will—submit many valid email addresses without either the knowledge or consent of the actual owner.

Fortunately, the validation link mechanism prevents most bogus subscriptions, and DynamoDB's Time To Live feature cleans them from the database automatically. A bounce or complaint also notifies the EListMan Lambda to remove the address and add it to the account-level suppression list. The suppression list ensures the system won't send to that address again, even if someone attempts to resubmit it.

This means most bogus subscriptions will not pollute the verified subscriber list, and such recipients will not receive further emails. However, generating these bogus subscriptions still consumes resources, and their verification emails can yield bounces and complaints that will harm your SES reputation metrics.

Having learned this the hard (naïve) way, I recommend using a CAPTCHA to prevent spam bot abuse:

When I first published my EListMan subscription form, my instance received dozens of bogus subscription requests a day—before I'd even announced it on my blog. (The form had been available before, but used a different subscription system.)
After deploying a CAPTCHA, the number of bogus subscriptions dropped to zero. (I hope I hadn't inadvertently been allowing subscription verification spam all those years before....)

Decide whether or not to use the AWS WAF CAPTCHA

EListMan's CloudFormation/SAM template configures an AWS Web Application Firewall (WAF) CAPTCHA, creating one Web ACL and one Rule associated with it. If you choose to use it, note that it does incur additional charges. See AWS WAF Pricing for details.

If you choose not to use it, comment out or delete the WebAcl and WebAclAssociation resources in template.yml.

Generate an AWS Web Application Firewall CAPTCHA API KEY (optional)

To use EListMan's Web ACL configuration, you'll need to generate an API key for the CAPTCHA API. Include whichever domain will serve the submission form in the list of domains used to generate the API key.

The default EListMan configuration expects this domain to be the same as EMAIL_DOMAIN_NAME, described above. If you use a different domain, set WebAcl > Properties > TokenDomains in template.yml appropriately.

Deployment

Deploy to AWS

If the smoke tests pass, deploy the EListMan system via:

make deploy

Once the deployment is running, run the smoke tests without the --local flag to ensure your instance is reachable:

./bin/smoke-tests.sh ./deploy.env

Publish your HTML subscription form

You'll need to publish a subscription <form> similar to the following, substituting API_DOMAIN_NAME with the custom domain name from the Configure AWS API Gateway step:

<!-- subscribe.html -->

<form method="post" action="https://API_DOMAIN_NAME/email/subscribe">
  <input name="email" type="email"
   placeholder="Please enter your email address."/>
  <button type="submit">Subscribe</button>
</form>

However, as mentioned above, spam bots are a thing, even for the humblest of sites publicly sporting a <form> element.

Generate your email submission form programmatically (optional)

You may gain extra protection from spam bots by generating the subscription form using JavaScript instead of embedding a <form> element directly in your HTML.

In other words, instead of embedding the <form> directly in your subscription page as shown above, use something like this:

<!-- subscribe.html -->

<div class="subscribe-form">
  <button>Show subscribe form</button>
</div>

// subscribe.js

"use strict";

document.addEventListener("DOMContentLoaded", () => {
  var container = document.querySelector(".subscribe-form")

  var showForm = () => {
    var f = document.createElement("form")
    // The following should generate the value for API_DOMAIN_NAME.
    var api_domain_name = ["my", "api", "com"].join(".")
    f.action = ["https:", "", api_domain_name, "email", "subscribe"].join("/")
    f.method = "post"

    var i = document.createElement("input")
    i.name = "email"
    i.type = "email"
    i.placeholder = "Please enter your email address."
    f.appendChild(i)

    var s = document.createElement("button")
    s.type = "submit"
    s.appendChild(document.createTextNode("Subscribe"))
    f.appendChild(s)

    container.parentNode.replaceChild(f, container)
  }

  container.querySelector("button").addEventListener('click', showForm)
})

Integrate the CAPTCHA into your subscription form (optional)

Of course, the ultimate protection would be to use an AWS WAF CAPTCHA to protect the /subscribe API endpoint.

Using the same HTML from above, the code below will render the AWS WAF CAPTCHA puzzle when the subscriber clicks the button. When they solve the puzzle, it will then reveal the submission form.

Remember to substitute YOUR_AWS_WAF_CAPTCHA_API_KEY with your own API key:

// subscribe.js

"use strict";

document.addEventListener("DOMContentLoaded", () => {
  var container = document.querySelector(".subscribe-form")

  var showForm = () => {
    // Same implementation as above
  }

  container.querySelector("button").addEventListener('click', () => {
    AwsWafCaptcha.renderCaptcha(container, {
      apiKey: YOUR_AWS_WAF_CAPTCHA_API_KEY,
      onSuccess: showForm,
      dynamicWidth: true,
      skipTitle: true
    });
  })
})

Understand the `{{UnsubscribeUrl}}` template

The {{UnsubscribeUrl}} generated for each recipient will be of the format:

https://${EMAIL_DOMAIN_NAME}/${UNSUBSCRIBE_FORM_PATH}?email=<email>&uid=<uid>

where:

<email> is the recipient's query encoded email address
<uid> is the recipient's query encoded user ID generated by the system

For example:

https://mike-bland.com/unsubscribe?email=foo%40bar.com&uid=00000000-1111-2222-3333-444444444444

For more background on URI encoding:

Publish your HTML unsubscribe form

You'll need to publish a page with an unsubscribe <form> at the location https://${EMAIL_DOMAIN_NAME}/${UNSUBSCRIBE_FORM_PATH}. This form will allow the user to confirm they really intend to unsubscribe. Use JavaScript to fill out the <form> using the email and uid URL query parameters provided when the user clicks on their unique {{UnsubscribeUrl}}.

For example, following the same pattern as the Generate your email submission form programmatically (optional) section above:

<!-- unsubscribe.html -->

<h2>Unsubscribe</h2>

<div class="unsubscribe"><p>If you would like to stop receiving email
updates, please click the "Unsubscribe" link at the bottom of one of the
emails.</p></div>

// unsubscribe.js

"use strict";

document.addEventListener("DOMContentLoaded", () => {
  var params = new URLSearchParams(window.location.search)

  if (!params.has("email") || !params.has("uid")) {
    return
  }

  var instructions = document.querySelector(".unsubscribe p")
  instructions.innerHTML = instructions.innerHTML.replace(/[\n ]+/g, " ")
    .replace("link at the bottom of one of the emails", "button below")

  var f = document.createElement("form")
  // The following should generate the value for API_DOMAIN_NAME.
  var api_domain_name = ["my", "api", "com"].join(".")
  f.action = [
    "https:", "", api_domain_name, "email", "unsubscribe",
    encodeURI(params.get("email")), encodeURI(params.get("uid")),
  ].join("/")
  f.method = "post"

  var s = document.createElement("button")
  s.type = "submit"
  s.appendChild(document.createTextNode("Unsubscribe"))
  f.appendChild(s)
  instructions.parentNode.appendChild(f)
})

Subscribe and send a test email to yourself

After deploying EListMan and publishing your subscription form, use the form to subscribe to the list. Then you can run the following command to send a test email to yourself (replacing STACK_NAME and MY_EMAIL_ADDRESS as appropriate):

$ ./bin/generate-test-message.sh ./deploy.env |
    ./elistman send -s STACK_NAME MY_EMAIL_ADDRESS

Send a production email to the list

Run ./elistman send -h to see an example email:

$ ./elistman send -h 

Reads a JSON object from standard input describing a message:

  {
    "From": "Foo Bar <foobar@example.com>",
    "Subject": "Test object",
    "TextBody": "Hello, World!",
    "TextFooter": "Unsubscribe: {{UnsubscribeUrl}}",
    "HtmlBody": "<!DOCTYPE html><html><head></head><body>Hello, World!<br/>",
    "HtmlFooter": "<a href='{{UnsubscribeUrl}}'>Unsubscribe</a></body></html>"
  }

You will need to generate a similar JSON object to feed into the standard input of ./elistman send:

From, Subject, TextBody, and TextFooter are required.
If HtmlBody is present, HtmlFooter must also be present.
TextFooter, and HtmlFooter if present, must contain one and only one instance of the {{UnsubscribeUrl}} template. The EListMan Lambda will replace this template with the unsubscribe URL unique to each subscriber.
TextFooter and HtmlFooter will appear on a new line immediately after TextBody and HtmlBody, respectively.

Provided you have a program to generate the JSON object above called generate-email, you can then send an email to the list via:

generate-email | ./elistman send -s STACK_NAME

Development

The Makefile is very short and readable. Use it to run common tasks, or learn common commands from it to use as you please.

For guidance on writing Go developer documentation, see Go Doc Comments.

There are two ways to view the developer documentation in a web browser.

Viewing documentation with `godoc`

godoc is reportedly deprecated, but still works well. See:

# Install the godoc tool.
$ go install -v golang.org/x/tools/cmd/godoc@latest

# Serve documentation from the local directory at http://localhost:6060.
$ godoc -http=:6060

You can then view the EListMan docs locally at:

http://localhost:6060/pkg/github.com/mbland/elistman/

One of the nice features of godoc is that you can view documentation for unexported symbols by adding ?m=all to the URL. For example:

http://localhost:6060/pkg/github.com/mbland/elistman/?m=all

Viewing documentation with `pkgsite`

pkgsite is the newer development documentation publishing system.

# Install the pkgsite tool.
$ go install golang.org/x/pkgsite/cmd/pkgsite@latest

# Serve documentation from the local directory at http://localhost:8080.
$ pkgsite

You can then view the EListMan docs locally at:

http://localhost:8080/github.com/mbland/elistman

Note that, unlike godoc, pkgsite doesn't provide an option to serve documentation for unexported symbols.

URI Schema

https://<api_hostname>/<route_key>/<operation>
mailto:<unsubscribe_user_name>@<email_domain_name>?subject=<email>%20<uid>

Where:

<api_hostname>: Hostname for the API Gateway instance
<route_key>: Route key for the API Gateway
<operation>: Endpoint for the list management operation:
- /subscribe
- /verify/<email>/<uid>
- /unsubscribe/<email>/<uid>
<email>: Subscriber's email address
<uid>: Identifier assigned to the subscriber by the system
<unsubscribe_user_name>: The username receiving unsubscribe emails, typically unsubscribe, set via UNSUBSCRIBE_USER_NAME.
<email_domain_name>: Hostname serving as an SES verified identity for sending and receiving email, set via EMAIL_DOMAIN_NAME

Algorithms

Unless otherwise noted, all responses will be HTTP 303 See Other, with the target page specified in the Location HTTP header.

The one exception will be unsubscribe requests from mail clients using the List-Unsubscribe and List-Unsubscribe-Post email headers.

Generating a new subscriber verification link

An HTTP request from the API Gateway comes in, containing the email address of a potential subscriber.
Validate the email address.
1. Parse the name as closely as possible to RFC 5322 Section 3.2.3 via net/mail.ParseAddress.
2. Reject any common aliases, like "no-reply" or "postmaster."
3. Check the MX records of the host by:
  1. Doing a reverse lookup on each mail host's IP addresses.
  2. Looking up the IP addresses of the hosts returned by the reverse lookup.
  3. Confirming at least one reverse lookup host IP address matches a mail host IP address.
4. If it fails validation, return the INVALID_REQUEST_PATH.
Look for an existing DynamoDB record for the email address.
1. If it exists, return the VERIFY_LINK_SENT_PATH for Pending subscribers and ALREADY_SUBSCRIBED_PATH for Verified subscribers.
Generate a UID.
Write a DynamoDB record containing the email address, the UID, a timestamp, and with SubscriberStatus set to Pending.
Generate a verification link using the email address and UID.
Send the verification link to the email address.
1. If the mail bounces or fails to send, return the INVALID_REQUEST_PATH.
Return the VERIFY_LINK_SENT_PATH.

Responding to a subscriber verification link

An HTTP request from the API Gateway comes in, containing a subscriber's email address and UID.
Check whether there is a record for the email address in DynamoDB.
1. If not, return the NOT_SUBSCRIBED_PATH.
Check whether the UID matches that from the DynamoDB record.
1. If not, return the NOT_SUBSCRIBED_PATH.
If the subscriber's status is Verified, return the ALREADY_SUBSCRIBED_PATH.
Set the SubscriberStatus of the record to Verified.
Return the SUBSCRIBED_PATH.

Responding to an unsubscribe request

Either an HTTP Request from the API Gateway or a mailto: event from SES comes in, containing a subscriber's email address and UID.
Check whether there is a record for the email address in DynamoDB.
1. If not, return the NOT_SUBSCRIBED_PATH.
Check whether the UID matches that from the DynamoDB record.
1. If not, return the NOT_SUBSCRIBED_PATH.
Delete the DynamoDB record for the email address.
If the request was an HTTP Request:
1. If it uses the POST method, and the data contains List-Unsubscribe=One-Click, return HTTP 204 No Content.
2. Otherwise return the UNSUBSCRIBED_PATH page.

Expiring unused subscriber verification links

DynamoDB's Time To Live feature will eventually remove expired pending subscriber records after 24 hours.

Send rate throttling and send quota capacity limiting

EListMan calls the SES v2 getAccount API method once a minute to monitor sending quotas and to adjust the send rate. Every individual message sent, including both subscription verification messages and messages sent to the list, will honor the current send rate.

The MAX_BULK_SEND_CAPACITY parameter specifies what percentage of the 24 hour send quota may be used for sending emails to the list. This helps avoid exceeding the daily quota before a message has been sent to all subscribers. elistman send will fail, before sending an email, if the percentage of the daily send quota specified by MAX_BULK_SEND_CAPACITY has already been consumed.

The default is to use 80% of the available daily send quota for list messages, expressed as MAX_BULK_SEND_CAPACITY="0.8". The remaining 20% acts as a buffer. For example, for a quota of 50,000 messages, up to 40,000 (50,000 * 0.8) are available to elistman send within a 24 hour period.

Note that this mechanism tries to prevent the operator from accidentally exceeding the 24 hour quota, but it's not foolproof. The operator is ultimately responsible for ensuring that elistman send won't exceed the quota if MAX_BULK_SEND_CAPACITY hasn't yet been reached, or for tuning it accordingly.

Building on the previous example, if there are 17,000 subscribers:

The first elistman send consumes 17,000 of the quota.
The second elistman send consumes the next 17,000 of the quota, for a total of 34,000.
The third elistman send will proceed, since 34,000 is less than the 40,000 calculated by MAX_BULK_SEND_CAPACITY="0.8". However, it will consume 17,000 more of the quota, for 51,000 total, exceeding the 50,000 quota.

Subscription verification messages are not affected by the MAX_BULK_SEND_CAPACITY constraint. The buffer defined by MAX_BULK_SEND_CAPACITY can ensure that there is always daily send quota available for such messages.

Unimplemented/possible future features

Automated End-to-End tests

This is something I really want to pull off, but without blocking the first release.

Here is what I anticipate the implementation will involve (beyond some of the existing cases in bin/smoke-test.sh):

Test Setup

Create a new random test username.
Create a S3 bucket for the emails received by the random test user.
Create a receipt rule for the active rule set on the domain to send emails to the new random test username to the S3 bucket.
- Add the random test username to the recipient conditions.
- Add an action to write to the S3 bucket
Bring up a CloudFormation/Serverless Application Model stack defining these resources.

Execution

For each permutation described below:

Send a request to subscribe the valid random username.
- Note: The /subscribe endpoint is CAPTCHA-protected on the dev and prod instances. We may need to bring up an alternate API Gateway instance for the test without CAPTCHA protection, or call ProdAgent.Subscribe() through another method.
Read the S3 bucket to get the validation URL.
Request the validation URL.
Form an unsubscribe request (either URL or mailto) from the validation URL.

Permutations

Subscribe via urlencoded params, unsubscribe via urlencoded params
Subscribe via form-data, unsubscribe via form-data params
Subscribe via urlencoded params, unsubscribe via email
Try to resubscribe to expect an ALREADY_SUBSCRIBED_PATH response.
Modify the UID in the verification URL to expect an INVALID_REQUEST_PATH response.

Teardown

Tear down the stack, which will:
- Tear down the receipt rules
- Tear down the test bucket