mulyoved / secure-storage-exercise

Service that exposes two endpoints to save and retrieve values while storing them securely.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Description

Service that exposes SET and GET endpoints to save and retrieve values while storing them securely. All data at rest is securely encrypted with the key provided by the clients.

Implementation Notes

  • Service use AWS lambda functions for fault tolerant and scalability
  • Use AWS DynamoDb for storage, again out of the box fault tolerant and scalability
  • Deployment is done using serverless
  • Language is TypeScript
  • Unit tetsing using jest
  • e2e testing using jest, as this is API product jest + axiom for rest API work well
  • Encryption is done using node build-in crypto node-crypto-gcm with a small wrapper crypto-gcm, GCM is rather popular authenticated encryption algorithm designed to provide both data authenticity (integrity) and confidentiality.

Usage

Encryption use node-crypto-gcm which need a fixed size key, to generate a key:

node

const key = crypto.randomBytes(32).toString("hex");

SET

curl \
-H "Content-Type: application/json" \
-X POST \
-d '{"id":"id-1","encryption_key":"920ec086a7f4cb35324444d655654b117a6f7f77519c123f4b5d0e4576bab524","value":"sample-string"}' \
https://iwiexmyvl1.execute-api.eu-west-2.amazonaws.com/dev/set

GET

curl \
-H "Content-Type: application/json" \
-X POST \
-d '{"id":"id-1","encryption_key":"920ec086a7f4cb35324444d655654b117a6f7f77519c123f4b5d0e4576bab524"}' \
https://iwiexmyvl1.execute-api.eu-west-2.amazonaws.com/dev/get

See ./e2e-tests/call-get-set.test.ts for more usage example

Install

Give AWS Credentials

yarn test
yarn deploy
yarn e2e-test

Limitations

  • GET with a wildcard support only wildcard in the end of the id
  • GET with wildcards use DynamoDB scan which scan the entire table, in most use cases this should be efficient enough, depending on the usage pattern maybe need to consider using SQL database or organize DynamoDB keys differently

Todo/To Improve

  • Documentation, browser examples if needed
  • Implement delete API, this will help to write more complete testing so can clean the database before running the test, right now after initial run values are just updated and not created
  • Set a fixed URL so in case the service need to be removed and deploy again or deploy in a different region end point url stay the same
  • Connect to centralized logging like splunk
  • Connect to centralized errors reports like sentry
  • AWS API pagination for large get results, currently will return error if result size exceed AWS limits

Hints

Project created using

serverless create --template aws-nodejs-typescript --path secure-storage-exercise
cd secure-storage-exercise
yarn install

Query DynamoDB from the terminal

find 1 item

aws dynamodb scan \
     --region eu-west-2 \
     --profile secure-storage-exercise \
     --table-name aws-nodejs-typescript-dev \
     --filter-expression "id = :search" \
     --expression-attribute-values '{":search":{"S":"s1"}}'

scan

aws dynamodb scan \
     --region eu-west-2 \
     --profile secure-storage-exercise \
     --table-name aws-nodejs-typescript-dev \
     --filter-expression "begins_with(id, :search)" \
     --expression-attribute-values '{":search":{"S":"s"}}'

get serverless functions logs

sls logs -f set
sls logs -f get

browser the records on AWS

aws-nodejs-typescript-dev

About

Service that exposes two endpoints to save and retrieve values while storing them securely.


Languages

Language:TypeScript 95.3%Language:JavaScript 4.7%