This project is an open source indexer for the game Path Of Exile. The goal is to build a scalable stash tab indexer that runs in AWS utilising hosted services where possible to give maximum scalability, minimum systems administration time and lower costs.
It should be easy for anyone to fork and run only specific services for their needs.
The app will be built primarily with the serverless framework and CloudFormation.
- Ingester function
- Translate function
- Elastic Search
- Cloudwatch
- Online function
- Private API
- Frontend
- Public API
- Public API signup frontend
Ingestion is done in three functions using SNS, SQS and Dynamo DB to link them together.
This is a lambda function that is called by a cloud watch event every minute it runs for a minute. While running, it calls the stash API passes the next id into DynamoDB along with caching it locally. It pushes only public stash's into an SNS "public stash" topic and pushes private stash player names to a "player active" SQS Queue. On start, it reads the next id from DynamoDB.
This function is run periodically it reads from the "player active" SQS queue in blocks and updates the Elastic search Cluster with players that are active. We separate this from the main translate function as although the translate function does it for public stashes it will run at a much higher rate, and we don't want to slow that down with empty stashes this also makes more efficient function time and therefore reduces cost.
This function is subscribed to the SNS "public stash" topic and is called once when an item is pushed to that topic. It takes a stash and transforms the item data into data that is searchable by elastic search and then passes it in. It also updates the "active player" collection in elastic search.
The elastic search cluster should store all items and active players in a searchable format. Each item will be its a document with normalised data inside an "items" collection. Each time a player is "active" a document will be created in the "active players" collection containing the player id and name. The "active players" collection will have a TTL on all documents, so they automatically expire, and we can just check for the existence of a player id to know if a player is online.
Although there are no plans, currently the API should be built so that a cache can sit in front of Elastic Search proving a warm cache for regular searches (chaos to ex, chaos to fuse, shavs, etc.).
There will be 2 API's a private, rate limited and CORS Allow-Origin limited version that is intended for use by our frontend and an API key required version that will be available to the public (via request) with much higher rate limits. There should be documentation and SDK's available (generated by AWS). The private API does not need to be versioned the public does.
This will be an HTML and JS frontend hosted in S3 that uses the limited rate API allowing anyone to search our data like similar tools. The front end should be asynchronous and not require reloading of the page. React, Bootstrap and SCSS will be used for frontend frameworks, and hexo will be used as a compiler.
All the services should log replication metrics to Cloudwatch.