mattiasw2 / cloudquery

Turn any website to serverless API (support SPA!)

CloudQuery

Turn any website to serverless API (with SPA support!)

Try online: https://2uvlp0y13b.execute-api.us-west-2.amazonaws.com/staging/

API

Example url: https://2uvlp0y13b.execute-api.us-west-2.amazonaws.com/staging/query?url=https://news.ycombinator.com/best&selectors=*:nth-child(1) > * > *:nth-child(2) > *:nth-child(2) > :nth-child(1),:nth-child(3) *:nth-child(1) > *:nth-child(3) > *:nth-child(1)

Query params:

• url: url of the webpage you want to fetch
• selectors: css selectors describing the elements interestes you

Returns:

• innerText: element innerText
• href: element href

Run locally

npm i
npm start

How to deploy your own CloudQuery to AWS lambda

1. Configuration

• update profile(AWS Credential Profiles) in up.json to use your own aws account
• update rateLimit in config.json to set your own rate limit (Default: 5 request/hour)

2. Deploy

1. npm install
2. install up
3. Input up in terminal and hit enter :)
4. up url to see the API baseURL of your CloudQuery

Contribute

Backend

vim app.js

Frontend

cd frontend
# develop
npm start
# build
npm run build

Thanks

• up for deploying serverless API to AWS with ease
• serverless-chrome for running chrome on AWS lambda
• finder for making it easy to select elements on webpage