IonicaBizau / scrape-it

🔮 A Node.js scraper for humans.

Home Page:http://ionicabizau.net/blog/30-how-to-write-a-web-scraper-in-node-js

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Hosted API

chrisvxd opened this issue · comments

Hey @IonicaBizau 👋 Thanks for the awesome library.

I'm looking to convert open source libs into SaaS APIs as part of my work with saasify.sh and thought scrape-it would be a great fit.

Here's a working demo of what I was thinking, generated from this source:

curl -X POST \
  'https://api.saasify.sh/1/call/dev/scrape-it@ad97b8f8/index' \
  -H 'content-type: application/json' \
  -d '{
  "url": "https://ionicabizau.net",
  "opts": {
    "title": ".header h1",
    "desc": ".header h2",
    "avatar": {
      "selector": ".header img",
      "attr": "src"
    }
  }
}'

We'd like to offer a rate-limited API and reserve most of the profits for you. As an example, we've also worked on converting sindresorhus' imagemin and minimaxir's stylecloud to APIs.

Looking forward to hearing your thoughts! Would also be happy to jump on a quick call to discuss.

That is amazing! ❤️

I do advise in that you will have accounts or some kind of user authentication so that in case they abuse your server (by making requests to weird urls) you will be able to stop these requests.

Anyways, it's a wonderful idea! ✨

Would also be happy to jump on a quick call to discuss.

I am very busy at the moment, but I think email works better: contact@ionicabizau.net.

@IonicaBizau so glad to hear you like it!

We automatically add public rate limits to each of our APIs, so that shouldn't be a concern. To remove the limit, people need to add billing information.

I'll keep the discussion in this thread, if that's alright? We also have an open slack if you want to join.

If you're keen, the next steps I'd take are:

  1. Polish the landing page by tweaking the saasify.json
  2. Add some documentation
  3. Publish it to the domain scrape-it.saasify.sh
  4. Set up payouts to you - we can discuss this again once everything else is put together, but let me know if you have any questions about that.

Ideally I'd also love to add a button to your README to help funnel some of your users towards the API. This should appeal to your visitors that aren't using node, or users who don't want to bloat their payload. Something like:

View Hosted API

That's wonderful! Just joined Slack.

Let me know if you have any questions and I will be happy to help.

Best!

I've just published the latest version to https://scrape-it.saasify.sh 🙌

See the src here: https://github.com/saasify-sh/scrape-it

Let me know what you think!

That's lovely! I like the HN example.