Be able to absorb 100s of new users at once
lourot opened this issue · comments
Demand is high: https://news.ycombinator.com/item?id=17951478
as per my comment on HN, a good way to do this might be using headless chrome via puppeteer from within a Google Cloud Function or using AWS Lambda. IP rate limiting might not be an issue given how the network of worker nodes is set up. You could even write one cloud function per provider (there are like 4 or 5 that do serverless functions including Google, AWS, CloudFlare, etc) and randomly select between these to further increase your IP uniqueness.
Yet another option might be to use Google's networking module to purchase a large pool of IPs and proxy through them via a cloud function as described above. I'm pretty sure that is possible.
Edit: looks like this might be possible from the Google Cloud control panel if you go to VPC Networking => External IP Addresses
edit again: apparently those can only be assigned to VPS instances. Still, you could use that to your advantage. Assign like 10 IPs to a beefy VPS, then you can do 10 profiles at a time if you can programattically select which IP to use for the current request via Google's API. I don't know amazon's stack as well but I'm sure they have the equivalent.
You might want to reach out to @zeit because they have a system that can build and deploy any Dockerfile and then auto scale as traffic increases. They will typically provide free or subsidized hosting for open source projects.
I am using zeit now to host a somewhat popular project with great success 😄
Maybe this can repo can help https://github.com/zeit/now-examples it's filled with lot of examples using Dockerfile deployments, also you can read more about it here https://zeit.co/blog/serverless-docker if you have questions let me know 😄
Yup. Now is the way to go. Or at least if the current plans are don't not enough, contact them, they may support OSS with some customized plan.
Now is just freaking zero pain and zero efforts, 3 characters on a terminal and nothing more, and gives so freaking much in return. It's not just a hype or whatever. This company is really pawning and rocketing the things.
One thing I would've recommended is to have a way to be notified when the service is available again when you removed the ability to join the queue. "We'll email you when your profile is ready" or even "we'll email you when our service can handle more profiles" would've been a better approach, though I am sure you were in a hurry to shut it off.
@ocdtrekkie Good idea, we have now created a twitter account: https://twitter.com/ghuserio . Thanks!
@AurelienLourot In case you weren't aware, Twitter is telling me that the profile is temporarily restricted when I try to visit.
We know and I just contacted Twitter about it https://twitter.com/brillout/status/1040643164774834176
Still? D:
Would love to use this if it is ready
@sam0x17 actually your profile is already there 😉 https://ghuser.io/sam0x17
@DiegoRBaquero yes still heavily working on scaling up. When this issue got opened we were serving about 200 profiles. Now there are 700 profiles and you are probably one of the 2000+ users who clicked "Get your profile" at some point, and we won't forget you 🙂 We're just kind of slow I admit. More details:
- the scaling issues I'm working on as we speak
- the roadmap
- the number of profiles is now visible on the landing page
Waiting for my profile 😃
How soon can I expect that I'll be able to create my profile in ghuser.io? Would love to contribute as well.
🎉 we have just managed to catch up. Everyone who has ever clicked on 'Get your profile' up to yesterday should now have a profile. A few more facts:
- We are not overloaded anymore. We serve 3k users and with the current design we can grow up to 7k users. Probably easily 20k users with a few tweaks. The bottleneck is GitHub's API rate limit but we're not yet out of ideas to go beyond, e.g. see ghuser-io/db#13
- About 10 people are joining every day at the moment. We don't have a satisfying automation implemented yet so when someone clicks on 'Get your profile', their username ends up in a queue which we process semi-automatically within 48 hours. Obviously this is one of the most urgent things to properly implement. This is what this issue is now about.
- The code base got dirtier at many places because we tried to rush at some point and we are now investing some time into cleaning this up.
- And as you may have noticed: we haven't stopped implementing new features :)
Update: Onboarding is currently stalled again: We are having an issue with our script that crawls GitHub (see #172). Good news though: We already have found a solution.
Update: #172 is fixed 🎉 Onboarding is reopened.
cool. i shall wait patiently... 📦
