blip.strongloop.com is dead - what is this?
STRML opened this issue · comments
Looks like you guys are (abusing) optionalDependencies to send a callback to your servers when someone downloads a package.
When it goes down, it has the nice side effect of making npm install take ages.
All SL projects are throwing something similar to:
npm http fetch GET http://blip.strongloop.com/loopback-connector-remote@1.0.3
npm ERR! fetch failed http://blip.strongloop.com/loopback-connector-remote@1.0.3
Oh, is that what blip.strongloop.com is? This is severely affecting us too.
👎
-1
👎
Just so you guys know, this affects us even when blip.strongloop.com is up because we are behind a firewall that does not allow connections to blip.strongloop.com.
👎
👎
👎
blip problem still annoying
Is blip still annoying and affecting the installation 'experience' of loopback?
I got time to test my self this evening: still an "issue"
ob_000@OB-X1-CARBON ~/tmp
$ npm install loopback
npm ERR! fetch failed http://blip.strongloop.com/loopback@2.18.0
npm WARN retry will retry, error on last attempt: Error: connect ECONNREFUSED
etc etc long time etc etc
The basic workaround is to:``
npm install loopback --no-optional
# and then the installation is sooo quick
I wonder why optional is not an option: why are optional components installed as default? npm problem
Why are the optional components needed in the package.json at all: they should be optional like so many other things, npm install theOptionalStuff
Optional doesn't mean that it won't install by default, just that it won't
abort the install if it fails.
On May 21, 2015 2:47 PM, "Owen Brotherwood" notifications@github.com
wrote:
I got time to test my self this evening: still an "issue"
ob_000@OB-X1-CARBON ~/tmp
$ npm install loopback
npm ERR! fetch failed http://blip.strongloop.com/loopback@2.18.0
npm WARN retry will retry, error on last attempt: Error: connect ECONNREFUSED
etc etc long time etc etc
The basic workaround is to:``
npm install loopback --no-optional
and then the installation is sooo quick
I wonder why optional is not an option: why are optional components
installed as default? npm problem
Why are the optional components needed in the package.json at all: they
should be optional like so many other things, npm install theOptionalStuff—
Reply to this email directly or view it on GitHub
#1079 (comment)
.
Yep: but it is loopback that has defined that the module should be in package.json.
npm does it's job (even though a bit illogical with optional being non-optional by default)
There are so many other modules that one has to install by purpose (ie optional to the working of loopback) but not "the blip"
@STRML possible edit Issue title to something about "the blip" should be installed by purpose as an extension of loopback ...
Edit: I wrote "npm problem" in my previous comment, I really meant that it was the way npm was being used/abused (opionionated)
I agree, I think they are abusing optionalDependencies here in order to track users; it is user-hostile.
@STRML I would probably say "not thoughtfull".
I do not know what the functionality is (yet).
End User Experience
There is a considerable time overhead with the current situation of "the blip" and installation of loopback where it is not possible to contact "the blip"
This feature is affecting the installation experience of any user that has to deal with enterprise politics for what is connectable and is related to firewall's
Solution but with a but
It may be that all the methods used to install loopback allow for a --no-optional flag, and that there are no conflicts.
Possible workaround
In the meantime ...
... adding a host entry for 127.6.6.6 for blip.strongloop.com will shorten the installation time as the initial network connect will not require an external dns lookup, and a network connect will fail quicker.
Note: if you use blip.strongloop.com for anything else, then this is not the solution for you
@crandmck : the workaround or description of the time out is a possible addition to the firewall area of the doc
@crandmck : are you able to give this issue a tag and/or a callout
strongloop/strong-build#29
#1389
Edit:
@sam-github : Thanks
Simply not installing optional dependencies will have side effects you probably don't want, such as not having a loopback explorer: https://github.com/strongloop/loopback-example-app/blob/master/package.json#L22
We'll look to see what options we have for making npm install fail faster on unreachable optional dependencies.
For now I'm just setting blip.strongloop.com to 127.0.0.1 in /etc/hosts.
Rather than thinking about a patch to npm, would somebody please explain what blip.strongloop.com is for and why this optional dependency exists?
The choice of 127.6.6.6 was chosen with care: 127.0.0.1 is sometimes used for listening on and could also be a port that is being used for collecting the tarball by npm (defined in package.json).
In the case of firewall installations, the protocol used for git is often http(s), and the chances of a process listening on 127.0.0.0 increase in a developer environment.
Unknown side effects for development environment
The chances of 127.6.6.6 being used is minimual: better safe than sorry...
Edit:
https://github.com/strongloop/strong-pm/blob/master/README.md?#troubleshooting : shows use of 127.0.0.0 with a port === better safe than sorry when such suprises are in store
Understood. I never run anything on 80/443 personally, but I get what
you're advocating.
On 5/22/15 1:11 PM, Owen Brotherwood wrote:
The choice of 127.6.6.6 was chosen with care: 127.0.0.1 is sometimes
used for listening on and could also be a port that is being used for
collecting the tarball by npm (defined in package.json).In the case of firewall installations, the protocol used for git is
often http(s), and the chances of a process being present increase in
a developer environment.
Unknown side effects for development environmentThe chances of 127.6.6.6 being used is minimual: better safe than sorry...
—
Reply to this email directly or view it on GitHub
#1079 (comment).
@ritch @bajtos Would like to see some movement on this user-hostile abuse of optionalDependencies. Not only has it caused many install problems for our servers behind firewalls, but it is also downloading from a non-secure URL, meaning it's a possible MITM vector. Given how npm packages support postinstall, anyone MITMing blip.strongloop.com gets code execution on any server running npm install and using a strongloop package.
@raymondfeng Deep issues with some organisational aspects StrongLoop / Loopback or just time to fix?
@STRML you should patch the inconvenience and submit a PR. Since you already know the POC attack, you are the most suited to contribute so we all can benefit.
That's what open source is about, isn't it?
@amenadiel This is not like other contributions, where there is an implementation cost. I am asking them to delete a line from the package.json of every strongloop project. Submitting a PR will only fragment the discussion.
This also kills installs with alternate package managers like ied.
Time to fork every strongloop package - what a pain.
-1 for the need to fork
👎
👎
This probably should have been closed by strongloop/strong-tools#42 a couple weeks ago.
👎
👎
👎
@rmg It appears the new scheme:
- Avoids use of optionalDependencies, blocking
npm install, and closes the mitm vuln, which is good, - Adds an
.sl-blip.jsto every git repo. This is good, at least we can see what is happening. - Times out after 500ms, so it won't significantly block installs on production servers that don't have internet access,
- Has not actually been deployed to any projects I can find, so this issue should remain open, and,
- My major complaint: does not prompt the user to see if they opt-in to tracking. I doubt that users want their production servers hitting your (still insecure) endpoints.
Why not use npm analytics?
The entire concept is overtly hostile to enterprise IT... It's stuff like this that hinders Node.js adoption in big companies like IBM.
I work at a Fortune 50 company. We run our own internal npm registries. We don't allow packages with GitHub dependencies (only versions tagged on npm). We shrinkwrap our application and audit version changes carefully.
We don't want our developer machines contacting the outside world. We don't want our build servers contacting the outside world. We can't use any StrongLoop code that contains this tracker.
Did I miss your sarcasm, or are you unaware that StrongLoop is IBM?
I thought it was a creative way to get some attention to this issue... :) My apologies for any confusion.
👎
This isn't an official StrongLoop or IBM response, just my own thoughts/opinions.
@STRML thanks for the feedback, and for looking at what has been done.
Has not actually been deployed to any projects I can find, so this issue should remain open, and,
Unfortunately this is all happening in the middle of a code freeze that is beyond our control, so the rollout is being hampered by that :-(
My major complaint: does not prompt the user to see if they opt-in to tracking. I doubt that users want their production servers hitting your (still insecure) endpoints.
My original thought was that I didn't want to incur any additional latency and that it wasn't a concern anyway since nothing was actually being downloaded, but that was my mistake. Thanks for catching that, I'll make sure it is updated before it is used.
Why not use npm analytics?
From my perspective it's mainly a matter of npm stats being at the package level while we wanted them at the version level so we could get an idea of how releases propagate.
Thanks @rmg for the response.
This is a good reason to reach out to the npm developers to see if they will offer a more featureful analytics solution that doesn't involve compromising the trust of your users. It could be a nice product for them to sell for a nominal fee to pay for development.
An official response has been posted on StrongLoop's blog: https://strongloop.com/strongblog/strongloop-and-sl-blip/
we are working on removing sl-blip in our next release while we find a more elegant and non-intrusive alternative with clearer opt-out options.
Does this mean that the .sl-blip.js in strong-tools will not be a part of new packages?
Does this mean that the .sl-blip.js in strong-tools will not be a part of new packages?
That's correct.
Great. Thanks. I'll close this issue once the dep has been removed from loopback and related packages and a release has been pushed. It should be backported to the 2.x branch as a patch as well as it is a serious vulnerability to all consumers of those packages, on par with curl http://insecure.com | bash.
nice find
I'm not just experiencing a slow npm install step, but my build is failing outright, and I'm not even behind a corporate firewall. I'm running a loopback app on a Dokku-Alt machine on Digital Ocean, and the build suddenly started failing with no explanation, right after outputting node .sl-blip.js. I've already tried changing the etc/hosts file so that blip.strongloop.com points to my own machine. but it didn't help.
npm install works fine locally, fwiw.
I tried deploying to a vanilla Dokku instance, ( via the one-click-app tool in Digital Ocean ) and it also failed in the same way. The weird thing is, I deployed to Heroku and it passed this step and finished the build. Ideally I don't have to migrate to Heroku, though. Any insights?
My next move is to try https://github.com/strongloop/strong-tools/pull/42/commits. <-- has this fixed the issue for other folks?
@bradwbradw if the build is stopping suddenly without warning it may be that your environment doesn't have enough memory. The blip stuff was always optional, so even if it failed then npm install would just keep on going.
@rmg thanks, i'll look into that
@rmg He's already running the new scheme (note node .sl-blip.js). The new scheme isn't optional in the same sense, it's a preinstall hook, which blocks npm install until it exits. If it fails for some unknown reason, it'll stop or hang the whole build.
Not sure how many user complaints your team needs to realize that blip has been a user-hostile decision from the start.
Hello everybody, I searched for all LoopBack projects with sl-blip in their dependencies, removed sl-blip from package.json and published new versions.
Here is the list of updated packages:
- loopback@2.27.0
- loopback-component-push@1.5.3
- loopback-component-storage@1.7.0
- loopback-connector-remote@1.2.0
- loopback-connector-soap@2.3.1
- loopback-connector-mssql@2.4.1
- loopback-connector-mysql@2.2.1
- loopback-connector-postgresql@2.4.1
- loopback-connector-oracle@2.3.1
- loopback-connector-sqlite3@1.1.1
- loopback-connector-rest@1.10.2
- loopback-component-oauth2@2.3.7
I hope that makes this issue resolved.
However, if anybody finds a loopback- package that still installs sl-blip, then please leave a comment here asking me (my GH handle) to fix it.
Thank you all for the patience you had while waiting for us to address this issue.
I'll close this issue once the dep has been removed from loopback and related packages and a release has been pushed.
@STRML I'll leave it up to you to close this issue when you agree we are done here.
+1 nice to hear that.... <3
Thanks @bajtos. Is the plan no longer to do the https ping via .sl-blip.js? As innocuous as that seems, I've already seen issues popping up where it's stalling builds.
@STRML removing the deps prevents the release tool from replacing it with the script version. That means sl-blip will be completely removed in any form from the packages :-)
@rmg So does this mean that loopback will no longer be collecting this type of analytics? I see blip is still maintained in strong-tools. What's the plan for the future?
@STRML it's dead as far as I know. I just didn't want to leave the code in a broken state, even if it isn't being used.
So does this mean that loopback will no longer be collecting this type of analytics
Yes, that's correct.
👍 Thank you for finally doing the right thing. It is unfortunate that it took a full year to do so.