Exposing less ports
wernight opened this issue · comments
Related to issue #1554, is there a way to only expose something like 10 ports or less? I find it especially strange to require exposing so many port for a single-user system.
Currently we are using docker auto-publish feature for port exposure. That's why we can't control the number of ports that should be opened on a server.
Oh I see. One way would be not to expose that port directly to the internet, and proxy it via the che server (the one running on port 8080), possibly on another port. Or let users specify the port on which it'd get exposed.
@wernight - that is correct. That is a much trickier solution. It's what we do in Codenvy and one of the ways that we are able to enforce user-based permissions. All traffic in Codenvy runs through a go-router that is installed as part of the system. And this way we can use tokens to enforce identity and permissions, including some port proxying, but not to the degree you are suggesting here.
We'd need to run some sort of in-memory port-port mapper. This could make the product more secure as you could have your docker daemon that is managing workspaces non-accessible by any IP external IP address, so that only the Che server is the sole point of contact.
This would be a major improvement to the underlying subsystems. Not sure how to label this, as we'd have to look at a generational major version for such a capability.
I'm not sure how Docker is used internally, but I expect something like this docker-compose.yml:
nginx:
image: nginx
ports:
- 80:80
- 8080:8080 # Probably needs one more, possibly one per container.
links:
- che-server
- some-docker-container
che-server:
image: ...
some-docker-container:
image: ...
I do see the issue that some-docker-container is actually dynamic, hence requires a dynamic port mapping. I'm willing to help, but I don't know where to put the code. For example I don't know if I should add a hook that opens a port externally upon being called, or if all connection can go through port 80 and some kind of virtualhost can be applied to then detect where it should go.
The push model (hook being called by Che when a workspace is started), would allow also to simply expose the port, which would allow a Kubernetes deployment with some hacks. The pull model (when clients try to connect they tell which container they're trying to reach) allows a much cleaner model that would be simpler to deploy and safer.
The current solution requires a dedicated machine which is too costly, not really safe IMO, and doesn't support re-deploy on failure like Kubernetes allows.
Another idea, for the current single-user would be to allow only a single running instance of a workspace/container. This would mean that the docker-compose above could be used directly.
PS: What are port 8000 and 49152-49162 used for in the Docker image?
The 8080 is the port of the Che server itself. The other ports you shouldn't be exposing - the docker run syntax was outdated vs. the latest syntax we support now. Each workspace starts by exposing ports at the beginning of the ephemeral port range, which starts at 32768, I believe. The ports are then opened in sequence.
I think @garagatyi believes the proper solution would be to enable some sort of port restrictions for each workspace when the workspace is launched. This would be modifying the syntax around our "docker run" command to pass in these parameters if they are defined in the che.properties file. So maybe it's restricted to a 10 port range, for example and then it passes in the publish attributes into the docker run.
We love pull requests - definitely want to improve here. Alex would have the sort of pointers necessary to outline the implementation plan for such a solution.
Yes simply cycling 10 or so ports sounds also pretty feasible, but still how is the network protected? Putting 8080 behind Nginx with basic authentication is easy, but what's the authentication that one can set up on the other ports?
@wernight - we support permissions API and full authentication inside of Codenvy right now. Codenvy is a sytem that is locked down. There are many elements to the engineering that has been done. We will bring some of this into Che, though we are trying to figure out what are the elements that we can bring into the system without adding too much complexity to the architecture.
Let me elaborate what codenvy does:
- There is a central user database.
- There are multiple forms of authentication, which can generate user-specific tokens.
- There is a broad-based permissions API that restricts access to resources within the Che server (can I create a new ws?), and also to resources within a ws (am I allowed to execute commands?).
- We then have an enforcement of permissions that lives in a few places - access to the dashboard, within the che server, within the raw workspace (agent inside), and then also customizations to the web terminal running in each workspace so that you cannot randomly open a terminal if you know the IP:PORT, and then also within each ws agent that is running which exposes REST commands from the workspace to the outside world.
- http/s support for all link points.
So authentication is a generic term - but because we have multiple touch points where software runs within a single che system, authentication / permissions have to be extended to each point on the system.
BTW - we do let orgs run Codenvy with a Fair Source 5 license - so you can operate it for small groups with certain restrictions for free.
But I think locking down the ports is absolutely essential within Che. We also know that Bitnami is adding some Apache filters to add in basic authentication that sits in front of a Che server.
That does seem pretty complicated. Jupyter uses WebSockets but through a single port. I have no idea about the protocol the agent/terminal/... are using through the exposed random Docker port. Supposing all those could use a HTTP protocol, it would simplify a lot of things while still allowing WebSocket performance. It'd allow only exposing a single port. It'd also allow basic AUTH to limit access to everything and HTTPs for everything.
I'll look again into Codenvy free tier (now that I found how to set up SSH key). Binatmi doesn't sound flexible and light enough.
This remains my No. 1 issue with Eclipse Che as I see no way to safely deploy it anywhere that is publicly accessible and I see no way to guard other opened ports. They may be opened only while a workspace is running but still. I it requires settings up an authenticated revert proxy with WebSocket support (not sure if nghttpx supports that).
Another fix that could make it a non-blocker for people like myself may be to have at least a token required for accessing those extra ports. This would allow to setup rate limits and https on the HTTP port and make it pretty safe. I'm not suggesting the whole multi-user, just a nonce provided on initial connection to HTTP (without authentication).
To protect ports with some sort of authentication we would need to change the way we allocate these ports.
@wernight Can you provide an example how opening ephemeral ports make system vulnerable?
I don't know what protocol you're using and I don't have the resources to investigate. Based on what I know, it seems possible that Eclipse Che exposes a port to access the terminal on a port in 32768+. That would mean that a port scanner may bypass any authentication set on port 80.
But port scanner can scan only 32768-... ports. And you probably don't have any other software listening on these ports. System services use ports lower then 10000 usually. Ephemeral ports are used for outgoing connections, so pretty safe in that case.
I have to open 32768-65535 ports for Che to work (else remains stuck during Starting workspace runtime after Successfully built ...). Also the network diagram shows that the server is listening on those port:
. Also the started Docker image opens ports:
- 0.0.0.0:32845->22/tcp
- 0.0.0.0:32844->4401/tcp
- 0.0.0.0:32843->4403/tcp
- 0.0.0.0:32842->4411/tcp
- 0.0.0.0:32841->8000/tcp
- 0.0.0.0:32840->8080/tcp
- 0.0.0.0:32839->9876/tcp
And I can just SSH in my public IP on port 32845 (it does seem to require a password but not sure if it's a nonce).
0.0.0.0:32845->22/tcp
0.0.0.0:32844->4401/tcp
0.0.0.0:32843->4403/tcp
0.0.0.0:32842->4411/tcp
0.0.0.0:32841->8000/tcp
0.0.0.0:32840->8080/tcp
0.0.0.0:32839->9876/tcp
It is a mapping of your public ports on host to ports inside of containers. So no one will be able to connect to your host system. Only inside containers. Because of that I don't think it is too risky to open 23768+ ports.
But if you afraid that someone gets into container then yes, authentication is required. But it doesn't have connection to range of ports you have to open.
About the risks:
- Docker containers do not contain. They limit but they are not a sandbox environment. That's why one should not run as
rootinside containers for example. - I believe one of the most important and useful Che runtime, is one able to build and run docker images. This means basically
rootaccess from within the container, which is pretty close to a root access on the host machine.
It is possible to map root in container to non-root on host, but docker inside container is very dangerous.
Yes latest Docker adds some improvements, still not sure that's on by default. I imagine new users making a custom workspace running as root or other mistakes resulting in an unexpected attack vector.
Outside of having root access, there is access to the entire project source code even if user had set up Nginx authentication for HTTP on port 8080.
root mapping is disabled by default AFAIK.
Outside of having root access, there is access to the entire project source code even if user had set up > Nginx authentication for HTTP on port 8080.
Can you describe that situation more. I didn't understand what do you mean
Assumptions: Putting Che port 8080 behind Nginx reverse proxy for authentication isn't hard so let's assume that this is safe. Again I'm assuming here that one of the exposed ports in 32768-65535 does not require authentication nor a cryptographic nonce.
Risk: As you've imported your project, an SSH access inside the container means at least read access to /projects/my-project and allows to dump your entire source code.
So setting authentication for all exposed ports will be secure solution, right?
Ideas for fixes:
- Require authentication or nonce on all exposed ports (which however doesn't allow deploying on Kubernetes I think).
- HTTP reverse proxy through port
8080, or any single fixed port actually, instead of exposing ports32768-65535(adding authentication to an HTTP port is easy). - Don't require opening other ports than 8080 and use hole punshing to establish a connection (probably the hardest to implement and not the most secure here)
- Require VPN, or using a proxy like SPDY or HTTP/2 (hack to achieve something similar to solution (2))
I believe solution (2) gives the most advantages, like allowing to deploy on Kubernetes and a lot more. It might even be the simplest to implement.
OMG! I just found out that it's actually the same on Codenvy. I didn't validate that it affects private repository but I think you should immediately block any SSH connection for users who didn't give a custom user/password and/or key.
The issue is that one can easily port scan the few codenvy IPs and SSH into any running workspace, private or public. This includes without limiting to copying the entire source code, using those machines to perform DDoS or other mischief, potentially access passwords or SSH keys to access more resources, inject code the project being developed, inject code into the browser of the user working on developing on web servers to potentially steal cookies...
We agree on the reverse proxy sokution. We have to make some modifications to Che to support having all browser - workspace comms go through a single reverse proxy port. But we know what to do and will get it scheduled.
@wernight - On the SSH of workspaces issue. It's possible to remove the SSHD from the workspaces when you create a custom workspace. We do make them necessarily simple for access by default. We have designs for a more advanced ipmlementation of the SSH stuff where we will do exactly what you suggested, where users can upload the appropriate key and / or a custom user name and password. We just haven't implemented that segment yet. It requires some coordination between the user dashboard and the IDE.
The issue is that one can easily port scan the few codenvy IPs and SSH into any running workspace, private or public.
@wernight Can you explain how it's possible without ssh keys?
ssh -p 34888 user@node1.beta.codenvy.com and the password is secret (scan here the port# and node# of course). I wouldn't put a full exploit here publicly, but I wouldn't mind to try it if you someone from Codenvy with authority allows me.
yes you right. @wernight do you think that randomising password will close this problem?
Sure it'll be a great improvement! Just not a complete fix. Most people deny password-only SSH access (i.e. without key). So I'd also include a rate limiting network rule to help.
I also didn't look at other ports, there may be similar issues.
@wernight @skabashnyuk we need to rerwork all of our base images to either create a passwordless user or generate password that isn't exposed.
You're focusing here on a single issue. It's an important one but there may be more. What about locking the entire network authenticated users (or given Che is single user, to users who used the web UI and have receive a nonce token).
@wernight - we do that in Codenvy - every user is tokenized, and then we have a permissions API with an enforcement system that restricts access for that user given their permissions in a number of places: SSH, web terminal, basic access to the terminal, and within the Che server.
There was a lot of complexity that was introduced by implementing that system, and we have not figured out if there is a simple way to bring it into Che or not.
Can you explain briefly each port:
- How is it used
- Protocol
For example I understand SSH si for... SSH and protocol is SSH. Easy but also one that is accessed outside of Che UI. My guess is that many may just go through the web UI and that may not be that hard especially on a single user. It's basically like a virtualhost per workspace ID.
Ports are documented in this graphic:
https://www.filepicker.io/api/file/isOAOgCRQcyvznVpkPYj
The SSHd is not a mandatory port. It is exposed in our base Docker images, but SSH can be removed. The other ports are necessary for the terminal, the communication of our agent back to Che, and for Che itself.
The ephemeral ports are for any servers launched from inside the workspace itself. We think of the SSHd that is in the workspace as a variation on this. So that list can be added to / removed from.
Looking at my previous inspection:
- 0.0.0.0:32845->22/tcp SSH
- 0.0.0.0:32844->4401/tcp agent via web UI. HTTP?
- 0.0.0.0:32843->4403/tcp agent via web UI. HTTP?
- 0.0.0.0:32842->4411/tcp terminal via web UI. HTTP?
- 0.0.0.0:32841->8000/tcp ???
- 0.0.0.0:32840->8080/tcp web UI via HTTP
- 0.0.0.0:32839->9876/tcp ???
0.0.0.0:32844->4401/tcp agent via web UI. HTTP?
0.0.0.0:32843->4403/tcp agent via web UI. HTTP?
0.0.0.0:32842->4411/tcp terminal via web UI. HTTP?
yes
0.0.0.0:32841->8000/tcp ???
0.0.0.0:32840->8080/tcp web UI via HTTP
not mandatory, exposed to help users run tomcat or other web server
@garagatyi - what is port 8000 and 9876 used for?
Is port 8000 the tomcat JMX port?
8000 Tomcat debug port
9876
I don't know why it is exposed. @eivantsov wrote that Dockerfile, maybe he knows.
Ok sounds like 3 ports could be already closed by default and the rest are all HTTP.
Could have Che on 8080 behind Nginx with 8080 reverse proxied to Che 8080 by default. Then if workspace foo is started, instead of going through example.com:4401 it could go to foo.example.com:8080.
Now to make this dynamic either:
- Update Nginx configuration on the fly when a workspace is started/ended (probably simplest because Nginx supports websockets and more)
- Write a custom server (like what you probably did for Codenvy)
There are couple of dynamic Nginx examples, I can help if needed.
What this achieves is that it only required port HTTP 8080 (let's call it port 80). Now this can be put behind authentication and that'd block most attacks, and allow more deployments.
We were thinking that to get things going as a start, we'd have a static nginx configuration and then within the che.properties file, you tell che where to route all workspace requests. After the browser is updated, the browser would communicate directly through nginx to each workspace. So we were already going to improve the che server to setup some sort of option where, if activated, then workspace URLs are encoded for nginx to be able to route to the proper location.
If you wanted to get involved with helping create an nginx configuration for this - it would be great. I don't think a dynamic one where Che has to update nginx is a good approach - too many failure points can appear I think. I think we need a design where everything nginx needs to know is baked into the URL.
That sound great! Any timeline for this? I was mostly thinking about fast websocket support, but may be Che is already doing that well. I don't think I'd have time to directly read and understand Che's code.
We already do all of the websocket stuff you can imagine. It's how che runs so fast to begin with - so you probably do not need to get into that layer.
I think figuring out this port stuff is definitely a Q3 priority for us. We get asked about it quite a bit. Our engineers are trying to figure out the right strategy. We have been discussing it this week. I think once we have it - it will not be difficult to implement the strategy.
@wernight - in discussing with the engineers, it turns out that we have all of this logic inside of Codenvy already. We have a mechanism within codenvy to embed all of the workspace IP + port numbers for every server into an underlying URL. We will move this into Eclipse Che and then provide a static configuration for nginx. So there isn't much work that you need to do. We have it all, just need to port it over.
Hi,
Just wondering if you had any update on this?
It's in the roadmap. You can track when it gets picked up for development in the milestone plans. Generally our milestone plans have been more backward looking, but I am requiring various dev teams to make them forward-looking, so you should always be able to see 1-2 milestones out. We are just wrapping up the 4.6 milestone and will get the 4.7 and 4.8 milestone plans documented shortly.
This change is dependent upon a bunch of things internal to the system around environmental SPIs.
If you're having difficulties building it yourself, the Kubernetes Nginx Ingress Controller does something pretty similar: It updates virtual hosts dynamically by calling nginx -s reload after building new config from templates. There is a very simple version of that entire logic at https://github.com/kubernetes/contrib/blob/master/ingress/controllers/nginx-alpha/controller.go
@TylerJewell what should we do this with issue?
We should leave the issue open - it's a long-lived issue. The topic is legitimate, and we need more time to consider solutions and weigh its priority.
Bump
Status hasn't changed. It is a long lived issue. If it is something that we take into the roadmap it will show up in the roadmap on the wiki. The roadmap covers the 6-9 month horizon. We have had some companies become codenvy customers and contract us to deliver roadmap items on their personal timeframe. Otherwise, it will be taken when a dev team gets it into the top of their backlog or others issue a pr.
If I understand it means that at the moment it's not planned for the next half year. It's sad because it's really blocking to use Eclipse Che without VPN, and IMO to have many people start using it.
It is open source and there are 300 issues in the backlog all of them complicated. Everyone who files an issue would say the very same thing you just said about adoption. So the best a project leader like myself can do is to help organize the backlog with the best available data at any time.
There is no quick fix for a solution even if we started today. It is a very complex issue with URL rerouting. And it is open source - cough cough free - so we would love for someone to give codenvy a partnership arrangement that helped finance getting this done sooner. We would like to get everyone's request done sooner.
May be you're right. I'm not sure how many people install GitLab for example on a publicly accessible IP. I'd imagine that there is a comparable amount of people that may be interested in Che but if they understand the risk they would wait for this issue (see also https://www.slant.co/topics/713/~cloud-ides). Up to the PM to find the most important features to add first :)
I am not sure that slant is a great site that reflects true community usage. The pros / cons for us - for example - were authored more than a year ago, and many of them are not accurate anymore. I should be bragging because we are at the top of the list, but we tend to take requests from github issues + voting here more signiicantly than something like slant.
For people interested I found a way to deploy it behind SPDY proxy. It's not as neat as a reverse proxy but it provide authentication and exposes a single port to the internet: https://github.com/wernight/kubernetes-che
Still this issue is really import IMO just not fully blocking anymore for me.
A off topic question
Yes you right. @wernight do you think that randomising password will close this problem?
This was done completely?
Looks done: https://github.com/codenvy/dockerfiles/blob/167dc9393cd79f0286577aa2210d91097fd2722d/debian_jre/Dockerfile#L30
+1
This is blocking me from being able to use Che. Using a VPN or proxy is not a solution because it's not at all portable between computers -- it requires a lot of setup across computers, which completely defeats the purpose of a cloud IDE. I was trying to deploy this for my own personal uses on a public-facing dedicated server, so that I could become familiar with it and use it in my own projects before recommending it at work.... buuuut it looks like I won't even be able to try it to recommend it for work, because as it stands, IMO, it's fundamentally broken. Kind of unfortunate...
@allquixotic - see this proposed improvement, which would help with port reduction on a system like OpenShift. #2004 (comment)
Also, please try codenvy. It has strict authentication and permissions enforcement. We give the software away for up to three users, so if it's just for personal use, it's going to meet your needs. It does not restrict the port range, but it does have strong forms of security to ensure that your workspaces are only accessible by people you give permission to. We are about to announce next week at CheConf that you can run Codenvy with the same CLI that you can run Che with.
IMO, it's a enough reasonable solution to use with a forward proxy server with authentications.
Che itself should be light-weight as possible as.
Probably your client browser have no WebSockets support also if it doesn't support PAC.
And some clients may be behind firewalls that open 80, 443 only.
I'm not against reducing ports. But it doesn't resolve all.
@wernight - working to improve Che so that all traffic goes through a single port, including workspaces, is going to be taken as an issue in Q1. it's a challenging issue, but everyone agrees on the general direction of what to do.
Is this still an issue? If so please let us know otherwise in 1 week we will close this due to inactivity.
This is still an issue - this should remain open until we have reverse proxy with port management.
No. This is going to take many quarters before we see a solution.
You mentioned something in the works by RedHat on one of my issues this year (#3603), has anything been released yet?
you can see all of the development in the repo. There is no open branch of dev yet on this. Just early designs. It is a massive undertaking to build it and test it. Probably thinking at least six months of work.
It seems that Codenvy also requires ports tcp:32768-65535 to be open. It states on the documentation that "This range can be limited." however it doesn't state how. Like one should inform Codenvy about there currently allowed range. Like, I tried just reducing the allowed range in the firewall and then info --network command fails to connect to the agent (so it'd fail to start).
This seems common to Che/Codenvy.
@wernight I think info --network uses the same port all the time. I can't find that function quickly. @TylerJewell is it correct? Does fakeagent container uses the same port all the time?
Also, I think reducing allowed range is firewall isn't the right way. Does it work if you do so and run a random container with -P. Does Docker use the beginning of your range or start picking ports from the beginning of the ephemeral port range?
In other words, you do not have to inform Codenvy that you have got a new ephemeral port range, but you should inform Docker. Codenvy just lets Docker published all ports exposed in a container.
So it sounds there is a possible bug on Codenvy, not sure for Che.
It's possible tell Docker to use a specific port (instead of using its auto-port selection) and manually select it within a limited range provided via some configuration flag. The reason limiting the ports to let's say 10, especially for Che:
- Can manually list them one by one (for example Kubernetes requires that, also reverse proxy require that)
- Who would need more than 10 on a single user Che instance? If they do, they could increase the range, or have multiple Che.
We use ports auto publishing implemented in docker. So it is docker who controls range. And docker gets this range from system ephemeral port range.
I agree that it would be better to configure ports range in Che/Codenvy but it is not implemented.
Looks like there is no way to limit that just for some containers in Docker right now, see moby/moby#13322
@wernight have you tried limiting this range on the OS level?
One could change it in /proc/sys/net/ipv4/ip_local_port_range, and I'll make a note of that in Che on k8s, but that seems very wrong generally speaking.
Given improvements in the recent versions of Che and Codenvy, those limitations will not achieve the goal you are seeking. The only solution here is for us to provide a new server resolution strategy that channels all traffic through a reverse proxy under a certain port. It will be done some time this calendar year, but we cannot offer any specific timeframe.
We now have a prototype for how we will channel everything through a single port. This does not yet channel SSH traffic, but all primary workspace traffic. #4440 (review)
I should note that this code may or may not get committed. We haven't been able to figure out a clean way yet to solve the SSH issue and we have to evaluate the impact to custom assemblies of Che like Codenvy, still.
I think SSH can be left separate for now as it's also an optional thing for a cloud IDE with a terminal. That could also allow to rate limit or block SSH if so desired, and it's a fixed port as far as I recall.
Can this issue be closed now we have #4361
Not only that. With Che on OpenShift all internal networking stuff is handled by OpenShift HaProxy, services and routes.
Awesome, but don't forget about those of us who don't use OpenShift ;)