UseCase #3: On-demand AR service

Question

UseCase #3: On-demand AR service

danielbehnke opened this issue 6 years ago · comments

Provide relevant data for AR maintenance. One idea is to use data from VNF Edge Analytics Engine here.

Stefan Schneider · Answer 1 · Mon Sep 16 2019 18:52:43 GMT+0800 (China Standard Time)

Idea: Hololens wants to connect to Internet through VPN VNF
VNF should contain simple VPN client (eg, OpenVPN) and connect to VPN server that we setup on some VM at UPB
Challenges:
- How to run VPN client on Kubernetes and server on VM and connect them to each other
- How to run and setup VPN client on Kubernetes such that it can be accessed from the outside. Bind physical network interface to the container? Goal: Connect the access point of the Hololens (or a Laptop) directly to that interface.

@KaiHannemann Start by researching for how to setup VPN with Kubernetes. How to setup VPN server (on VM here). How to connect VPN client on Kubernetes with physical interface. Etc.

Put relevant notes and findings on the wiki page: https://github.com/sonata-nfv/tng-industrial-pilot/wiki/use-case-3

Use this issue for questions, discussions, etc

Kai Hannemann · Answer 2 · Wed Sep 18 2019 20:07:08 GMT+0800 (China Standard Time)

The VPN server on the VM is running. Port 1194 needs to be opened though for outside access. Proceeding to VPN Client in Kubernetes

Stefan Schneider · Answer 3 · Mon Sep 23 2019 20:05:25 GMT+0800 (China Standard Time)

https://stackoverflow.com/questions/58062059/kubernetes-bind-physical-network-interface

Stefan Schneider · Answer 4 · Thu Sep 26 2019 13:53:59 GMT+0800 (China Standard Time)

If we run into problems with OpenVPN, apparently WireGuard is a good and very simple alternative. Just leaving it here for future reference if we need it.

Stefan Schneider · Answer 5 · Mon Nov 04 2019 21:42:46 GMT+0800 (China Standard Time)

@KaiHannemann Running containers in privileged mode in Kubernetes should work without a problem :)

For future reference (once we tested it successfully on native k8s), if we want to start a privileged container via the Tango SP, we need to define it in the VNFD. In the CDU section, add for the corresponding container:

parameters:
    capabilities:
    -   NET_ADMIN 
    -   SYS_ADMIN

Where the capabilities correspond to these. For our use case, NET_ADMIN should suffice.

Kai Hannemann · Answer 6 · Fri Dec 06 2019 20:19:02 GMT+0800 (China Standard Time)

Currently the addition of openvpn-systemd-resolved package caused a problem with the docker.socket.
The logger returns the error: logger: socket: No such file or directory

Stefan Schneider · Answer 7 · Fri Dec 06 2019 21:10:37 GMT+0800 (China Standard Time)

Inside the Docker (client) or at the VPN server? What if you remove the package again?

Kai Hannemann · Answer 8 · Fri Dec 06 2019 21:17:18 GMT+0800 (China Standard Time)

Inside the client. The removal is possible. We then require another fix to the DNS problem. Will try to do so.

Kai Hannemann · Answer 9 · Wed Dec 11 2019 00:50:05 GMT+0800 (China Standard Time)

I got the setup to work on my VM, the current push on my fork is up to date.
Had to adjust the proxy to let all traffic pass. This will need more work but does its job for now.
Tried it on my local setup, did not work. Did some troubleshooting and found that the proxy is functioning but the VPN runs into an DNS error. Will work on a fix for this, seems to be server sided.

Stefan Schneider · Answer 10 · Wed Dec 11 2019 22:20:26 GMT+0800 (China Standard Time)

@KaiHannemann I was struggeling to reproduce the steps for use case 3 that we discussed in the meeintgs. Could you please document the exact steps and commands to run the container and test it with curl?

I started here: https://github.com/sonata-nfv/tng-industrial-pilot/wiki/use-case-3#steps
Please check, update, correct, and extend the steps.

Currently I'm stuck testing the container with curl. I'm not sure what the correct command is

Kai Hannemann · Answer 11 · Sat Dec 14 2019 22:04:47 GMT+0800 (China Standard Time)

Current issue with the container outside of my VM is the /etc/resolv.conf file does not get updated. It works on the VM but not on my local docker installation. Working on a fix
Edit: It seems like this bug exists since 2013 and different linux versions.

Stefan Schneider · Answer 12 · Sat Dec 14 2019 23:32:47 GMT+0800 (China Standard Time)

Either way, please document all steps to reproduce what you have done so far and how to reproduce your problem. Ideally also how to configure the server

Stefan Schneider · Answer 13 · Thu Jan 09 2020 00:26:46 GMT+0800 (China Standard Time)

Status update:

Using the Proxy-VPN-Container and starting it as Docker container works
When configuring the proxy, internet access, eg, YT videos, is only possible if the container is running
Prepared descriptors for deployment on k8s and 5GTANGO SP. Needs privileged mode for creating the tunnel.
So far not successful: Without privileged and just NET_ADMIN capabilities, can't open the tunnel. With privileged, it works, but Squid returns an DNS error:

...
<p>The following error was encountered while trying to retrieve the URL: <a href="http://ifconfig.me/">http://ifconfig.me/</a></p>

<blockquote id="error">
<p><b>Unable to determine IP address from host name <q>ifconfig.me</q></b></p>
</blockquote>

<p>The DNS server returned:</p>
<blockquote id="data">
<pre>No DNS records</pre>
</blockquote>

<p>This means that the cache was not able to resolve the hostname presented in the URL. Check if the address is correct.</p>

Not sure where the issue comes from and how to deal with it.

Also updated the wiki: https://github.com/sonata-nfv/tng-industrial-pilot/wiki/use-case-3

Useful links:

OpenVPN client docker with instructions: https://hub.docker.com/r/dperson/openvpn-client/
K8s descriptor with privileged mode: kubevirt/kubevirt#1276

Stefan Schneider · Answer 14 · Thu Jan 09 2020 00:58:53 GMT+0800 (China Standard Time)

After Felipes, patch, privileged containers also work on Tango SP on int-sp-ath (and probably soon the others).

Now the error with the DNS resolution remains. Maybe check this: https://serverfault.com/questions/536975/dns-issue-with-squid

Not sure why it works with Docker but not K8s.

Stefan Schneider · Answer 15 · Fri Jan 10 2020 23:45:06 GMT+0800 (China Standard Time)

Let's test UC3 again with plain Docker deployment (should work) and with deployment on native K8s as well as through the Tango SP. @KaiHannemann Use the Tango machine that I mentioned today both for the k8s deployment and to check/debug the deployment with the Tango SP. I already uploaded NS3: https://int-sp-ath.5gtango.eu/service-management/network-services/services
I got a DNS error from Squid in both these cases, but we'll have to test more extensively.

If this is the case even though both are using the same container and have privileged mode, Manuel's suggestion was to log into the running container in K8s and try to curl from within the container. If it doesn't work, maybe the combination of K8s + VPN leads to DNS issues. Let's test and google into that direction.

Also update the wiki with any relevant info

Stefan Schneider · Answer 16 · Fri Jan 17 2020 17:19:49 GMT+0800 (China Standard Time)

Tried again with plain Docker deployment:

Running on extra VM on VirtualBox locally, everything works as expected
Also works fine on VM inside UPB's cluster
Running on machine of K8s master, even the plain Docker (no K8s, no Tango yet) deployment doesn't work and I get the DNS error! However plain curl ifconfig.me without the proxy does work.

This means it seems like this isn't a K8s isse (at least not yet), but more an issue with running the container on the specific machine which is used for K8s and Tango.

Stefan Schneider · Answer 17 · Fri Jan 17 2020 17:25:57 GMT+0800 (China Standard Time)

Also connecting to the running container from another machine in the same network to use it as a proxy doesn't work: Connection times out.

Seems like this is due to the VPN client. If starting without --privileged, the VPN client crashes and only the proxy is running. In this case, it works also remotely.
The problem may be similar to this one: https://stackoverflow.com/questions/59685231/docker-compose-ports-forwarding-not-working-if-not-on-localhost-proxy-vpn
May be hard to fix if we want to deploy the container on k8s later.
Alternative: Use one of the existing images: http://haugene.github.io/docker-transmission-openvpn/ or https://hub.docker.com/r/dceschmidt/openvpn-proxy

I still don't really get, why connecting locally to the deployed container works fine.

Simpler approach: For potential live demo, prepare container without VPN client and just proxy. Can still argue that it provides connectivity on demand and can limit traffic to certain destinations.

Stefan Schneider · Answer 18 · Fri Jan 17 2020 20:44:47 GMT+0800 (China Standard Time)

Due to the issues with the VPN described above, we drop the VPN for the live demo and just use the proxy. Hence, the proxy is deployed on demand to provide connectivity. If desired, the proxy could be configured to only allow access to pre-defined resources.

This version of NS3 now works fine also when deployed through the Tango SP on K8s.