No changes written to file

Question

No changes written to file

Panaetius opened this issue 6 years ago · comments

I'm running ksync in a DIND local cluster scenario.

I've configured everything, ran ksync init then ksync create --name coordinator -l app=mlbench,component=coordinator --context local --reload=false /home/zenon/DEV/epfl/mlbench/mlbench/coordinator/ /app/code/

(I tried with reload=true as well, but the application should reload by itself anyways)

sync watch works as well , debug output:

ksync watch --log-level debug
DEBU[0000] initializing kubernetes client context=local
DEBU[0000] kubernetes client created host="http://localhost:8080"
DEBU[0000] watching for updates ContainerName= LocalPath=/home/zenon/DEV/epfl/mlbench/mlbench/coordinator/ LocalReadOnly=false Name=coordinator Namespace=default Pod= Reload=false RemotePath=/app/code/ RemoteReadOnly=false Selector="app=mlbench,component=coordinator"
DEBU[0000] cleaning background daemon
DEBU[0000] starting syncthing args="[/home/zenon/.ksync/bin/syncthing -gui-address localhost:8384 -gui-apikey ksync -home /home/zenon/.ksync/syncthing -no-browser]" cmd=/home/zenon/.ksync/bin/syncthing
INFO[0000] listening bind=127.0.0.1 port=40322
DEBU[0000] RESTY 2018/07/06 16:14:26 ERROR [Get http://localhost:8384/rest/system/config: dial tcp [::1]:8384: connect: connection refused] Attempt [1]
DEBU[0000] new event deleted=false name=virtuous-fly-mlbench-coordinator-699747c74b-2gnmb status=Running type=ADDED
INFO[0000] new pod detected pod=virtuous-fly-mlbench-coordinator-699747c74b-2gnmb spec=coordinator
DEBU[0000] added service ID=9b01da22760a145e12ecc4d03c02d64f4fdcb0804f81c36c96cc7be10612e07d Name=mlbench-coordinator NodeName=kube-node-2 PodName=virtuous-fly-mlbench-coordinator-699747c74b-2gnmb
DEBU[0000] checking to see if radar is ready nodeName=kube-node-2
DEBU[0000] found pod name Namespace=kube-system RadarPort=40321 SyncthingAPI=8384 SyncthingListener=22000 nodeName=kube-node-2 podName=ksync-dmp8h
DEBU[0000] found pod nodeName=kube-node-2 podName=ksync-dmp8h status=Running
DEBU[0000] starting tunnel LocalPort=42425 Namespace=kube-system Out= PodName=ksync-dmp8h RemotePort=40321 url="http://localhost:8080/api/v1/namespaces/kube-system/pods/ksync-dmp8h/portforward"
DEBU[0000] tunnel running LocalPort=42425 Namespace=kube-system Out="Forwarding from 127.0.0.1:42425 -> 40321\nForwarding from [::1]:42425 -> 40321\n" PodName=ksync-dmp8h RemotePort=40321
DEBU[0000] [monitor] 16:14:26 INFO: Starting syncthing name=syncthing
DEBU[0000] [start] 16:14:26 INFO: Generating ECDSA key and certificate for syncthing... name=syncthing
DEBU[0000] [CV3OM] 16:14:26 INFO: syncthing v0.14.48 "Dysprosium Dragonfly" (go1.10.2 linux-amd64) teamcity@build.syncthing.net 2018-05-14 06:53:06 UTC name=syncthing
DEBU[0000] [CV3OM] 16:14:26 INFO: My ID: CV3OM2R-UEGMGXC-DZXVZTE-AZTPJMX-ELTLMGM-7G47SQH-LQHAYYL-FWLOPQH name=syncthing
DEBU[0000] checking to see if radar is ready nodeName=kube-node-2
DEBU[0000] found pod name Namespace=kube-system RadarPort=40321 SyncthingAPI=8384 SyncthingListener=22000 nodeName=kube-node-2 podName=ksync-dmp8h
DEBU[0000] found pod nodeName=kube-node-2 podName=ksync-dmp8h status=Running
DEBU[0000] starting tunnel LocalPort=39651 Namespace=kube-system Out= PodName=ksync-dmp8h RemotePort=8384 url="http://localhost:8080/api/v1/namespaces/kube-system/pods/ksync-dmp8h/portforward"
DEBU[0000] tunnel running LocalPort=39651 Namespace=kube-system Out="Forwarding from 127.0.0.1:39651 -> 8384\nForwarding from [::1]:39651 -> 8384\n" PodName=ksync-dmp8h RemotePort=8384
DEBU[0000] checking to see if radar is ready nodeName=kube-node-2
DEBU[0000] found pod name Namespace=kube-system RadarPort=40321 SyncthingAPI=8384 SyncthingListener=22000 nodeName=kube-node-2 podName=ksync-dmp8h
DEBU[0000] found pod nodeName=kube-node-2 podName=ksync-dmp8h status=Running
DEBU[0000] starting tunnel LocalPort=35347 Namespace=kube-system Out= PodName=ksync-dmp8h RemotePort=22000 url="http://localhost:8080/api/v1/namespaces/kube-system/pods/ksync-dmp8h/portforward"
DEBU[0000] tunnel running LocalPort=35347 Namespace=kube-system Out="Forwarding from 127.0.0.1:35347 -> 22000\nForwarding from [::1]:35347 -> 22000\n" PodName=ksync-dmp8h RemotePort=22000
DEBU[0000] RESTY 2018/07/06 16:14:26 ERROR [Get http://localhost:8384/rest/system/config: dial tcp [::1]:8384: connect: connection refused] Attempt [1]
DEBU[0001] RESTY 2018/07/06 16:14:27 ERROR [Get http://localhost:8384/rest/system/config: dial tcp [::1]:8384: connect: connection refused] Attempt [2]
DEBU[0001] [CV3OM] 16:14:27 INFO: Single thread SHA256 performance is 235 MB/s using minio/sha256-simd (174 MB/s using crypto/sha256). name=syncthing
DEBU[0001] [CV3OM] 16:14:27 INFO: Archiving a copy of old config file format at: /home/zenon/.ksync/syncthing/config.xml.v26 name=syncthing
DEBU[0001] RESTY 2018/07/06 16:14:27 ERROR [Get http://localhost:8384/rest/system/config: dial tcp [::1]:8384: connect: connection refused] Attempt [2]
DEBU[0002] [CV3OM] 16:14:28 INFO: Hashing performance is 206.12 MB/s name=syncthing
DEBU[0002] [CV3OM] 16:14:28 INFO: Updating database schema version from 0 to 2... name=syncthing
DEBU[0002] [CV3OM] 16:14:28 INFO: Updated symlink type for 0 index entries and added 0 invalid files to global list name=syncthing
DEBU[0002] [CV3OM] 16:14:28 INFO: Finished updating database schema version from 0 to 2 name=syncthing
DEBU[0002] [CV3OM] 16:14:28 INFO: Overall send rate is unlimited, receive rate is unlimited name=syncthing
DEBU[0002] [CV3OM] 16:14:28 INFO: Rate limits do not apply to LAN connections name=syncthing
DEBU[0002] [CV3OM] 16:14:28 INFO: TCP listener ([::]:22000) starting name=syncthing
DEBU[0002] [CV3OM] 16:14:28 INFO: Device CV3OM2R-UEGMGXC-DZXVZTE-AZTPJMX-ELTLMGM-7G47SQH-LQHAYYL-FWLOPQH is "ZENON-PC" at [dynamic] name=syncthing
DEBU[0002] [CV3OM] 16:14:28 INFO: Loading HTTPS certificate: open /home/zenon/.ksync/syncthing/https-cert.pem: no such file or directory name=syncthing
DEBU[0002] [CV3OM] 16:14:28 INFO: Creating new HTTPS certificate name=syncthing
DEBU[0002] RESTY 2018/07/06 16:14:28 ERROR [Get http://localhost:8384/rest/system/config: dial tcp [::1]:8384: connect: connection refused] Attempt [3]
DEBU[0002] [CV3OM] 16:14:28 INFO: GUI and API listening on 127.0.0.1:8384 name=syncthing
DEBU[0002] [CV3OM] 16:14:28 INFO: Access the GUI via the following URL: http://localhost:8384/ name=syncthing
DEBU[0003] [CV3OM] 16:14:29 INFO: Adding folder "coordinator-virtuous-fly-mlbench-coordinator-699747c74b-2gnmb" (coordinator-virtuous-fly-mlbench-coordinator-699747c74b-2gnmb) name=syncthing
DEBU[0003] [CV3OM] 16:14:29 INFO: No stored folder metadata for "coordinator-virtuous-fly-mlbench-coordinator-699747c74b-2gnmb": recalculating name=syncthing
DEBU[0003] [CV3OM] 16:14:29 INFO: Ready to synchronize "coordinator-virtuous-fly-mlbench-coordinator-699747c74b-2gnmb" (coordinator-virtuous-fly-mlbench-coordinator-699747c74b-2gnmb) (readwrite) name=syncthing
DEBU[0003] [CV3OM] 16:14:29 INFO: Completed initial scan of readwrite folder "coordinator-virtuous-fly-mlbench-coordinator-699747c74b-2gnmb" (coordinator-virtuous-fly-mlbench-coordinator-699747c74b-2gnmb) name=syncthing
INFO[0003] updating pod=virtuous-fly-mlbench-coordinator-699747c74b-2gnmb spec=coordinator
INFO[0003] updating pod=virtuous-fly-mlbench-coordinator-699747c74b-2gnmb spec=coordinator
WARN[0003] {"alloc":9305640,"connectionServiceStatus":{"tcp://0.0.0.0:22000":{"lanAddresses":["tcp://0.0.0.0:22000"],"wanAddresses":["tcp://0.0.0.0:22000"]}},"cpuPercent":0,"goroutines":42,"myID":"CV3OM2R-UEGMGXC-DZXVZTE-AZTPJMX-ELTLMGM-7G47SQH-LQHAYYL-FWLOPQH","pathSeparator":"/","startTime":"2018-07-06T16:14:26.610132871+02:00","sys":40863992,"tilde":"/home/zenon","uptime":2,"urVersionMax":3}
INFO[0003] syncthing listening port=8384 syncthing=localhost
WARN[0003] {"alloc":9332440,"connectionServiceStatus":{"tcp://0.0.0.0:22000":{"lanAddresses":["tcp://0.0.0.0:22000"],"wanAddresses":["tcp://0.0.0.0:22000"]}},"cpuPercent":0,"goroutines":42,"myID":"CV3OM2R-UEGMGXC-DZXVZTE-AZTPJMX-ELTLMGM-7G47SQH-LQHAYYL-FWLOPQH","pathSeparator":"/","startTime":"2018-07-06T16:14:26.610132871+02:00","sys":40863992,"tilde":"/home/zenon","uptime":2,"urVersionMax":3}
DEBU[0003] restarting local syncthing
DEBU[0003] [CV3OM] 16:14:29 INFO: Restarting name=syncthing
INFO[0003] finished unary call with code OK grpc.code=OK grpc.method=RestartSyncthing grpc.service=proto.ksync.Ksync grpc.start_time="2018-07-06T16:14:29+02:00" grpc.time_ns=718597 peer.address="127.0.0.1:40044" span.kind=server system=grpc
DEBU[0003] [CV3OM] 16:14:29 INFO: Exiting name=syncthing
INFO[0003] folder sync running pod=virtuous-fly-mlbench-coordinator-699747c74b-2gnmb spec=coordinator
DEBU[0003] RESTY 2018/07/06 16:14:29 ERROR [unexpected EOF] Attempt [1]
DEBU[0003] [monitor] 16:14:29 INFO: Syncthing exited: exit status 3 name=syncthing
DEBU[0004] [monitor] 16:14:30 INFO: Starting syncthing name=syncthing
DEBU[0004] RESTY 2018/07/06 16:14:30 ERROR [Get http://localhost:8384/rest/events?since=14: dial tcp [::1]:8384: connect: connection refused] Attempt [2]
DEBU[0004] [CV3OM] 16:14:30 INFO: syncthing v0.14.48 "Dysprosium Dragonfly" (go1.10.2 linux-amd64) teamcity@build.syncthing.net 2018-05-14 06:53:06 UTC name=syncthing
DEBU[0004] [CV3OM] 16:14:30 INFO: My ID: CV3OM2R-UEGMGXC-DZXVZTE-AZTPJMX-ELTLMGM-7G47SQH-LQHAYYL-FWLOPQH name=syncthing
DEBU[0005] [CV3OM] 16:14:31 INFO: Single thread SHA256 performance is 235 MB/s using minio/sha256-simd (177 MB/s using crypto/sha256). name=syncthing
DEBU[0006] RESTY 2018/07/06 16:14:32 ERROR [Get http://localhost:8384/rest/events?since=14: dial tcp [::1]:8384: connect: connection refused] Attempt [3]
DEBU[0006] [CV3OM] 16:14:32 INFO: Hashing performance is 198.40 MB/s name=syncthing
DEBU[0006] [CV3OM] 16:14:32 INFO: Ready to synchronize "coordinator-virtuous-fly-mlbench-coordinator-699747c74b-2gnmb" (coordinator-virtuous-fly-mlbench-coordinator-699747c74b-2gnmb) (readwrite) name=syncthing
DEBU[0006] [CV3OM] 16:14:32 INFO: Overall send rate is unlimited, receive rate is unlimited name=syncthing
DEBU[0006] [CV3OM] 16:14:32 INFO: Rate limits do not apply to LAN connections name=syncthing
DEBU[0006] [CV3OM] 16:14:32 INFO: TCP listener ([::]:22000) starting name=syncthing
DEBU[0006] [CV3OM] 16:14:32 INFO: Device CV3OM2R-UEGMGXC-DZXVZTE-AZTPJMX-ELTLMGM-7G47SQH-LQHAYYL-FWLOPQH is "ZENON-PC" at [dynamic] name=syncthing
DEBU[0006] [CV3OM] 16:14:32 INFO: Device HCF2NSO-ALBGN5R-4E7LGV2-CNNTDYA-EKWTBCK-44IA6QK-2IAXMHI-LK2KVQL is "virtuous-fly-mlbench-coordinator-699747c74b-2gnmb" at [tcp://127.0.0.1:35347] name=syncthing
DEBU[0006] [CV3OM] 16:14:32 INFO: Completed initial scan of readwrite folder "coordinator-virtuous-fly-mlbench-coordinator-699747c74b-2gnmb" (coordinator-virtuous-fly-mlbench-coordinator-699747c74b-2gnmb) name=syncthing
DEBU[0006] [CV3OM] 16:14:32 INFO: GUI and API listening on 127.0.0.1:8384 name=syncthing
DEBU[0006] [CV3OM] 16:14:32 INFO: Access the GUI via the following URL: http://localhost:8384/ name=syncthing
DEBU[0006] [CV3OM] 16:14:32 INFO: Established secure connection to HCF2NSO-ALBGN5R-4E7LGV2-CNNTDYA-EKWTBCK-44IA6QK-2IAXMHI-LK2KVQL at 127.0.0.1:36380-127.0.0.1:35347/tcp-client (TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305) name=syncthing
DEBU[0006] [CV3OM] 16:14:32 INFO: Device HCF2NSO-ALBGN5R-4E7LGV2-CNNTDYA-EKWTBCK-44IA6QK-2IAXMHI-LK2KVQL client is "syncthing v0.14.48" named "ksync-dmp8h" at 127.0.0.1:36380-127.0.0.1:35347/tcp-client name=syncthing
DEBU[0006] [CV3OM] 16:14:32 INFO: Device HCF2NSO-ALBGN5R-4E7LGV2-CNNTDYA-EKWTBCK-44IA6QK-2IAXMHI-LK2KVQL folder "coordinator-virtuous-fly-mlbench-coordinator-699747c74b-2gnmb" (coordinator-virtuous-fly-mlbench-coordinator-699747c74b-2gnmb) has a new index ID (0x84FED7B664DF945D) name=syncthing
INFO[0007] updating pod=virtuous-fly-mlbench-coordinator-699747c74b-2gnmb spec=coordinator
INFO[0007] update complete pod=virtuous-fly-mlbench-coordinator-699747c74b-2gnmb spec=coordinator
INFO[0018] updating pod=virtuous-fly-mlbench-coordinator-699747c74b-2gnmb spec=coordinator
INFO[0018] update complete pod=virtuous-fly-mlbench-coordinator-699747c74b-2gnmb spec=coordinator
INFO[0018] updating pod=virtuous-fly-mlbench-coordinator-699747c74b-2gnmb spec=coordinator
INFO[0018] update complete pod=virtuous-fly-mlbench-coordinator-699747c74b-2gnmb spec=coordinator

so everything seems to work ok.

ksync get outputs:
NAME LOCAL REMOTE STATUS POD CONTAINER +-------------+---------------------+------------+----------+---------------------------------------------------+-----------+ coordinator mlbench/coordinator /app/code/ watching virtuous-fly-mlbench-coordinator-699747c74b-2gnmb

but if I connect to the pod and look at the content of a file I changed locally, it still has the original content. I.e. changes don't take effect even though ksync registers the change and writes "update complete"

Changing the remote file (kubectl exec -it virtuous-fly-mlbench-coordinator-699747c74b-2gnmb -- /bin/sh and editing it) doesn't get registered in ksync watch or syncthing.

Since everything seems to be working, except it's not working, I'm a bit at a loss and any help would be appreciated.

Tim · Answer 1 · Tue Jul 10 2018 05:03:27 GMT+0800 (China Standard Time)

@Panaetius it can be confusing to debug at this point, so let me clear a few pieces up.

I'm not entirely sure that the -l some=something,somethingelse=somewhere (i.e. with the comma separator) works in ksync, but admittedly I haven't tried it. Usually I specify multiple selectors with multiple -l flags. I only mention it because I suspect the problem may be that the entire selector comment was ingested and processed as a single string, meaning nothing would match. Try recreating your ksync create and see what happens.

The "pod reload" triggering/complete messages are actually local, generated from events emitted by the remote syncthing event stream. Since the selectors aren't applied when looking for a pod (after it is first found) that would explain why the reloads were triggered when things changed locally, but the file was not changed.

As a side note, reload just determines whether the remote container is rolled after any file change.

See if that change fixes things for you and we can work from there. If not, you might also try specifying a pod without the selectors and see if that works (as a test).

Ralf Grubenmann · Answer 2 · Tue Jul 10 2018 19:12:16 GMT+0800 (China Standard Time)

@timfallmk Regarding the -l some=something,somethingelse=somewhere, that indeed does not work, it just picks the first selector specified and ignores the rest. I already tried it with multiple -l flags yesterday, that didn't work either.

I just tried
ksync create -p jumpy-rodent-mlbench-coordinator-6bc59dd765-5hglw --local-read-only --name coordinator --context=local /home/zenon/DEV/epfl/mlbench/mlbench/coordinator/ /app/code/

ksync create -p jumpy-rodent-mlbench-coordinator-6bc59dd765-5hglw --local-read-only --name coordinator /home/zenon/DEV/epfl/mlbench/mlbench/coordinator/ /app/code/

ksync create -p jumpy-rodent-mlbench-coordinator-6bc59dd765-5hglw --local-read-only --name coordinator --context=local --reload=false /home/zenon/DEV/epfl/mlbench/mlbench/coordinator/ /app/code/

with the same results (not working).

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
jumpy-rodent-mlbench-coordinator-6bc59dd765-5hglw 1/1 Running 0 20h
jumpy-rodent-mlbench-experiment-94ff54498-rjjwt 1/1 Running 0 20h
jumpy-rodent-mlbench-experiment-94ff54498-t8pcw 1/1 Running 0 20h

I checked all 3 containers to see if there's files there, just to make sure it didn't pick the wrong folder. None of them contain my changed file (it's called views.py and I did find / -iname "views.py" on all three containers. Only the one on coordinator (the one I'm trying to use ksync with) exists and that one still has the original content.

Tim · Answer 3 · Wed Jul 11 2018 03:11:15 GMT+0800 (China Standard Time)

If you visit the syncthing web UI (I believe it defaults to 8384), does it show the correct target destinations?

Ralf Grubenmann · Answer 4 · Wed Jul 11 2018 04:40:06 GMT+0800 (China Standard Time)

It looks fine, the paths are correct and it lists modifications to the file correctly.

Are there any additional logs I could provide to help solve the issue?

And thank you for the support!

Tim · Answer 5 · Wed Jul 11 2018 06:51:15 GMT+0800 (China Standard Time)

That's very odd. The only thing I can think of, other than the above, is a file permissions issue. Even that should trigger an error though.

@grampelberg ideas?

Thomas Rampelberg · Answer 6 · Wed Jul 11 2018 07:14:10 GMT+0800 (China Standard Time)

ksync makes some assumptions about how the FS is laid out under the covers. What FS driver are you using and where is it putting files? I'm wondering if DIND is the issue.

Ralf Grubenmann · Answer 7 · Wed Jul 11 2018 16:13:10 GMT+0800 (China Standard Time)

@grampelberg I use https://github.com/kubernetes-sigs/kubeadm-dind-cluster if that helps, v1.10.

It's all Overlay2 and stored in /var/lib/docker and /dind/docker on the host and in the kubernetes nodes, respectively.

On the host:

$ docker info
Containers: 52
Running: 4
Paused: 0
Stopped: 48
Images: 188
Server Version: 18.05.0-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 4.17.3-1-ARCH
Operating System: Arch Linux
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 31.36GiB
Name: ZENON-PC
ID: SCNS:IMKI:NVHE:GUYO:VWEH:QUHC:FQFR:N3NJ:AV7Z:WF2A:S2HW:LXQS
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
localhost:5000
127.0.0.0/8
Live Restore Enabled: false

Inside the kubernetes master:

$ docker exec -it ec7e0 /bin/sh
# docker info
Containers: 19
Running: 11
Paused: 0
Stopped: 8
Images: 5
Server Version: 17.03.2-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 4.17.3-1-ARCH
Operating System: Debian GNU/Linux 9 (stretch) (containerized)
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 31.36 GiB
Name: kube-master
ID: C4TN:JNI7:SJIY:PC6Y:TWNT:GS3L:UW5F:TDBC:EQ5V:TOF6:CMLF:VIMV
Docker Root Dir: /dind/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Inside a kubernetes node:

$ docker exec -it 380d08 /bin/sh
# docker info
Containers: 10
Running: 10
Paused: 0
Stopped: 0
Images: 7
Server Version: 17.03.2-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 4.17.3-1-ARCH
Operating System: Debian GNU/Linux 9 (stretch) (containerized)
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 31.36 GiB
Name: kube-node-1
ID: CR2C:LL2W:JZGT:XZZE:AH5Z:5MIP:OYWC:GUOM:7YKE:4BNF:LJLW:J7NC
Docker Root Dir: /dind/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Thomas Rampelberg · Answer 8 · Wed Jul 11 2018 22:57:50 GMT+0800 (China Standard Time)

Yup, that's the problem. We're only mounting /var/lib/docker in (https://github.com/vapor-ware/ksync/blob/master/pkg/ksync/cluster/daemon_set.go#L78).

The solution to this should be adding a configuration option to install that lets you configure that path. @timfallmk want to take a look?

Tim · Answer 9 · Thu Jul 12 2018 03:04:07 GMT+0800 (China Standard Time)

It would probably make more sense to detect what the docker daemon is pointed at, but I'll look into it.

Ralf Grubenmann · Answer 10 · Thu Jul 12 2018 04:31:27 GMT+0800 (China Standard Time)

Great, looking forward to it being configurable. Thanks a lot for the quick responses

For now I'll try just getting the dind cluster to use the regular path, I think changing the /dind/docker path on line 89 of https://github.com/kubernetes-sigs/kubeadm-dind-cluster/blob/5e48a6377148b1e0d1b3d8c4609a140c7b00b57b/image/rundocker should do. Haven't had a chance to test it yet. I'll report back once I tried it, in case there's some other issues with kubeadm DIND.

Tim · Answer 11 · Thu Jul 12 2018 04:39:15 GMT+0800 (China Standard Time)

@Panaetius It might take some time before I get to it. In the meantime, changing the configured volume mount @grampelberg listed and recompiling should let you use it.

Thomas Rampelberg · Answer 12 · Thu Jul 12 2018 04:57:11 GMT+0800 (China Standard Time)

Good call @timfallmk, that's just a daemonset. Just use kubectl to edit the daemonset to the right path and it should (tm) work (hopefully).

Tim · Answer 13 · Thu Jul 12 2018 04:57:48 GMT+0800 (China Standard Time)

That would *also* work.

Ralf Grubenmann · Answer 14 · Thu Jul 12 2018 17:29:44 GMT+0800 (China Standard Time)

@timfallmk

I just tried again with the newest version and sadly it still doesn't seem to work.

I connected to the syncthing service running on the node and enabled (most) debug log settings. The path is now correct. E.g.:

http: GET "/rest/system/log?since=2018-07-12T09%3A20%3A28.292799137Z": status 200, 716 bytes in 0.09 ms
2018-07-12 09:20:32 read IndexUpdate message
2018-07-12 09:20:32 queueing IndexUpdate(2AMD5RW-GVNA4P3-CGBIJH2-G6OECKY-73OEILD-LDEN5TX-PDJS7AQ-2DBL7Q2, concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx, 1 files)
2018-07-12 09:20:32 Index update (in): 2AMD5RW-GVNA4P3-CGBIJH2-G6OECKY-73OEILD-LDEN5TX-PDJS7AQ-2DBL7Q2 / "concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx": 1 files
2018-07-12 09:20:32 concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx Update(2AMD5RW-GVNA4P3-CGBIJH2-G6OECKY-73OEILD-LDEN5TX-PDJS7AQ-2DBL7Q2, [1])
2018-07-12 09:20:32 insert; folder="concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx" device=2AMD5RW-GVNA4P3-CGBIJH2-G6OECKY-73OEILD-LDEN5TX-PDJS7AQ-2DBL7Q2 File{Name:"main/views.py", Sequence:57, Permissions:0644, ModTime:2018-07-12 09:20:31.590656547 +0000 UTC, Version:{[{J7HTQSK 9} {2AMD5RW 6}]}, Length:900, Deleted:false, Invalid:false, NoPermissions:false, BlockSize:131072, Blocks:[Block{0/900/1563634915/51e3f54be12434506e24c60afc79af3f73349cb2e3ed24131c08cffb9fe22e54}]}
2018-07-12 09:20:32 update global; folder="concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx" device=2AMD5RW-GVNA4P3-CGBIJH2-G6OECKY-73OEILD-LDEN5TX-PDJS7AQ-2DBL7Q2 file="main/views.py" version={[{J7HTQSK 9} {2AMD5RW 6}]} invalid=false
2018-07-12 09:20:32 new global after update: {{{[{J7HTQSK 9} {2AMD5RW 6}]}, 2AMD5RW-GVNA4P3-CGBIJH2-G6OECKY-73OEILD-LDEN5TX-PDJS7AQ-2DBL7Q2}, {{[{J7HTQSK 9} {2AMD5RW 5}]}, 7777777-777777N-7777777-777777N-7777777-777777N-7777777-77777Q4}}
2018-07-12 09:20:32 log 87 RemoteIndexUpdated map[items:1 version:57 device:2AMD5RW-GVNA4P3-CGBIJH2-G6OECKY-73OEILD-LDEN5TX-PDJS7AQ-2DBL7Q2 folder:concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx]
2018-07-12 09:20:32 Device 2AMD5RW-GVNA4P3-CGBIJH2-G6OECKY-73OEILD-LDEN5TX-PDJS7AQ-2DBL7Q2 sent an index update for "concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx" with 1 items
2018-07-12 09:20:32 http: GET "/rest/events?since=81": status 200, 282 bytes in 58212.87 ms
2018-07-12 09:20:32 http: GET "/rest/events?since=81": status 200, 282 bytes in 31767.09 ms
2018-07-12 09:20:32 size.go:85 basic /var/syncthing/config/ Usage . {90801340416 203373813760}
2018-07-12 09:20:32 sendReceiveFolder/concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx@0xc4200b9680 pulling (ignoresChanged=false)
2018-07-12 09:20:32 log 88 StateChanged map[to:syncing from:idle duration:58.237464443 folder:concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx]
2018-07-12 09:20:32 Folder "concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx" is now syncing
2018-07-12 09:20:32 sendReceiveFolder/concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx@0xc4200b9680 copiers: 2 pullerPendingKiB: 32768
2018-07-12 09:20:32 concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx WithNeed(7777777-777777N-7777777-777777N-7777777-777777N-7777777-77777Q4)
2018-07-12 09:20:32 need folder="concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx" device=7777777-777777N-7777777-777777N-7777777-777777N-7777777-77777Q4 name="main/views.py" need=true have=true invalid=false haveV={[{J7HTQSK 9} {2AMD5RW 5}]} globalV={[{J7HTQSK 9} {2AMD5RW 6}]} globalDev=2AMD5RW-GVNA4P3-CGBIJH2-G6OECKY-73OEILD-LDEN5TX-PDJS7AQ-2DBL7Q2
2018-07-12 09:20:32 stats.DeviceStatisticsReference.WasSeen: 2AMD5RW-GVNA4P3-CGBIJH2-G6OECKY-73OEILD-LDEN5TX-PDJS7AQ-2DBL7Q2
2018-07-12 09:20:32 open: open /dind/docker/overlay2/2c73c728b130d6eb4d923838a400ad6b51df9947946a0a7cd4c0e882efcd05c3/merged/app/code/main/.syncthing.views.py.tmp: no such file or directory
2018-07-12 09:20:32 log 89 ItemStarted map[type:file action:update folder:concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx item:main/views.py]
2018-07-12 09:20:32 Started syncing "concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx" / "main/views.py" (update file)
2018-07-12 09:20:32 sendReceiveFolder/concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx@0xc4200b9680 need file main/views.py; copy 1, reused 0
2018-07-12 09:20:32 progress emitter: registering concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx main/views.py
2018-07-12 09:20:32 basic /dind/docker/overlay2/2c73c728b130d6eb4d923838a400ad6b51df9947946a0a7cd4c0e882efcd05c3/merged/app/code/ Watch: Ignoring main/.syncthing.views.py.tmp
2018-07-12 09:20:32 basic /dind/docker/overlay2/2c73c728b130d6eb4d923838a400ad6b51df9947946a0a7cd4c0e882efcd05c3/merged/app/code/ Watch: Ignoring main/.syncthing.views.py.tmp
2018-07-12 09:20:32 basic /dind/docker/overlay2/2c73c728b130d6eb4d923838a400ad6b51df9947946a0a7cd4c0e882efcd05c3/merged/app/code/ Watch: Ignoring main/.syncthing.views.py.tmp
2018-07-12 09:20:32 basic /dind/docker/overlay2/2c73c728b130d6eb4d923838a400ad6b51df9947946a0a7cd4c0e882efcd05c3/merged/app/code/ Watch: Ignoring main/.syncthing.views.py.tmp
2018-07-12 09:20:32 sharedPullerState concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx main/views.py pullNeeded start -> 1
2018-07-12 09:20:32 model@0xc420096fc0 REQ(out): 2AMD5RW-GVNA4P3-CGBIJH2-G6OECKY-73OEILD-LDEN5TX-PDJS7AQ-2DBL7Q2: "concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx" / "main/views.py" o=0 s=900 h=51e3f54be12434506e24c60afc79af3f73349cb2e3ed24131c08cffb9fe22e54 wh=5d332ce3 ft=false
2018-07-12 09:20:32 wrote 148 bytes on the wire (2 bytes length, 4 bytes header, 4 bytes message length, 138 bytes message (132 uncompressed)), err=
2018-07-12 09:20:32 read Response message
2018-07-12 09:20:32 sharedPullerState concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx main/views.py pullNeeded done -> 0
2018-07-12 09:20:32 basic /dind/docker/overlay2/2c73c728b130d6eb4d923838a400ad6b51df9947946a0a7cd4c0e882efcd05c3/merged/app/code/ Watch: Ignoring main/.syncthing.views.py.tmp
2018-07-12 09:20:32 basic /dind/docker/overlay2/2c73c728b130d6eb4d923838a400ad6b51df9947946a0a7cd4c0e882efcd05c3/merged/app/code/ Watch: Ignoring main/.syncthing.views.py.tmp
2018-07-12 09:20:32 http: GET "/rest/events?since=82": status 200, 461 bytes in 0.15 ms
2018-07-12 09:20:32 http: GET "/assets/img/favicon-sync.png": status 304, 0 bytes in 0.05 ms
2018-07-12 09:20:32 sendReceiveFolder/concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx@0xc4200b9680 closing main/views.py
2018-07-12 09:20:32 basic /dind/docker/overlay2/2c73c728b130d6eb4d923838a400ad6b51df9947946a0a7cd4c0e882efcd05c3/merged/app/code/ Watch: Ignoring main/.syncthing.views.py.tmp
2018-07-12 09:20:32 basic /dind/docker/overlay2/2c73c728b130d6eb4d923838a400ad6b51df9947946a0a7cd4c0e882efcd05c3/merged/app/code/ Watch: Ignoring main/.syncthing.views.py.tmp
2018-07-12 09:20:32 basic /dind/docker/overlay2/2c73c728b130d6eb4d923838a400ad6b51df9947946a0a7cd4c0e882efcd05c3/merged/app/code/ Watch: Sending main/views.py non-remove
2018-07-12 09:20:32 basic /dind/docker/overlay2/2c73c728b130d6eb4d923838a400ad6b51df9947946a0a7cd4c0e882efcd05c3/merged/app/code/ Watch: Ignoring main/.syncthing.views.py.tmp
2018-07-12 09:20:32 basic /dind/docker/overlay2/2c73c728b130d6eb4d923838a400ad6b51df9947946a0a7cd4c0e882efcd05c3/merged/app/code/ Watch: Sending main/views.py non-remove
2018-07-12 09:20:32 basic /dind/docker/overlay2/2c73c728b130d6eb4d923838a400ad6b51df9947946a0a7cd4c0e882efcd05c3/merged/app/code/ Watch: Ignoring main/.syncthing.views.py.tmp
2018-07-12 09:20:32 log 90 ItemFinished map[action:update folder:concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx item:main/views.py error: type:file]
2018-07-12 09:20:32 progress emitter: deregistering concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx main/views.py
2018-07-12 09:20:32 aggregator/"coordinator-s-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx" (coordinator-s-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx): Skipping path we modified: main/views.py
2018-07-12 09:20:32 aggregator/"coordinator-s-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx" (coordinator-s-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx): Skipping path we modified: main/views.py
2018-07-12 09:20:32 Finished syncing "concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx" / "main/views.py" (update file): Success
2018-07-12 09:20:32 http: GET "/rest/events?since=84": status 200, 246 bytes in 45.53 ms
2018-07-12 09:20:32 aggregator/"concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx" (concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx): Skipping path we modified: main/views.py
2018-07-12 09:20:32 aggregator/"concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx" (concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx): Creating eventDir at: main
2018-07-12 09:20:32 aggregator/"concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx" (concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx): Tracking (type non-remove): main/views.py
2018-07-12 09:20:32 aggregator/"concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx" (concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx): Resetting notifyTimer to 1s
2018-07-12 09:20:32 basic /dind/docker/overlay2/2c73c728b130d6eb4d923838a400ad6b51df9947946a0a7cd4c0e882efcd05c3/merged/app/code/ Watch: Sending main/views.py non-remove
2018-07-12 09:20:32 basic /dind/docker/overlay2/2c73c728b130d6eb4d923838a400ad6b51df9947946a0a7cd4c0e882efcd05c3/merged/app/code/ Watch: Sending main/views.py non-remove
2018-07-12 09:20:32 concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx Update(7777777-777777N-7777777-777777N-7777777-777777N-7777777-77777Q4, [1])
2018-07-12 09:20:32 removing sequence; folder="concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx" sequence=39 main/views.py
2018-07-12 09:20:32 adding sequence; folder="concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx" sequence=40 main/views.py
2018-07-12 09:20:32 insert; folder="concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx" device=7777777-777777N-7777777-777777N-7777777-777777N-7777777-77777Q4 File{Name:"main/views.py", Sequence:40, Permissions:0644, ModTime:2018-07-12 09:20:31.590656547 +0000 UTC, Version:{[{J7HTQSK 9} {2AMD5RW 6}]}, Length:900, Deleted:false, Invalid:false, NoPermissions:false, BlockSize:131072, Blocks:[Block{0/900/1563634915/51e3f54be12434506e24c60afc79af3f73349cb2e3ed24131c08cffb9fe22e54}]}
2018-07-12 09:20:32 update global; folder="concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx" device=7777777-777777N-7777777-777777N-7777777-777777N-7777777-77777Q4 file="main/views.py" version={[{J7HTQSK 9} {2AMD5RW 6}]} invalid=false
2018-07-12 09:20:32 new global after update: {{{[{J7HTQSK 9} {2AMD5RW 6}]}, 7777777-777777N-7777777-777777N-7777777-777777N-7777777-77777Q4}, {{[{J7HTQSK 9} {2AMD5RW 6}]}, 2AMD5RW-GVNA4P3-CGBIJH2-G6OECKY-73OEILD-LDEN5TX-PDJS7AQ-2DBL7Q2}}
2018-07-12 09:20:32 log 91 LocalIndexUpdated map[version:40 folder:concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx items:1 filenames:[main/views.py]]
2018-07-12 09:20:32 log 92 RemoteChangeDetected map[path:main/views.py modifiedBy:2AMD5RW folder:concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx folderID:concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx label:concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx action:modified type:file]
2018-07-12 09:20:32 stats.FolderStatisticsReference.ReceivedFile: concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx main/views.py
2018-07-12 09:20:32 http: GET "/rest/events?since=85": status 200, 234 bytes in 36.62 ms
2018-07-12 09:20:32 concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx WithHaveSequence(40)
2018-07-12 09:20:32 sendReceiveFolder/concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx@0xc4200b9680 changed 1
2018-07-12 09:20:32 sendReceiveFolder/concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx@0xc4200b9680 copiers: 2 pullerPendingKiB: 32768
2018-07-12 09:20:32 Remote change detected in folder "concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx": modified file main/views.py
2018-07-12 09:20:32 Sending indexes for concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx to 2AMD5RW-GVNA4P3-CGBIJH2-G6OECKY-73OEILD-LDEN5TX-PDJS7AQ-2DBL7Q2 at 127.0.0.1:22000-127.0.0.1:36182/tcp-server: 1 files (<127 bytes) (last batch)
2018-07-12 09:20:32 concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx WithNeed(7777777-777777N-7777777-777777N-7777777-777777N-7777777-77777Q4)
2018-07-12 09:20:32 wrote 211 bytes on the wire (2 bytes length, 4 bytes header, 4 bytes message length, 201 bytes message (201 uncompressed)), err=
2018-07-12 09:20:32 sendReceiveFolder/concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx@0xc4200b9680 changed 0
2018-07-12 09:20:32 log 93 StateChanged map[folder:concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx to:idle from:syncing duration:0.103977409]
2018-07-12 09:20:32 Folder "concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx" is now idle
2018-07-12 09:20:32 concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx WithNeedTruncated(7777777-777777N-7777777-777777N-7777777-777777N-7777777-77777Q4)
2018-07-12 09:20:32 progress emitter: bytes completed for concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx: 0
2018-07-12 09:20:32 model@0xc420096fc0 NeedSize("concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx"): {0 0 0 0 0 0 []}
2018-07-12 09:20:32 log 94 FolderSummary map[summary:map[localDirectories:5 localDeleted:0 localBytes:143268 needDirectories:0 needDeletes:0 needBytes:0 invalid: globalSymlinks:0 globalDeleted:0 needFiles:0 inSyncBytes:143268 stateChanged:2018-07-12 09:20:32.701780007 +0000 UTC m=+202.825836477 version:97 pullErrors:0 globalDirectories:5 needSymlinks:0 localFiles:21 localSymlinks:0 inSyncFiles:21 state:idle sequence:97 ignorePatterns:false globalFiles:21 globalBytes:143268] folder:concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx]
2018-07-12 09:20:32 stats.DeviceStatisticsReference.WasSeen: 2AMD5RW-GVNA4P3-CGBIJH2-G6OECKY-73OEILD-LDEN5TX-PDJS7AQ-2DBL7Q2
2018-07-12 09:20:32 Summary for folder "concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx" is map[localDirectories:5 inSyncBytes:143268 globalFiles:21 globalSymlinks:0 globalDirectories:5 localDeleted:0 pullErrors:0 needDirectories:0 globalDeleted:0 needFiles:0 localSymlinks:0 inSyncFiles:21 sequence:97 localBytes:143268 needDeletes:0 needBytes:0 version:97 localFiles:21 needSymlinks:0 globalBytes:143268 state:idle]
2018-07-12 09:20:32 concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx WithNeedTruncated(2AMD5RW-GVNA4P3-CGBIJH2-G6OECKY-73OEILD-LDEN5TX-PDJS7AQ-2DBL7Q2)
2018-07-12 09:20:32 model@0xc420096fc0 Completion(2AMD5RW-GVNA4P3-CGBIJH2-G6OECKY-73OEILD-LDEN5TX-PDJS7AQ-2DBL7Q2, "concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx"): 100.000000 (0 / 143268 = 0.000000)
2018-07-12 09:20:32 log 95 FolderCompletion map[device:2AMD5RW-GVNA4P3-CGBIJH2-G6OECKY-73OEILD-LDEN5TX-PDJS7AQ-2DBL7Q2 completion:100 needBytes:0 needItems:0 globalBytes:143268 needDeletes:0 folder:concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx]
2018-07-12 09:20:32 Completion for folder "concrete-termite-eponymous-turtle-mlbench-coordinator-555447fdc7-q2ncx" on device 2AMD5RW-GVNA4P3-CGBIJH2-G6OECKY-73OEILD-LDEN5TX-PDJS7AQ-2DBL7Q2 is 100%

We can see it saying that it's updating the file /dind/docker/overlay2/2c73c728b130d6eb4d923838a400ad6b51df9947946a0a7cd4c0e882efcd05c3/merged/app/code/main/views.py so the new docker path of "/dind/docker" is working correctly.

But if I connect to the kube node that the pod is running on, while the path is correct, the file is still unchanged (last modified yesterday, content is unchanged as well):

ls -alh /dind/docker/overlay2/2c73c728b130d6eb4d923838a400ad6b51df9947946a0a7cd4c0e882efcd05c3/merged/app/code/main/
total 48K
drwxr-xr-x 1 root root 4.0K Jul 11 10:45 .
drwxr-xr-x 1 root root 4.0K Jul 11 20:11 ..
-rw-r--r-- 1 root root 0 Jul 6 12:02 init.py
drwxr-xr-x 1 root root 4.0K Jul 12 08:37 pycache
-rw-r--r-- 1 root root 63 Jul 6 12:02 admin.py
-rw-r--r-- 1 root root 83 Jul 6 12:02 apps.py
drwxr-xr-x 2 root root 4.0K Jul 6 12:02 migrations
-rw-r--r-- 1 root root 57 Jul 6 12:02 models.py
-rw-r--r-- 1 root root 60 Jul 6 12:02 tests.py
-rw-r--r-- 1 root root 169 Jul 11 10:45 urls.py
-rwxrwxrwx 1 root root 795 Jul 11 20:11 views.py

I can edit the file manually in the shell on the node, which does change the application code, i.e. the change persists on the node.

Tim · Answer 15 · Tue Jul 17 2018 01:58:14 GMT+0800 (China Standard Time)

This has to be something with how they're handling volume mounts. What happens if you roll the container after making a change (and seeing that change go through in the logs)?

Ralf Grubenmann · Answer 16 · Tue Jul 17 2018 03:13:16 GMT+0800 (China Standard Time)

What happens if you roll the container

Do you mean rollback? Or what do you mean with "roll"?

I can try it first thing tomorrow morning.

Tim · Answer 17 · Tue Jul 17 2018 03:14:13 GMT+0800 (China Standard Time)

Sorry, "restart". It's possible the filesystem changes are not being picked up after the file is changed on disk. Other than that, I'm kinda out of ideas.

Ralf Grubenmann · Answer 18 · Tue Jul 17 2018 16:12:22 GMT+0800 (China Standard Time)

Nothing in particular happens when I restart the container.

But I did some more digging and think I figured it out (well, the problem, not a solution).

I just changed the code file locally with one change (Adding test_source), waited for it to propagate, then connected to a shell inside the container and changed the same file, but with a different string (Test2), then I just ran grep for those strings in the /var/lib/docker/ folder of my host system.

Results:

[zenon@ZENON-PC docker]$ cd /var/lib/docker/ [zenon@ZENON-PC docker]$ sudo grep -r --include views.py 'Welcome!test_source!' overlay2/18e66711fe3374c4a29dcc1564692ae28bfb47e4daed4288714a39b350fab71b/diff/var/lib/docker/overlay2/e700956c8a5208553f4c03555f240cf992910f684bc063ac65333e6c3dbf48e6/merged/app/code/main/views.py: return HttpResponse("Welcome!test_source!") overlay2/18e66711fe3374c4a29dcc1564692ae28bfb47e4daed4288714a39b350fab71b/merged/var/lib/docker/overlay2/e700956c8a5208553f4c03555f240cf992910f684bc063ac65333e6c3dbf48e6/merged/app/code/main/views.py: return HttpResponse("Welcome!test_source!") [zenon@ZENON-PC docker]$ sudo grep -r --include views.py 'Welcome!Test2' volumes/kubeadm-dind-kube-node-1/_data/docker/overlay2/e700956c8a5208553f4c03555f240cf992910f684bc063ac65333e6c3dbf48e6/diff/app/code/main/views.py: return HttpResponse("Welcome!Test2")

So syncthing changes the respective entry in the overlay2 folder of the docker contents in the host overlay 2 folder, whereas changing the content directly changes the content of the overlay2 folder in the kubeadm-dind-kube-node-1 volume.

Hope this helps solve the problem.

Tim · Answer 19 · Fri Jul 20 2018 03:11:52 GMT+0800 (China Standard Time)

In that case it seems like something else is using the default graph driver path. I'll root around and see if I can find it, unless @grampelberg knows already.

abecruzs · Answer 20 · Fri Jul 26 2019 21:12:05 GMT+0800 (China Standard Time)

Hello. I'm having this exact issue. Local files trigger "reloads" but nothing is change in the remote. And if I modify a file in the remote, that does not trigger nothing and I don't get the file locally.

I have a default installation of kubernetes and my "poc" is using nginx with a simple html website.