Spin's KV storage doesn't run on K8s (Docker Desktop)
felipecruz91 opened this issue · comments
Hi,
I'm trying to run the new key-value storage example that comes with Spin v0.9.0 from this article in the K8s cluster provided by Docker Desktop.
To configure the K8s cluster to support running WASM workloads, I use a modified version of the daemonset image defined in this approach. Instead of kwasm/kwasm-node-installer, I use felipecruz/kwasm-node-installer containing Spin v0.9.0 as per:
After deploying the daemonset and the key-value Go deployment resources to the K8s cluster (see /deploy folder to find the manifests in this repo), the Pod's status is RunContainerError.
Kubelet logs
kubelet Error: failed to start container "tinygo-key-value-spin": Error response from daemon: Cannot read manifest file from │ │ "/var/run/desktop-containerd/daemon/io.containerd.runtime.v2.task/moby/tinygo-key-value-spin/rootfs/spin.toml": unknown
Containerd logs from Docker Desktop (containerd.log)
[INFO] server listen started
[INFO] server started
[INFO] Shim successfully started, waiting for exit signal...
[DEBUG] Got new client
[DEBUG] create: id: "0298b703c88281f40ff21341fc5c3c09b65ed672f274202a15c077492f301a67" bundle: "/var/run/desktop-containerd/daemon/io.containerd.runtime.v2.task/moby/0298b703c88281f40ff21341fc5c3c09b65ed672f274202a15c077492f301a67" rootfs {type: "overlay" source: "overlay" options: "workdir=/var/lib/containerd-stargz-grpc/snapshotter/snapshots/440/work" options: "upperdir=/var/lib/containerd-stargz-grpc/snapshotter/snapshots/440/fs" options: "lowerdir=/var/lib/containerd-stargz-grpc/snapshotter/snapshots/397/fs:/var/lib/containerd-stargz-grpc/snapshotter/snapshots/223/fs"} stdout: "/var/run/docker/containerd/0298b703c88281f40ff21341fc5c3c09b65ed672f274202a15c077492f301a67/init-stdout" stderr: "/var/run/docker/containerd/0298b703c88281f40ff21341fc5c3c09b65ed672f274202a15c077492f301a67/init-stderr"
[DEBUG] create done
[2023-02-26T10:16:57.286022615Z][containerd][I] time="2023-02-26T10:16:57.285763458Z" level=debug msg="event forwarded" ns=moby topic=/tasks/create type=containerd.events.TaskCreate
[DEBUG] connect: id: "0298b703c88281f40ff21341fc5c3c09b65ed672f274202a15c077492f301a67"
[DEBUG] start: id: "0298b703c88281f40ff21341fc5c3c09b65ed672f274202a15c077492f301a67"
[INFO] >>> stdin: "", stdout: "/var/run/docker/containerd/0298b703c88281f40ff21341fc5c3c09b65ed672f274202a15c077492f301a67/init-stdout", stderr: "/var/run/docker/containerd/0298b703c88281f40ff21341fc5c3c09b65ed672f274202a15c077492f301a67/init-stderr"
[INFO] >>> waiting for start notification
[INFO] >>> loading module: /var/run/desktop-containerd/daemon/io.containerd.runtime.v2.task/moby/0298b703c88281f40ff21341fc5c3c09b65ed672f274202a15c077492f301a67/rootfs/spin.toml
[INFO] >>> working dir: /var/run/desktop-containerd/daemon/io.containerd.runtime.v2.task/moby/0298b703c88281f40ff21341fc5c3c09b65ed672f274202a15c077492f301a67/rootfs
[INFO] >>> starting spin
[INFO] >>> building spin application
[INFO] >>> error starting instance: Cannot read manifest file from "/var/run/desktop-containerd/daemon/io.containerd.runtime.v2.task/moby/0298b703c88281f40ff21341fc5c3c09b65ed672f274202a15c077492f301a67/rootfs/spin.toml"
From my findings, it seems the error happens when the line key_value_stores = ["default"] is defined in the spin.toml file and the shim fails right after this line:
/cc @rumpl
Does you image contain the spin.toml file?
Yes, here's the Dockerfile:
https://github.com/felipecruz91/tinygo-key-value/blob/main/Dockerfile#L12
Hi,
I can't reproduce the problem. But I can run your app. 😊
Since the container in the gist used the spin shim v0.3.3 I just updated it to v0.4.0.
So I did a factory reset in Docker Desktop and started with the gist.
Then I built your app with:
docker buildx build \
--push \
--platform=wasi/wasm \
-t 0xe282b0/tinygo-key-value .
And finally deployed it with this manifes (You image is working as well.)
I can't really guarantee that the installation process every time results in a working cluster. Especially when already Spin containers are running and Docker Desktop restarts the init container has problems because the shim is already in use. Therefore, it is the safest way to do a factory reset when something is strange.
So you may try it after a factory reset with the update installer image. Hope that helps.
Closing this issue now.