Kubernetes ReadWriteMany mode problems
pstadler opened this issue · comments
Hi there
I'm running into issues when I delete a pod that uses a ReadWriteMany volume claim with px-dev. Nowadays, pods are mostly scheduled using Deployment, which means that a new pod is scheduled as soon as the the previous one gets deleted. The pod using the RWM volume can't get killed and is stuck in the Terminating state forever, while the new pod successfully binds to the same volume and gets Ready immediately. At the same time the following warnings start to appear and continue until the persistent volume (not the claim) gets deleted:
time="2017-04-08T08:22:57Z" level=warning msg="Volume (Name: pvc-0567564a-1c33-11e7-b46b-de2b4425e003 Id: 808123070627826950 Path: /var/lib/kubelet/pods/089721e3-1c33-11e7-b46b-de2b4425e003/volumes/kubernetes.io~portworx-volume/pvc-0567564a-1c33-11e7-b46b-de2b4425e003) unmount failed with error: Mountpath is not mounted" AbortOnError=false BackgroundProcessing=false Driver=pxd Error="Mountpath is not mounted" Format=FS_TYPE_EXT4 Function=Unmount ID=808123070627826950 State=VOLUME_STATE_ATTACHED Version=1570545
Please note that this does not happen with ReadWriteOnce volume claims. Force deleting the stuck pod using kubectl delete pod <POD> --grace-period=0 --force has no impact, px (or kubernetes itself) will try to unmount the volume forever. This all happens on Kubernetes 1.6.
Thanks for your great work. I'm currently writing a guide where I will most probably mention portworx as the first choice for getting distributed persistent storage in a small-scale setup on virtually any cloud provider.
ping @adityadani
Thanks for reporting this..
@pstadler We have fixed this issue in our latest release. You can get it by using the following tag
portworx/px-dev:latest.
I finally upgraded to the latest px-dev version. Seems to work well 👍