PATCH 500 error, "Lockfile created, but doesn't exist" on Azure Kubernetes Service

[Jonathan Takahashi]

I have tusd running in a Ubuntu docker container. I have a persistent volume claim with ReadWriteMany access mounted on the pod, and am running the image using the docker CMD command. The docker image is a custom image built with the following dockerfile:

FROM ubuntu:18.04
  
RUN apt-get update && apt-get install -y wget dpkg
RUN wget https://github.com/tus/tusd/releases/download/0.11.0/tusd_snapshot_amd64.deb
RUN dpkg -i tusd_snapshot_amd64.deb

WORKDIR /uploads

EXPOSE 1080
ENTRYPOINT ["tusd"]
CMD ["-behind-proxy","-dir","/uploads"]

The pvc is mounted to /uploads using the following Kubernetes YAML file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tusd-deployment
  labels:
    app: tusd
    type: web
spec:
  selector:
    matchLabels:
      app: tusd
      type: web
  replicas: 2
  template:
    metadata:
      labels:
        app: tusd
        type: web
    spec:
      containers:
        - name: tusd
          image: $DOCKERHUB_USER/tusd:latest
          imagePullPolicy: "Always"
          ports:
            - containerPort: 1080
          volumeMounts:
            - mountPath: /uploads
              name: upload-pv-claim
      volumes:
        - name: upload-pv-claim
          persistentVolumeClaim:
            claimName: upload-pv-claim

and this is the YAML for the pvc:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: upload-pv-claim
  labels:
    app: upload
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: azurefile
  resources:
    requests:
      storage: 2Gi

I am also running tusd behind nginx, however I believe this issue is unrelated. I can provide my nginx.conf if anyone thinks it is relevant. A snippet of the output from tusd is as follows:

[tusd] event="ResponseOutgoing" status="201" method="POST" path="" 
[tusd] event="RequestIncoming" method="PATCH" path="3a75c8ff5fca05aecd8621427581cbda" 
[tusd] event="ResponseOutgoing" status="500" method="PATCH" path="3a75c8ff5fca05aecd8621427581cbda" error="Lockfile created, but doesn't exist" 
[tusd] event="RequestIncoming" method="PATCH" path="3a75c8ff5fca05aecd8621427581cbda" 
[tusd] event="ResponseOutgoing" status="500" method="PATCH" path="3a75c8ff5fca05aecd8621427581cbda" error="Lockfile created, but doesn't exist" 
[tusd] event="RequestIncoming" method="PATCH" path="032342d32c16b0e5db275c2af4423069" 
[tusd] event="ResponseOutgoing" status="500" method="PATCH" path="032342d32c16b0e5db275c2af4423069" error="Lockfile created, but doesn't exist"

Any help or advice is appreciated, I realize this might be an issue with Azure Files.

[Marius]

Thank you for the detailed report. I am quite certain that this is caused by the Azure Files mount not supporting symbolic links. These links are used by tusd internally for the lockfiles that are mentioned in the error messages.

Something similar can happen when using tusd inside a VM where symbolic links are also not supported, see https://github.com/tus/tusd#can-i-run-tusd-inside-a-vmvagrantvirtualbox.

It would be nice if you could try to manually create a symbolic link on the Azure Files volume and see if that actually works. If it doesn't than my theory is probably right. If it does work, then we need to keep searching.

[Jonathan Takahashi]

You are correct, I got the following error:

ln: failed to create symbolic link '/uploads/blah_link': Operation not supported

So is the only course of action to switch to a different storage type? If so I can try setting up an NFS container connected to Azure Disks. I tried creating a symbolic link on there and didn't have an issue.

[Jonathan Takahashi]

I was able to get around this by using an NFS server. Thanks for very much for your help!

[Marius]

I am glad to hear that you found a solution! I extended the FAQ entry in the Readme to cover this situation.

[Nemanja Lakicevic]

I have the some problem with IBM Cloud Object Storage. Any possible ways to work around this?

[Marius]

@Nemox Please open a new issue with more details on your setup and what the problem is.