Image copy to Originals folder fails

Question

Image copy to Originals folder fails

r3dsouza opened this issue 4 years ago · comments

Describe the bug
I recently upgraded to the latest version and noticed the following warning in the logs on trigger :

2020-11-14T23:22:00+00:00 [Local storage] Unable to copy to local storage: EPERM: operation not permitted, copyfile '/aiinput/2020/11/14/Camera1_01_20201114160455 copy.jpg' -> '/node-deepstackai-trigger/originals/Camera1_01_20201114160455 copy.jpg'

At first I thought it might be a permissions issue on the host bind mount. So I gave the host folder full read-write access. But I still kept getting the same warning whenever it tries to copy the image file to the originals folder. I then tried consoling into the docker container and manually copied an image the image file over using the cp command and it worked fine, so it doesn't appear to be a permissions issue.

Need some help to figure out how to make the copying of images to the originals folder work. Any help appreciated.

Installation details

Running in Docker on QNAP TS-451+
Running the latest release v5.5.1

Docker Compose
deepstackai-trigger:
container_name: deepstackai-trigger
image: danecreekphotography/node-deepstackai-trigger:latest
hostname: deepstackai-trigger
networks:
- qnet-dhcp-eth0-3d3dab
ports:
- 4242:4242
environment:
- TZ=Europe/London
volumes:
- /share/CCTV/AI_Input:/aiinput
- /share/Containers/DeepStackAI_Trigger/config:/config
- /share/Containers/DeepStackAI_Trigger/node-deepstackai-trigger:/node-deepstackai-trigger
restart: always

Neil Enns · Answer 1 · Sun Nov 15 2020 09:08:43 GMT+0800 (China Standard Time)

Not entirely sure why this would be happening since nothing's changed in that part of the code forever. I checked my local system and it's not having any errors.

This smells like something specific to your docker configuration so my suggestion is to blow away the volume and the image and just start with a fresh one.

r3dsouza · Answer 2 · Mon Nov 16 2020 00:14:59 GMT+0800 (China Standard Time)

So I gave it a go. I tore down the docker image, volume and deleted the host bind folder. Then I recreated the folder on host and ran docker-compose up. After the container was up and running, I checked the contents of the bind mounted folder on the host and noticed that the container had successfully created the 3 subfolders under it for annotations, originals and snapshots.

Unfortunately though, when the container detects a person and tries to copy the trigger image to the originals folder, it still fails with the same error message as in my original post. Any ideas?

2020-11-15T15:57:12+00:00 [Local storage] Unable to copy to local storage: EPERM: operation not permitted, copyfile '/aiinput/2020/11/15/Camera1_01_20201114160455.jpg' -> '/node-deepstackai-trigger/originals/Camera1_01_20201114160455.jpg'

Neil Enns · Answer 3 · Mon Nov 16 2020 00:51:52 GMT+0800 (China Standard Time)

Can you try without this line in your docker-compose?

- /share/Containers/DeepStackAI_Trigger/node-deepstackai-trigger:/node-deepstackai-trigger

There's no real need to have a local mapped folder for it, you can just let Docker create it within the image itself. It won't persist between image creations but that's fine.

r3dsouza · Answer 4 · Mon Nov 16 2020 01:10:10 GMT+0800 (China Standard Time)

Just tried it, but still getting the same error. :(

Neil Enns · Answer 5 · Mon Nov 16 2020 01:21:02 GMT+0800 (China Standard Time)

Make sure you are deleting the Docker volume, and for good measure blow away the Docker image too. Docker looooooves to cache stuff. Just deleting the local bind folder may not be enough.

When you are consoling in you are likely running as root. All the code runs as the node user (UID 1000). One thing you could also do is connect in via the terminal and look at the owner user/group on the folders down the chain to the path that's failing.

r3dsouza · Answer 6 · Mon Nov 16 2020 02:27:17 GMT+0800 (China Standard Time)

Thanks so much for your suggestion. Really appreciate it. So I tried tearing down the container, deleting the image and starting from scratch. This time though, I enabled the webserver and annotations, and commented out the bind mount to node-deepstackai-trigger so it would be creating this folder/subfolders internal to the container. On starting up the container, and when it matches a person trigger I noticed that there was a copy of the image in the annotations folder, however, I still get the error about copying to the originals folder! So, not sure what is happening with the copy to the originals folder. Any ideas?

2020-11-15T18:12:29+00:00 [Local storage] Unable to copy to local storage: EPERM: operation not permitted, copyfile '/aiinput/2020/11/15/Camera1_01_20201114160459.jpg' -> '/node-deepstackai-trigger/originals/Camera1_01_20201114160459.jpg'

Neil Enns · Answer 7 · Mon Nov 16 2020 05:30:05 GMT+0800 (China Standard Time)

I've got an idea to try. In your settings.json add "awaitWriteFinish": true and see if that fixes it.

r3dsouza · Answer 8 · Mon Nov 16 2020 05:54:17 GMT+0800 (China Standard Time)

awaitWriteFinish is already set to true in my settings.json. I believe you had implemented this a while back when I was it was having trouble reading the image file on my network share. I has been set to true ever since.

r3dsouza · Answer 9 · Mon Nov 16 2020 05:57:08 GMT+0800 (China Standard Time)

Is there a difference in how the annotation saves the file to the annotations folder vs how the file gets copied to the originals folder? Could that give us some clue? Let me know if there is anything else I can try

Neil Enns · Answer 10 · Mon Nov 16 2020 06:13:13 GMT+0800 (China Standard Time)

I'm running out of ideas :( The difference between originals and annotations is originals is copying a file and annotations is creating a new one. But if you can read the file to make the annotated version it should be able to copy.

What's the user and group listed as the owner on the originals folder in the Docker image?

r3dsouza · Answer 11 · Mon Nov 16 2020 08:48:05 GMT+0800 (China Standard Time)

I consoled into the container and the node-deepstackai-trigger and all subfolders including the originals folder is owned by user node and group node. I ran 'id node' and the uid:gid is 1000:1000 (uid=1000(node) gid=1000(node) groups=1000(node))

Neil Enns · Answer 12 · Mon Nov 16 2020 09:08:45 GMT+0800 (China Standard Time)

Well I'm out of ideas :( I don't know what could be causing it at this point.

r3dsouza · Answer 13 · Mon Nov 16 2020 20:21:12 GMT+0800 (China Standard Time)

SOLVED! After having exhausted all other avenues, the only thing I had not checked was the version of docker that was running on my QNAP NAS and it was 17.09.1-ce. I updated this to version 19.03.13 and the issue was resolved. Adding it here in case it helps someone.

Thanks again for your help!

Neil Enns · Answer 14 · Tue Nov 17 2020 00:15:23 GMT+0800 (China Standard Time)

That's... whack. Glad it fixed it though!

r3dsouza · Answer 15 · Tue Nov 17 2020 01:44:35 GMT+0800 (China Standard Time)

Yep, it was a relief to get to the bottom of the issue! I believe I was having this issue for a while but only noticed it after I updated the docker image to the latest version. This is one of the downsides of using docker on a NAS platform, the updates are not as quick as this has to be packaged and made available by the NAS vendor :(