Web Dav and FYSETC SD-WIFI-PRO?
luc-github opened this issue · comments
@y2kdread wrote :
When I try accessing the files through WebDAV (mapped through windows), everything works fine. Whenever I try copying from the SD to the PC, the ESP locks up. The file transfers never completes and the web UI locks up. The only way for this to recover is to eject the SD card fully and reinsert it.
On My side webdav is working well
Are there any limitations on the numbers of files or directors that can be used? I hit this issue when I try to access a folder with a large number of directories, 782 to be exact. If I go to a folder that has a small number of files and folders inside of it, things work fine.
If it matters, I am trying to use this card to allow me to remotely access the files that my CPAP machine creates so I can load it into a tool that analyses the data. The CPAP machine creates a new directory with the data every day, and each directory will contain between 20-50 files depending on what it decides to capture that night. The CPAP only writes to the card and doesn't seem to modify the nightly data after it has been written.
Are there any ways that I can optimize ESP to be able to deal with this large number of files and folder through WebDAV better?
if large number of files is a triger then I will see if I can reroduce as I never tested with so many for sure. webdav and ftp
I will work on webdav rewrite starting today but will try reproduce your issue first
i will prepare a folder with 1000 files for the tests .
so just accessing such dir crash , right ?
Yes, this is correct.
so I should be easy to reproduce and to fix I have some idea of possible root cause
well I did 999 files and copied on SD manualy from Windows explorer (Win 11)
Then I put back to daughter board because my PC reader does not seems to power the SD enough for ESP32
Connect the webdav as Drive and open the SD/testdir where the 999 files are - after a couple of minutes - looks long for a directory listing but got all and no crash on serial monitor
I still found a problem with creating a new folder under windows but I suspect a bug in long name - I will have a look at it
I also need to see if the directory listing speed can be improved.
so about your issue, I need more details like the names of the files, and files sizes and directory name or full path of location on SD to see if I can reproduce or guess the issue
I did test this again and it looks like it does act the same way it does for you. I am able to list the 784 directories in about 5 minutes. I repeated this test multiple times, and never had an issue with the ESP locking up. I am not sure what I had done originally to cause it to lock up.
I know you said you are working on seeing if there is a way to improve the directory listing speed, but is there any way for me to disable unneeded services to increase the speed of the WebDAV file system until then?
You can disable telnet as I guess you do not use it, but that is all, and it won't speed up the transfert actually
You may decrease the SPI Div to increase Sd access but be sure you do not get corrupted content if too low
The webdav protocol is very chatty, especially on windows that does not seems actually follow it properly, it need to access each file, one by one, to stat the size and the file type (file or dir), and for each file it need to generate a part of xml response
5min x 60 x 1000 = 300000 ms / 780 files = 385ms per file, this time include: SD access + response generation + message send
This time (385ms / file) is ok on few files but start to be terrible on long listing
I was hoping to speed up display by sending partial content response but this feature is only for file transfert not for PROPFIND response which list the content of the directory, so I am not sure what can be done for the moment, I need to dig in
Ok still digging back to code and cleaning a little - I have couple of things that I will do to speed up a little the webdav communications:
1 -reject PROPFIND with depth=infinity but do depth=1, windows scan the complete SD content - which is useless and take long time in SD with several sub directories with lot of files. the scan will be limited to the target and childs if depth = infinity
2 - remove the etag generation on each file for the PROPFIND, it is a unique identifier made doing some calculation, in esp3d case this tag is totaly useless, also per specification the lastmodified tag is the alternative tag to be used and we also provide it as it is mandatory - so etag can safely be removed - on ESP3D-TFT it is not generated/used and no issue under windows.
There are some unecessary checks in code that can be safely removed so I will do it also
I am not sure how much these change will improve but it worth a try
Sounds great, thanks for your hard work on this!
I have rewrite totaly the WebDav based on code I did for ESP3D-TFT, it simplify a lot many things but did not really improve response time for directory with huge amount of files
999 files need around 7 minutes to be parsed, so no improvement,
It seems it is the parsing itself that take time
sorry I am dry on this
It's no problem, I really apricate the time you've taken to work on this. I have no way to prove it, but using WebDAV now seems much more stable after the changes you made.
I think i your case need a dedicated fw and dedicated client:
1 to improve speed of parsing directory
2 to send listing by packets that can be immediatly sisplayed while others packets are built and send
3 use idf instead of arduino because it has async web server that is more efficient than arduino one if still need web interface
many possibilities, it is project by itself actually
I close issue as limitation for delay of displaying ~700 files
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.