Harden against tough files
dzmitry-lahoda opened this issue · comments
Research, investigation and ediscovery meet hard files in a wild. You may harden your service against these https://gitlab.com/dzmitry-lahoda/ediscovery-files/tree/master/assets .
Link 404s...
Dot in the end attached to url. Now it is fine.
Previous hyperink is without dot, only the text is wrong. In any case, it 404s. Maybe there is something wrong with your repos privacy settings?
Link says it is public, but for some reason repository is not visible without login. At least 404 is not user friendly. Will create issue for gitlab. At least I believe files are accessible here after login to gitlab via your or any account https://gitlab.com/dzmitry-lahoda/ediscovery-files . I do work on ediscovery system, and in my spare free hours do collect files. Then upload into various analisys tools. Some just die or produce unresonable stuff of these.
As an outside observer who wandered into this issue. The link is public, but even when you are logged in there are no files in the project.
Seems like a valuable resource. Do you have a write-up about it somewhere? How you collected/generated them, etc.?
I'm not seeing any files either.
Sorry for wasting your time. Blame to me. I created repository as private and then made public. But there where more settings. Now I am able to access from private browser tab.
Files are here:
https://gitlab.com/dzmitry-lahoda/ediscovery-files/tree/master/assets
Some generators are here:
https://gitlab.com/dzmitry-lahoda/ediscovery-files/tree/master/src
But all generations results are in assets.
Overall I scattered internet for bad files or generated bad files or strange/broken files. Not all documented.
I did not targeted files with meaning or so too much. Some semantics data sets are here: https://www.diigo.com/user/dzmitry_lahoda?query=ediscovery+data
My files are more pushing some limits of systems - virus scan, unarchive(zip bombs, deep nesting, formats, etc), object character recognition, conversion to html5, scale/throughput, content based file type detection, big files, search indexing, culling useless files(e.g. os system binaries), etc. I make more descriptions or generations from time to time.
Interesting stuff! Seems relevant - a lot of projects should probably check against such files
Thanks for sharing your work! Are you planning to extend it to include expected output? I imagine it could become a formal test suite.
Overall description - given these files uploaded then system does not crash or fails or prevented from processing valid simple documents. So expected output that uploading your known file and any other from my set still makes processing of your file fine and do not crash system. I think of my set as https://github.com/minimaxir/big-list-of-naughty-strings, but for files. My set have other subset just to check some formats and relations, I will update each file with expected failure type with time.