Make TempFileTask maximum file length configurable
wladimirleite opened this issue · comments
Currently, there is a fixed limit of 1 GB.
I guess this limit was created to avoid using too much space in the temporary folder. Right, @lfcnassif?
I am working on a case with some HDDs full of ~4 GB ISO's. And as the E01s are quite large, I had to place them in a HDD (not a SSD).
Monitoring the processing and the I/O, I noticed that there is a bottleneck in my workstation HDD (where the evidence files are), as expected.
However, a few tasks (like HashTask and ParsingTask) need to read the whole original file. If it was copied to the temporary folder (in a very fast SSD), it would speed things up (at least by a factor of 2).
That is the goal of the TempFileTask, but as all files are larger than 1 GB, it is not being used.
My suggestion is to make this maximum file length configurable. In my case, I have ~1 TB of free space for the Temp, so I could use a larger value.
@lfcnassif, is there any reason for not creating this configuration?
Where should it be placed? In a new config file ("TempFileTaskConfig.txt"), or as a new entry in "FileSystemConfig.txt", or somewhere else?
I guess this limit was created to avoid using too much space in the temporary folder. Right, @lfcnassif?
Exactly!
@lfcnassif, is there any reason for not creating this configuration?
No, and actually I've already considered to create it a few times in the past :-)
Where should it be placed? In a new config file ("TempFileTaskConfig.txt"), or as a new entry in "FileSystemConfig.txt", or somewhere else?
Good question. Not sure, but given current config files organization, I think a new TempFileTaskConfig.txt would fit good.
Thanks @lfcnassif!
As this is trivial and may help a lot in the case I am working on, I will create a PR.
Just to warn users reading this thread, you must have much more space in temp folder than numThreads x maxTempFileSize. We have to always create temp files for some file types, regardless of their size, because of some libraries API, like for PST, OST, RAR, ISO, SQLITE, videos and some image formats, etc... Some modules also need to create additional auxiliary temp files to work properly.