Multi-threaded
billy1380 opened this issue · comments
Add a command line flag that allows the user to specify that the tool should use more than 1 thread.
Not sure if it is a good idea to be reading and seeking on the same file but could we be reading more than one file and importing them at the same time?
If all that is happening is processing multiple files simultaneously, can't you just use the OS to handle running multiple threads, each instructed to only handle a single file?
So for different files, this can be done via multiple configuration (json config file)... once the gzip is unpacked, all you would need to do is configure a different file for each instance to work on and pass the config file name it on the command line.
It would still be nice for the app to figure it out on its own and spawn a thread for each (or a subset).