Loading purely in memory
jkudla opened this issue · comments
Currently, networks are downloaded from the KONECT.cc site, written to a ZIP archive file, unpacked and then loaded into memory. This requires writing to disk and leaves the ZIP archive along with the unpacked directory.
As a result of the HTTP request performed, the entire ZIP archive is already stored in memory before writing to a file. Hence the following
Suggestion: Allow unpacking and reading the network file without relying on any disk operations. Introduce an option to toggle this behaviour.
Edit: Regarding design, it could make sense to have three classes Loader
, DiskLoader
and MemoryLoader
such that the last two inherit from Loader
. In main.py
, an object of type Loader
is instantiated suitably to obtain the network from disk or the ZIP file stored in memory.