Need for OrderedDict
saswat0 opened this issue · comments
Hey @aosokin
Large datasets (~1.7M target images) take forever to be read (loaded into memory) via the _read_dataset_images function, and I was looking for a way to parallelize it with multiprocessing. But your implementation uses OrderedDict in many places, which isn't sharable across threads. Is there any particular reason for using this data structure, or can a normal dict serve the same purpose here?
I do not actually remember if we ever used the fixed ordering of keys.
However , dict should be equivalent to Ordered dict in modern python.
…On Wed, Dec 14, 2022, 12:40 Saswat Subhajyoti ***@***.***> wrote:
Hey @aosokin <https://github.com/aosokin>
Large datasets (~1.7M target images) take forever to be read (loaded into
memory) via the _read_dataset_images function, and I was looking for a
way to parallelize it with multiprocessing. But your implementation uses
OrderedDict in many places, which isn't sharable across threads. Is there
any particular reason for using this data structure, or can a normal dict
serve the same purpose here?
—
Reply to this email directly, view it on GitHub
<#40>, or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAQAQW6S2M5XD7YQXN7CEL3WNG52HANCNFSM6AAAAAAS6N4QM4>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
Thanks