microsoft / infinibatch

Efficient, check-pointed data loading for deep learning with massive data sets.

microsoft/infinibatch Issues

Support for cloud streaming
Updated a year ago
Iterator will stuck when use PrefetchIterator after ParallelMapIterator, hangs indefinitely at pool.map operation of ParallelMapIterator, any ideas?
Updated 2 years ago
Can we get the length of entire dataset from `chunked_dataset_iterator`?
Updated 3 years ago
InfinitePermutationSourceIterator + distributed training + shuffle
Closed 3 years ago2
Fix triggers for unit tests
Closed 3 years ago1
Few questions on real world usage of infinibatch while training models
Updated 4 years ago
Implement wrapper iterator that inherits from PyTorch's IterableDataset
Updated 4 years ago1
training stuck if data loader worker process has exceptions
Updated 4 years ago
Why is setup.py empty?
Closed 4 years ago5
Add FAQ for typical programming patterns
Updated 4 years ago
Edit author's list in setup.py
Updated 4 years ago