segmentation fault on Plugin tests
mahb324 opened this issue · comments
Hi,
I am trying to build and install the plugin from git source on EC2 (with RHEL 8).
when I do the smoke, or any test as soon as I hit the _pywrap_s3_io.s3Init() a segmentation fault is thrown.
do you have any suggestion ? does plugin works on RHEL?
Can you use containers for the plugin?
It is available through following containers:
763104351884.dkr.ecr.us-west-2.amazonaws.com/pytorch-training:1.9.0-cpu-py38-ubuntu20.04-v1.1
763104351884.dkr.ecr.us-west-2.amazonaws.com/pytorch-training:1.9.0-gpu-py38-cu111-ubuntu20.04-v1.1
763104351884.dkr.ecr.us-west-2.amazonaws.com/pytorch-training:1.8.1-cpu-py36-ubuntu18.04-v1.6
763104351884.dkr.ecr.us-west-2.amazonaws.com/pytorch-training:1.8.1-gpu-py36-cu111-ubuntu18.04-v1.7
I switched over to ubuntu and the segmentation fault went away, I guess you might not supporting RHEL, is that true?
I am now trying to run my code using 2.7G data I am using map style dataset and get a very poor performance (~900 sec) to load the data. the network utilization is so low, I tried different number dataloading workers, did not help.
is there a way I can initiate some streaming during the dataset and dataloading , to improve the performance?
How is your data distributed in S3 bucket?
For datasets consisting of a large number of smaller objects, accessing each object individually can be inefficient. For such datasets, it is recommended to create shards of the training data(50-200MB) and use S3IterableDataset for better performance.
We're upstreaming the amazon-s3-plugin-for-pytorch into the torchdata package (pytorch/data#165).
We're dropping support for this plugin.
Closing this issue as there's no further request from user. Feel free to re-open if necessary.