[Bug]: Failed when used docker cpuset
wangxiong172086864 opened this issue · comments
What happened?
When docker using cpuset, onnxruntime can not set the session option inter_op_num_threads properly, then the fastdup run can't not get the results.
What did you expect to see?
No response
What version of fastdup were you runnning on?
1.44
What version of Python were you running on?
Python 3.7
Operating System
CentOS
Reproduction steps
No response
Relevant log output
No response
Attach a screenshot [Optional]
No response
Contact Details [Optional]
No response
Hello @wangxiong172086864 we will be happy to help but we need some more detailed information.
You are running on centos, but what docker OS are you using? We do have native Centos 7 build for fastup you can install directly from our release page.
In addition send us the full error message you see.
How are you using cpuset? If you are running without it does fastdup run?
Thanks
Hello @wangxiong172086864 we will be happy to help but we need some more detailed information. You are running on centos, but what docker OS are you using? We do have native Centos 7 build for fastup you can install directly from our release page. In addition send us the full error message you see. How are you using cpuset? If you are running without it does fastdup run? Thanks
Hi, docker os is ubuntu 18.04.1 LTS. We using cpuset like this --cpuset-cpus="0-1,10-13,18-29,46-83", we can't catch the error message, the process stop at fd.run() method. I have tried to find the reason for this, I found someone said that onnxruntime may set sess_options.intra_op_num_threads an improper value when cpuset is used., but I can't figure it out since I can't change the intra_op_num_threads value.
running the code without cpuset or just set cpuset in a continuous interval will be ok.
Under the hood it is possible to change the intra_op_num_threads value using 0.2 API:
fastdup.run(intput_dir, work_dir, turi_param='num_onnx_intra_threads=1') however the default is 1 anyway and I don't think it will help.. Maybe try to give the cpu opst a continuous range?
Under the hood it is possible to change the intra_op_num_threads value using 0.2 API:
fastdup.run(intput_dir, work_dir, turi_param='num_onnx_intra_threads=1')however the default is 1 anyway and I don't think it will help.. Maybe try to give the cpu opst a continuous range?
Just tried fastdup.run(intput_dir, work_dir, turi_param='num_onnx_intra_threads=1') and still get stuck, we run the code in the environment where cpu opst can't be decided by users.
Closing