How to configure the parameters of the main thread in the case of multiple workers?
gclm opened this issue · comments
孤城落寞 commented
At present, I want to run a torch program with multiple workers, but GPU CUDA needs to configure compatible parameters in the main thread. At present, the relevant configuration documents are not found in the documents
Adam Hopkins commented
I am not entirely sure what you are trying to achieve. Maybe main_process_start
?
孤城落寞 commented
no way. This seems to be configured every time the worker is created. At present, an error is still reported.
config code:
@app.listener("main_process_start")
async def init_processor(app, loop):
mp.set_start_method('spawn')
app.processor = Processor(MODEL_WEIGHTS, MODEL_LABELS)
error message:
[2021-12-02 16:36:56 +0800] [64614] [ERROR] Exception occurred while handling uri: 'http://localhost:8000/predict/file'
Traceback (most recent call last):
File "handle_request", line 83, in handle_request
class Sanic(BaseSanic, metaclass=TouchUpMeta):
File "/opt/projects/ai-server/server.py", line 29, in predict_file
result = app.processor.process(img)
File "/opt/projects/ai-server/models/Detect.py", line 82, in process
img = self.pre_processor(image)
File "/opt/projects/ai-server/models/Detect.py", line 74, in pre_processor
img = torch.from_numpy(img).to(self.device)
File "/home/breakfox/anaconda3/envs/ai-server/lib/python3.9/site-packages/torch/cuda/__init__.py", line 204, in _lazy_init
raise RuntimeError(
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
Exception occurred while handling uri: 'http://localhost:8000/predict/file'
Traceback (most recent call last):
File "handle_request", line 83, in handle_request
class Sanic(BaseSanic, metaclass=TouchUpMeta):
File "/opt/projects/ai-server/server.py", line 29, in predict_file
result = app.processor.process(img)
File "/opt/projects/ai-server/models/Detect.py", line 82, in process
img = self.pre_processor(image)
File "/opt/projects/ai-server/models/Detect.py", line 74, in pre_processor
img = torch.from_numpy(img).to(self.device)
File "/home/breakfox/anaconda3/envs/ai-server/lib/python3.9/site-packages/torch/cuda/__init__.py", line 204, in _lazy_init
raise RuntimeError(
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
error message docs: https://discuss.pytorch.org/t/not-using-multiprocessing-but-getting-cuda-error-re-forked-subprocess/54610
How to configure it in this case?