How to configure the parameters of the main thread in the case of multiple workers?
gclm opened this issue · comments
At present, I want to run a torch program with multiple workers, but GPU CUDA needs to configure compatible parameters in the main thread. At present, the relevant configuration documents are not found in the documents
I am not entirely sure what you are trying to achieve. Maybe main_process_start
?
no way. This seems to be configured every time the worker is created. At present, an error is still reported.
config code:
@app.listener("main_process_start")
async def init_processor(app, loop):
mp.set_start_method('spawn')
app.processor = Processor(MODEL_WEIGHTS, MODEL_LABELS)
error message:
[2021-12-02 16:36:56 +0800] [64614] [ERROR] Exception occurred while handling uri: 'http://localhost:8000/predict/file'
Traceback (most recent call last):
File "handle_request", line 83, in handle_request
class Sanic(BaseSanic, metaclass=TouchUpMeta):
File "/opt/projects/ai-server/server.py", line 29, in predict_file
result = app.processor.process(img)
File "/opt/projects/ai-server/models/Detect.py", line 82, in process
img = self.pre_processor(image)
File "/opt/projects/ai-server/models/Detect.py", line 74, in pre_processor
img = torch.from_numpy(img).to(self.device)
File "/home/breakfox/anaconda3/envs/ai-server/lib/python3.9/site-packages/torch/cuda/__init__.py", line 204, in _lazy_init
raise RuntimeError(
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
Exception occurred while handling uri: 'http://localhost:8000/predict/file'
Traceback (most recent call last):
File "handle_request", line 83, in handle_request
class Sanic(BaseSanic, metaclass=TouchUpMeta):
File "/opt/projects/ai-server/server.py", line 29, in predict_file
result = app.processor.process(img)
File "/opt/projects/ai-server/models/Detect.py", line 82, in process
img = self.pre_processor(image)
File "/opt/projects/ai-server/models/Detect.py", line 74, in pre_processor
img = torch.from_numpy(img).to(self.device)
File "/home/breakfox/anaconda3/envs/ai-server/lib/python3.9/site-packages/torch/cuda/__init__.py", line 204, in _lazy_init
raise RuntimeError(
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
error message docs: https://discuss.pytorch.org/t/not-using-multiprocessing-but-getting-cuda-error-re-forked-subprocess/54610
How to configure it in this case?