concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
Adnan-annan opened this issue · comments
I am getting this error when i run poker_ai cluster. Did you ever get this error?
(poker3.8) adnan@adnan-GL65-Leopard-10SFSK:~/poker_ai$ poker_ai cluster
Creating flop info combos: 100%|██████████████| 190/190 [00:07<00:00, 26.43it/s]
[15:52:24] INFO created flop card_combos.py:33
Creating turn info combos: 100%|██████████████| 190/190 [00:35<00:00, 5.42it/s]
[15:52:59] INFO created turn card_combos.py:37
Creating river info combos: 100%|█████████████| 190/190 [02:07<00:00, 1.49it/s]
[15:55:07] INFO created river card_combos.py:41
INFO Starting computation of clusters.card_info_lut_builder.py:65
INFO Starting computation of river card_info_lut_builder.py:91
clusters.
100%|████████████████████████████████| 1627920/1627920 [36:57<00:00, 734.06it/s]
/home/adnan/anaconda3/envs/poker3.8/lib/python3.8/site-packages/sklearn/cluster/kmeans.py:1008: ConvergenceWarning: Number of distinct clusters (27) found smaller than n_clusters (50). Possibly due to duplicate points in X.
return self.fit(X, sample_weight=sample_weight).labels
[16:32:23] INFO Finished computation of river card_info_lut_builder.py:108
clusters - took
2236.6857018470764 seconds.
INFO Creating lookup table. card_info_lut_builder.py:365
100%|█████████████████████████████| 1627920/1627920 [00:04<00:00, 343651.60it/s]
[16:33:09] INFO Starting computation of turn card_info_lut_builder.py:115
clusters.
0%| | 0/581400 [00:00<?, ?it/s]Traceback (most recent call last):
File "/home/adnan/anaconda3/envs/poker3.8/bin/poker_ai", line 6, in
cli()
File "/home/adnan/anaconda3/envs/poker3.8/lib/python3.8/site-packages/click/core.py", line 764, in call
return self.main(*args, **kwargs)
File "/home/adnan/anaconda3/envs/poker3.8/lib/python3.8/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/home/adnan/anaconda3/envs/poker3.8/lib/python3.8/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/adnan/anaconda3/envs/poker3.8/lib/python3.8/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/adnan/anaconda3/envs/poker3.8/lib/python3.8/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/home/adnan/anaconda3/envs/poker3.8/lib/python3.8/site-packages/poker_ai/clustering/runner.py", line 133, in cluster
builder.compute(
File "/home/adnan/anaconda3/envs/poker3.8/lib/python3.8/site-packages/poker_ai/clustering/card_info_lut_builder.py", line 79, in compute
self.card_info_lut["turn"] = self._compute_turn_clusters(n_turn_clusters)
File "/home/adnan/anaconda3/envs/poker3.8/lib/python3.8/site-packages/poker_ai/clustering/card_info_lut_builder.py", line 118, in _compute_turn_clusters
self._turn_ehs_distributions = list(
File "/home/adnan/anaconda3/envs/poker3.8/lib/python3.8/site-packages/tqdm/std.py", line 1108, in iter
for obj in iterable:
File "/home/adnan/anaconda3/envs/poker3.8/lib/python3.8/concurrent/futures/process.py", line 484, in _chain_from_iterable_of_lists
for element in iterable:
File "/home/adnan/anaconda3/envs/poker3.8/lib/python3.8/concurrent/futures/_base.py", line 611, in result_iterator
yield fs.pop().result()
File "/home/adnan/anaconda3/envs/poker3.8/lib/python3.8/concurrent/futures/_base.py", line 439, in result
return self.__get_result()
File "/home/adnan/anaconda3/envs/poker3.8/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
0%| | 0/581400 [00:39<?, ?it/s]
File "/home/adnan/anaconda3/lib/python3.8/site-packages/joblib/numpy_pickle.py", line 479, in dump
with open(filename, 'wb') as f:
PermissionError: [Errno 13] Permission denied: 'card_info_lut.joblib'
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
(poker3.8) adnan@adnan-GL65-Leopard-10SFSK:~/poker_ai$ poker_ai cluster
Creating flop info combos: 100%|████████████████████████████████████████████████| 190/190 [00:07<00:00, 27.11it/s]
[16:44:00] INFO created flop card_combos.py:33
Creating turn info combos: 100%|████████████████████████████████████████████████| 190/190 [00:35<00:00, 5.36it/s]
[16:44:35] INFO created turn card_combos.py:37
Creating river info combos: 100%|███████████████████████████████████████████████| 190/190 [02:09<00:00, 1.47it/s]
[16:46:45] INFO created river card_combos.py:41
[16:47:04] INFO Starting computation of clusters. card_info_lut_builder.py:65
INFO Starting computation of turn clusters. card_info_lut_builder.py:115
0%| | 0/581400 [00:00<?, ?it/s]Killed
I hit this error as well. Mine was able to run the river and turn clusters, yet on the flop cluster computation, it gave me this same error. It is stuck at 0/155040 (0% completion) for ~15 min, and then spits out this error. Any thoughts or recommendations on how to get around this would be helpful.
This wasn't clear to me, but if you look through the "--help" command, it recommends "starting low" with the parameters. So I followed that advice, and it looks like it completed.
$ ./poker_ai.bin cluster --low_card_rank=10 --high_card_rank=14 --n_river_clusters=3 --n_turn_clusters=3 --n_flop_clusters=3 --n_simulations_{river,turn,flop}=3
check your cpu and mem useage,try concurrent.futures.ProcessPoolExecutor() ->concurrent.futures.ProcessPoolExecutor(max_workers=?)
QQ 276749553
I had the same problem as OP.
I suspected it was due to lack of memory, so I upgraded my VM to 4 vCPUS and 16GB RAM.
The computation of turn clusters is now on 52%, and the RAM consumption is skyrockting (currently, 9.72GB)
did you guys create pkl files? There is no documentation on creating them.
I'm still encountering the same issues as OP.
I've tried the suggestion from @GaussianZero
--low_card_rank=10 --high_card_rank=14 --n_river_clusters=3 --n_turn_clusters=3 --n_flop_clusters=3 --n_simulations_{river,turn,flop}=3
But it still consumes too much RAM. I'm running it on a 16 GB machine and when I ran the poker_ai cluster command, it creates many processes which eat up 100% of my RAM and kills the process automatically.
Is there a way to limit the RAM usage from this clustering process? Or how did you guys deal with the RAM consumption?
Thanks in advance!