Tensorflow with future package for asynchronous call.

Question

Tensorflow with future package for asynchronous call.

Ash2023qqq opened this issue a year ago · comments

Ash2023qqq commented a year ago

Hi Team,

My application is created using R base version : 4.0.3 with tensorflow = 2.5.0, keras=2.4.0 using python 3.8 & future 1.21.1
We use future package with multicore process for our application. The multicore will fork the processes and our application is getting crashed during this process. Kindly let us know whether tensorflow is fork-safe to handle the process.

Tomasz Kalinowski · Answer 1 · Thu Mar 30 2023 22:34:30 GMT+0800 (China Standard Time)

Hi, thanks for reaching out.

TensorFlow is not fork safe, either from R or Python. TensorFlow manages its own thread pool, and that is the recommended way to get multi-core parallelism using TensorFlow.

(e.g., using tfdatasets::dataset_map(..., num_parallel_calls = <n>), tf_function() in combination with tfautograph::ag_while_opts(), tf$map_fn(), etc).

Ash2023qqq · Answer 2 · Fri Mar 31 2023 11:56:43 GMT+0800 (China Standard Time)

Hi Tomasz Kalinowski,

Thanks for the quick reply. Sure will check on that.