An efficient way to compute the derivative of large computer programs in a distributed manner.
The computation of the Jacobian Matrix can be done without Map-Reduce. But computing it iteratively for each Cartesian basis vector, without map-reduce is extremely slow. Moreover, this is only feasible for the computations having less number of dimensions. In real life numerical calculations, the number of dimensions is quite large and it's not practically possible to compute the Jacobian iteratively for each subroutine.
Downgrade your python version or create a virtual environment with Python 3.7.
Then use pip install pyspark and you are good to go.