OpenMP threads are all assigned to same CPU

[Louis Legrand]

Running lenscarf on NERSC, it seems that the OpenMP threads are all assigned to the same CPU, instead of spreading into OMP_NUM_THREADS number of CPUs.

Typing in the termnial:

export OMP_DISPLAY_ENV=true
export OMP_NUM_THREADS=16
export OMP_PROC_BIND=true
srun -n 1 -c 16 -u --cpu_bind=cores python -c "from lenscarf.fortran import remapping; remapping.helloworld()"

This gives in the OpenMP display:
OMP_PLACES = '{0},{32},{1},{33},{2},{34},{3},{35},{4},{36},{5},{37},{6},{38},{7},{39}'

But if I do

export OMP_DISPLAY_ENV=true
export OMP_NUM_THREADS=16
export OMP_PROC_BIND=true
srun -n 1 -c 16 -u --cpu_bind=cores python  $HOME/lenscarf/lenscarf/params/cmbs4wide_idealized.py -k p_p -itmax 10  -imin 0 -imax 0

I get
OMP_PLACES = '{0}'
meaning that all the threads will be launched on the CPU indexed 0.

Maybe there is something happening in the cmbs4wide_idealized.py causing this issue?

Setting OMP_PROC_BIND=false avoids this problem by letting threads being randomly assigned to CPUs, but this could be suboptimal if the OS assigns several threads to the same CPU.

[Sebastian Belkner]

Hey Louis,

can you please,
mpstat -P ALL
when running
srun -n 1 -c 16 -u --cpu_bind=cores python $HOME/lenscarf/lenscarf/params/cmbs4wide_idealized.py -k p_p -itmax 10 -imin 0 -imax 0
and post the output?

[Sebastian Belkner]

Assuming this is somewhat outdated. Plus we haven't run into any issues with it lately
--> closing