Giters

firedrakeproject / gusto

Three dimensional atmospheric dynamical core using the Gung Ho numerics.

Home Page:http://firedrakeproject.org/gusto/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Deprecate PointDataOutput?

JDBetteridge opened this issue · comments

commented

The non-hydrostatic Skamarock Klemp example does not run in parallel, is this expected?

Full traceback below:

$ mpiexec -n 2 python examples/compressible/skamarock_klemp_nonhydrostatic.py --running-tests
firedrake:WARNING OMP_NUM_THREADS is not set or is set to a value greater than 1, we suggest setting OMP_NUM_THREADS=1 to improve performance
gusto:INFO Physical parameters that take non-default values:
gusto:INFO 
Traceback (most recent call last):
  File "/scratch/jbetteri/firedrake_py311_opt/src/gusto/examples/compressible/skamarock_klemp_nonhydrostatic.py", line 125, in <module>
Traceback (most recent call last):
  File "/scratch/jbetteri/firedrake_py311_opt/src/gusto/examples/compressible/skamarock_klemp_nonhydrostatic.py", line 125, in <module>
    stepper.run(t=0, tmax=tmax)
    stepper.run(t=0, tmax=tmax)
  File "/scratch/jbetteri/firedrake_py311_opt/src/gusto/gusto/timeloop.py", line 582, in run
  File "/scratch/jbetteri/firedrake_py311_opt/src/gusto/gusto/timeloop.py", line 582, in run
    super().run(t, tmax, pick_up=pick_up)
    super().run(t, tmax, pick_up=pick_up)
  File "/scratch/jbetteri/firedrake_py311_opt/src/gusto/gusto/timeloop.py", line 162, in run
  File "/scratch/jbetteri/firedrake_py311_opt/src/gusto/gusto/timeloop.py", line 162, in run
    self.io.setup_dump(self.fields, t, pick_up)
    self.io.setup_dump(self.fields, t, pick_up)
  File "/scratch/jbetteri/firedrake_py311_opt/src/gusto/gusto/io.py", line 412, in setup_dump
  File "/scratch/jbetteri/firedrake_py311_opt/src/gusto/gusto/io.py", line 412, in setup_dump
    self.dump(state_fields, t)
  File "/scratch/jbetteri/firedrake_py311_opt/src/gusto/gusto/io.py", line 549, in dump
    self.dump(state_fields, t)
  File "/scratch/jbetteri/firedrake_py311_opt/src/gusto/gusto/io.py", line 549, in dump
    self.pointdata_output.dump(state_fields, t)
  File "/scratch/jbetteri/firedrake_py311_opt/src/gusto/gusto/io.py", line 115, in dump
    self.pointdata_output.dump(state_fields, t)
  File "/scratch/jbetteri/firedrake_py311_opt/src/gusto/gusto/io.py", line 115, in dump
    val_list.append((field_name, np.asarray(field_creator(field_name).at(points, tolerance=self.tolerance))))
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "petsc4py/PETSc/Log.pyx", line 115, in petsc4py.PETSc.Log.EventDecorator.decorator.wrapped_func
    val_list.append((field_name, np.asarray(field_creator(field_name).at(points, tolerance=self.tolerance))))
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "petsc4py/PETSc/Log.pyx", line 115, in petsc4py.PETSc.Log.EventDecorator.decorator.wrapped_func
  File "petsc4py/PETSc/Log.pyx", line 116, in petsc4py.PETSc.Log.EventDecorator.decorator.wrapped_func
  File "/scratch/jbetteri/firedrake_py311_opt/src/firedrake/firedrake/function.py", line 622, in at
  File "petsc4py/PETSc/Log.pyx", line 116, in petsc4py.PETSc.Log.EventDecorator.decorator.wrapped_func
  File "/scratch/jbetteri/firedrake_py311_opt/src/firedrake/firedrake/function.py", line 622, in at
    raise RuntimeError("Point evaluation gave different results across processes.")
RuntimeError: Point evaluation gave different results across processes.
Abort(1) on node 0 (rank 0 in comm 496): application called MPI_Abort(PYOP2_COMM_WORLD, 1) - process 0
    raise RuntimeError("Point evaluation gave different results across processes.")
RuntimeError: Point evaluation gave different results across processes.
Abort(1) on node 1 (rank 1 in comm 496): application called MPI_Abort(PYOP2_COMM_WORLD, 1) - process 1
commented

What is going on here is that this example uses the "point-data" output capability, which does not work in parallel.

This "point-data" outputting is now quite old, and uses the .at routine to sample the finite element fields at particular points. Our newer netcdf outputting gives us a similar capability and does work in parallel**, so I don't know if we still need the "point-data" outputting any more ...

The short term fix for this example is just to remove the "point-data" outputting for this example, since we don't actually use the output, it's just in this example to ensure that we still test this capability. The longer term fix is either to remove "point-data" outputting or to test it properly (and fail gracefully if we try to use it in parallel).

I suggest we leave this issue open for Jemma to comment on before making a longer-term decision, but I would be happy for you to remove the "point-data" output from that example for now.

**other than bugs with it that Jack steadily finds xD

commented

PointDataOutput now raises an exception if you try to construct it in parallel:

gusto/gusto/io.py

Line 75 in 4882454

if self.comm.size > 1:

Non-hydrostatic Compressible Skamarock Klemp will print a warning if you run in parallel:

gusto/examples/compressible/skamarock_klemp_nonhydrostatic.py

Line 66 in 4882454

logger.warning(

Hopefully all the tests pass on the new PR 🤞

I will change the title of this issue to reflect why it's being kept open.

commented