Simulation crashes when it runs in mpi mode after decomposePar

Question

Simulation crashes when it runs in mpi mode after decomposePar

TimoLin opened this issue 4 years ago · comments

Creating field dpdt

Creating field kinetic energy K

No finite volume options present

Courant Number mean: 8.26212e-07 max: 0.214792

PIMPLE: Operating solver in PISO mode


Starting time loop

Courant Number mean: 8.26212e-07 max: 0.214792
Time = 1e-06

diagonal:  Solving for rho, Initial residual = 0, Final residual = 0, No Iterations 0
HDF5-DIAG: Error detected in HDF5 (1.8.4) thread 0:
  #000: /home/hdftest/snapshots-hdf5_1_8_4/current/src/H5F.c line 1499 in H5Fopen(): invalid file name
    major: Invalid arguments to routine
    minor: Bad value
HDF5-DIAG: Error detected in HDF5 (1.8.4) thread 0:
  #000: /home/hdftest/snapshots-hdf5_1_8_4/current/src/H5D.c line 303 in H5Dopen2(): not a location
    major: Invalid arguments to routine
    minor: Inappropriate type
  #001: /home/hdftest/snapshots-hdf5_1_8_4/current/src/H5Gloc.c line 241 in H5G_loc(): invalid object ID
    major: Invalid arguments to routine
    minor: Bad value
HDF5-DIAG: Error detected in HDF5 (1.8.4) thread 0:
  #000: /home/hdftest/snapshots-hdf5_1_8_4/current/src/H5D.c line 303 in H5Dopen2(): not a location
    major: Invalid arguments to routine
    minor: Inappropriate type
  #001: /home/hdftest/snapshots-hdf5_1_8_4/current/src/H5Gloc.c line 241 in H5G_loc(): invalid object ID
    major: Invalid arguments to routine
    minor: Bad value
HDF5-DIAG: Error detected in HDF5 (1.8.4) thread 0:
  #000: /home/hdftest/snapshots-hdf5_1_8_4/current/src/H5Dio.c line 136 in H5Dread(): not a dataset
    major: Invalid arguments to routine
    minor: Inappropriate type
HDF5-DIAG: Error detected in HDF5 (1.8.4) thread 0:
  #000: /home/hdftest/snapshots-hdf5_1_8_4/current/src/H5D.c line 378 in H5Dclose(): not a dataset
    major: Invalid arguments to routine
    minor: Inappropriate type
HDF5-DIAG: Error detected in HDF5 (1.8.4) thread 0:
  #000: /home/hdftest/snapshots-hdf5_1_8_4/current/src/H5F.c line 1499 in H5Fopen(): invalid file name
    major: Invalid arguments to routine
    minor: Bad value
HDF5-DIAG: Error detected in HDF5 (1.8.4) thread 0:
  #000: /home/hdftest/snapshots-hdf5_1_8_4/current/src/H5D.c line 303 in H5Dopen2(): not a location
    major: Invalid arguments to routine
    minor: Inappropriate type
  #001: /home/hdftest/snapshots-hdf5_1_8_4/current/src/H5Gloc.c line 241 in H5G_loc(): invalid object ID
    major: Invalid arguments to routine
    minor: Bad value
HDF5-DIAG: Error detected in HDF5 (1.8.4) thread 0:
  #000: /home/hdftest/snapshots-hdf5_1_8_4/current/src/H5D.c line 303 in H5Dopen2(): not a location
[3] 
[3] 
[3] --> FOAM FATAL ERROR: 
[3] bad size -671088640
[3] 
[3]     From function HDF5-DIAG: Error detected in HDF5 (1.8.4) thread 0:
  #000: /home/hdftest/snapshots-hdf5_1_8_4/current/src/H5F.c line 1499 in H5Fopen(): invalid file name
    major: Invalid arguments to routine
    minor: Bad value
HDF5-DIAG: Error detected in HDF5 (1.8.4) thread 0:
  #000: /home/hdftest/snapshots-hdf5_1_8_4/current/src/H5D.c line 303 in H5Dopen2(): not a location
    major: Invalid arguments to routine
    minor: Inappropriate type
  #001: /home/hdftest/snapshots-hdf5_1_8_4/current/src/H5Gloc.c line 241 in H5G_loc(): invalid object ID
    major: Invalid arguments to routine
    minor: Bad value
    major: Invalid arguments to routine
    minor: Inappropriate type
  #001: /home/hdftest/snapshots-hdf5_1_8_4/current/src/H5Gloc.c line 241 in H5G_loc(): invalid object ID
    major: Invalid arguments to routine
    minor: Bad value
HDF5-DIAG: Error detected in HDF5 (1.8.4) thread 0:
  #000: /home/hdftest/snapshots-hdf5_1_8_4/current/src/H5Dio.c line 136 in H5Dread(): not a dataset
    major: Invalid arguments to routine
    minor: Inappropriate type
HDF5-DIAG: Error detected in HDF5 (1.8.4) thread 0:
  #000: /home/hdftest/snapshots-hdf5_1_8_4/current/src/H5D.c line 378 in H5Dclose(): not a dataset
    major: Invalid arguments to routine
    minor: Inappropriate type
List<T>::List(const label size)
[3]     in file /home/zt/OpenFOAM/OpenFOAM-2.3.1/src/OpenFOAM/lnInclude/List.C at line 49.
[3] 
FOAM parallel run aborting
[3] 
HDF5-DIAG: Error detected in HDF5 (1.8.4) thread 0:
  #000: /home/hdftest/snapshots-hdf5_1_8_4/current/src/H5D.c line 303 in H5Dopen2(): not a location
    major: Invalid arguments to routine
    minor: Inappropriate type
  #001: /home/hdftest/snapshots-hdf5_1_8_4/current/src/H5Gloc.c line 241 in H5G_loc(): invalid object ID
    major: Invalid arguments to routine
    minor: Bad value
HDF5-DIAG: Error detected in HDF5 (1.8.4) thread 0:
  #000: /home/hdftest/snapshots-hdf5_1_8_4/current/src/H5Dio.c line 136 in H5Dread(): not a dataset
    major: Invalid arguments to routine
    minor: Inappropriate type
HDF5 error when opening points dataset
HDF5 error when closing points dataset
precursorHDF5FvPatchField : Read 0 points
[0] 
[0] 
[0] --> FOAM FATAL ERROR: 
[0] Only 0 provided.
Need at least three non-colinear points to be able to interpolate.
[0] 
[0]     From function pointToPointPlanarInterpolation::calcCoordinateSystem(const pointField&)
[0]     in file triSurface/triSurfaceTools/pointToPointPlanarInterpolation.C at line 58.
[0] 
FOAM parallel run exiting
[0] 
HDF5-DIAG: Error detected in HDF5 (1.8.4) thread 0:
  #000: /home/hdftest/snapshots-hdf5_1_8_4/current/src/H5D.c line 378 in H5Dclose(): not a dataset
    major: Invalid arguments to routine
    minor: Inappropriate type
[2] 
[2] 
[2] --> FOAM FATAL ERROR: 
[2] bad size -1717986918
[2] 
[2]     From function List<T>::List(const label size)
[2]     in file /home/zt/OpenFOAM/OpenFOAM-2.3.1/src/OpenFOAM/lnInclude/List.C at line 49.
[2] 
FOAM parallel run aborting
[2] 
HDF5-DIAG: Error detected in HDF5 (1.8.4) thread 0:
  #000: /home/hdftest/snapshots-hdf5_1_8_4/current/src/H5F.c line 1499 in H5Fopen(): invalid file name
    major: Invalid arguments to routine
    minor: Bad value
HDF5-DIAG: Error detected in HDF5 (1.8.4) thread 0:
  #000: /home/hdftest/snapshots-hdf5_1_8_4/current/src/H5D.c line 303 in H5Dopen2(): not a location
    major: Invalid arguments to routine
    minor: Inappropriate type
  #001: /home/hdftest/snapshots-hdf5_1_8_4/current/src/H5Gloc.c line 241 in H5G_loc(): invalid object ID
    major: Invalid arguments to routine
    minor: Bad value
HDF5-DIAG: Error detected in HDF5 (1.8.4) thread 0:
  #000: /home/hdftest/snapshots-hdf5_1_8_4/current/src/H5D.c line 303 in H5Dopen2(): not a location
    major: Invalid arguments to routine
    minor: Inappropriate type
  #001: /home/hdftest/snapshots-hdf5_1_8_4/current/src/H5Gloc.c line 241 in H5G_loc(): invalid object ID
    major: Invalid arguments to routine
    minor: Bad value
HDF5-DIAG: Error detected in HDF5 (1.8.4) thread 0:
  #000: /home/hdftest/snapshots-hdf5_1_8_4/current/src/H5Dio.c line 136 in H5Dread(): not a dataset
    major: Invalid arguments to routine
    minor: Inappropriate type
HDF5-DIAG: Error detected in HDF5 (1.8.4) thread 0:
  #000: /home/hdftest/snapshots-hdf5_1_8_4/current/src/H5D.c line 378 in H5Dclose(): not a dataset
    major: Invalid arguments to routine
    minor: Inappropriate type
[3] #0  Foam::error::printStack(Foam::Ostream&)[2] #0  Foam::error::printStack(Foam::Ostream&)--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
 at ??:?
[3] #1  Foam::error::abort() at ??:?
[2] #1  Foam::error::abort() at ??:

Timo Zhang · Answer 1 · Mon Jun 22 2020 22:18:17 GMT+0800 (China Standard Time)

Like in this issue #2 , the error message shows the HDF5 file name is invalid file name. Therefore, the bug may occurs in the write procedure when decomposePar writes out boundary files.
Check file processor0/0/U, locate to precursorHDF5 and it shows:

    FUEL
    {
        type            precursorHDF5;
        setAverage      0;
        perturb         0;
        mapMethod       nearest;
        recycling       0;
        offset          (0 0 0);
        hdf5FileName    ;
        hdf5PointsDatasetName points;
        hdf5SampleTimesDatasetName ;
        hdf5FieldValuesDatasetName ;
        value           nonuniform 0();
    }

Timo Zhang · Answer 2 · Mon Jun 22 2020 22:24:28 GMT+0800 (China Standard Time)

The hdf5 related options' values are missed. After some debug, I found that the precursorHDF5FvPatchField has 3 constructors (except clone or copy):

        //- Construct from patch and internal field
        precursorHDF5FvPatchField
        (
            const fvPatch&,
            const DimensionedField<Type, volMesh>&
        );

        //- Construct from patch, internal field and dictionary
        precursorHDF5FvPatchField
        (
            const fvPatch&,
            const DimensionedField<Type, volMesh>&,
            const dictionary&
        );

        //- Construct by mapping given precursorHDF5FvPatchField
        //  onto a new patch
        precursorHDF5FvPatchField
        (
            const precursorHDF5FvPatchField<Type>&,
            const fvPatch&,
            const DimensionedField<Type, volMesh>&,
            const fvPatchFieldMapper&
        );

When decomposePar wirtes out, it uses the second one which is:

template<class Type>
precursorHDF5FvPatchField<Type>::
precursorHDF5FvPatchField
(
    const precursorHDF5FvPatchField<Type>& ptf,
    const fvPatch& p,
    const DimensionedField<Type, volMesh>& iF,
    const fvPatchFieldMapper& mapper
)
:
    fixedValueFvPatchField<Type>(ptf, p, iF, mapper),
    fieldTableName_(ptf.fieldTableName_),
    setAverage_(ptf.setAverage_),
    perturb_(ptf.perturb_),
    mapMethod_(ptf.mapMethod_),
    recycling_(ptf.recycling_),
    mapperPtr_(NULL),
    sampleTimes_(0),
    startSampleTime_(-1),
    startSampledValues_(0),
    startAverage_(pTraits<Type>::zero),
    endSampleTime_(-1),
    endSampledValues_(0),
    endAverage_(pTraits<Type>::zero),
    offset_
    (
        ptf.offset_.valid()
      ? ptf.offset_().clone().ptr()
      : NULL
    ),
    hdf5PointsDatasetName_("points")
{}

This means, the values of hdf5FileName_, hdf5PointsDatasetName_, hdf5SampleTimesDatasetName_, hdf5FieldValuesDatasetName_ defined at 0/U is not used when construct this patchField. Therefore, just change it to:

template<class Type>
...
    hdf5FileName_(ptf.hdf5FileName_),
    hdf5PointsDatasetName_(ptf.hdf5PointsDatasetName_),
    hdf5SampleTimesDatasetName_(ptf.hdf5SampleTimesDatasetName_),
    hdf5FieldValuesDatasetName_(ptf.hdf5FieldValuesDatasetName_)
{}
`

Timo Zhang · Answer 3 · Thu Mar 31 2022 14:29:37 GMT+0800 (China Standard Time)

Flag `recycling` is not correctly written in the checkpoint

    FUEL
    {
        type            precursorHDF5;
        setAverage      0;
        perturb         0;
        mapMethod       nearest;
        recycling       72;
        offset          (0 0 0);
        hdf5FileName    sfd-20k.hdf5;
        hdf5PointsDatasetName points;
        hdf5SampleTimesDatasetName times;
        hdf5FieldValuesDatasetName velocity;
        value           nonuniform List<vector> 
     ...

Simulation crashes when it runs in mpi mode after decomposePar

Flag recycling is not correctly written in the checkpoint

Flag `recycling` is not correctly written in the checkpoint