ADIOS2 endpoint does not always work (vtkUnstructuredGrid)

[Victor Mateevitsi]

Hi,

I was testing the ADIOS2 transport adaptor for unstructured grids (ecp_adios_hdf5_hardening_part_2 branch), and discovered that depending on the input ADIOS2 might not work/crash.

The tests below are with a Nek5000 SENSEI instrumentation that passes a vtkUnstructuredGrid to SENSEI.

In particular, here are the tests I tried:

turbPipe dataset (one of the example cases of Nek5000)

• Run a simulation with the ADIOS2 analysis adaptor (BP4 engine)

<sensei>

  <!-- configure ADIOS2 write -->
  <analysis type="adios2" filename="adios2/test_%05d.bp" engine="BP4"
    debug_mode="0" enabled="1" steps_per_file="2" >

    <!-- ADIOS2 engine parameters -->
    <engine_parameters>
      NumAggregators = 0
      InitialBufferSize = 1Mb
      BufferGrowthFactor = 2
      MaxBufferSize = 1024Mb
      StatsLevel = 0
      Profile = Off
    </engine_parameters>

    <!-- subset by mesh and array -->
    <mesh name="mesh">
      <point_arrays> pressure, velocity_x, velocity_y, velocity_z </point_arrays>
    </mesh>

  </analysis>
</sensei>

Works and I can see the bp directories populated.

• Run the SENSEIEndPoint with a catalyst script:
  SENSEIEndPoint -a analysis.xml -t read_adios2_bp4.xml

analysis.xml

<sensei>
  <analysis type="catalyst" pipeline="pythonscript" filename="analysis.py" enabled="1" />
</sensei>

read_adios2_bp4.xml

<sensei>
  <transport type="adios2" filename="/lus/grand/projects/visualization/mvictoras/src/Nek5000/run/DISI_2_SENSEI_ADIOS2_128/adios2/test_%05d.bp" debug_mode="1" steps_per_file="2" engine="bp4">
    <engine_parameters>
        NumAggregators = 0
        InitialBufferSize = 1Mb
        BufferGrowthFactor = 2
        MaxBufferSize = 1024Mb
        StatsLevel = 0
        Profile = Off
      </engine_parameters>

      <!-- subset by mesh and array -->
      <mesh name="mesh">
        <point_arrays> pressure, velocity_x, velocity_y, velocity_z </point_arrays>
      </mesh>
    </transport>
</sensei>

Works and I can see the correct images rendered. Note: The catalyst script has also been tested by running the simulation with the catalyst adaptor.

real big dataset (this is a real dataset from one of our collaborators)

To give you a sense of scale, I am running this on 128 nodes (8192 ranks) and each Nek5000 checkpoint is 28GB in size.

• Run a simulation with the ADIOS2 analysis adaptor (BP4 engine)

<sensei>

  <!-- configure ADIOS2 write -->
  <analysis type="adios2" filename="adios2/test_%05d.bp" engine="BP4"
    debug_mode="0" enabled="1" steps_per_file="2" >

    <!-- ADIOS2 engine parameters -->
    <engine_parameters>
      NumAggregators = 0
      InitialBufferSize = 1Mb
      BufferGrowthFactor = 2
      MaxBufferSize = 1024Mb
      StatsLevel = 0
      Profile = Off
    </engine_parameters>

    <!-- subset by mesh and array -->
    <mesh name="mesh">
      <point_arrays> pressure, velocity_x, velocity_y, velocity_z </point_arrays>
    </mesh>

  </analysis>
</sensei>

Works and I can see the bp directories populated.

• Run the SENSEIEndPoint with a catalyst script:
  SENSEIEndPoint -a catalyst.xml -t read_adios2_bp4.xml

analysis.xml

<sensei>
  <analysis type="catalyst" pipeline="pythonscript" filename="analysis.py" enabled="1" />
</sensei>

read_adios2_bp4.xml

<sensei>
  <transport type="adios2" filename="/lus/grand/projects/visualization/mvictoras/src/Nek5000/run/DISI_2_SENSEI_ADIOS2_128/adios2/test_%05d.bp" debug_mode="1" steps_per_file="2" engine="bp4">
    <engine_parameters>
        NumAggregators = 0
        InitialBufferSize = 1Mb
        BufferGrowthFactor = 2
        MaxBufferSize = 1024Mb
        StatsLevel = 0
        Profile = Off
      </engine_parameters>

      <!-- subset by mesh and array -->
      <mesh name="mesh">
        <point_arrays> pressure, velocity_x, velocity_y, velocity_z </point_arrays>
      </mesh>
    </transport>
</sensei>

Runs with no errors, but the rendered image is blank. I can see the axis and the camera position is correct, but the rendered data is missing.

If I switch to PosthocIO analysis, then I get errors:

<sensei>
  <analysis type="PosthocIO"
    output_dir="vtk" file_name="output" mode="paraview" enabled="1">
    <mesh name="mesh">
      <point_arrays> pressure, velocity_x, velocity_y, velocity_z </point_arrays>
    </mesh>
  </analysis>
</sensei>

2021-09-10 15:08:10.954 ( 175.626s) [        AABFA640]       vtkExecutive.cxx:753    ERR| vtkCompositeDataPipeline (0x125a6c0): Algorithm vtkXMLUnstructuredGridWriter(0x901db0) returned failure for request: vtkInformation (0x126b640)
  Debug: Off
  Modified Time: 110885
  Reference Count: 1
  Registered Events: (none)
  Request: REQUEST_DATA
  FROM_OUTPUT_PORT: -1
  ALGORITHM_AFTER_FORWARD: 1
  FORWARD_DIRECTION: 0

[Burlen Loring]

@mvictoras Thanks for documenting this. There is a lot of information here. What's clear from the report, is that there is something preventing the post hoc IO adaptor from working. I think we should focus on that since this would flesh out potential bugs in the nek adaptor and metadata it's providing. with the output vtk files we can validate the Catalyst Python script. Once we track these issues down we can focus on the ADIOS side. Does that make sense?

[Victor Mateevitsi]

Absolutely!

[Burlen Loring]

@mvictoras was this ever resolved?

[Victor Mateevitsi]

Yes it has! At least for v3 it has, have not tested for v4 yet.

[Burlen Loring]

nice! also for future reference of anyone finding this issue : we can add some error reports to the ADIOS2 code since we store the lengths of all the writes as described by the metadata. we could report when these don't match the vtkDataArray length passed to the write call.

[Burlen Loring]

Closing this, however, additional error checking mentioned in the above comment is still an open issue.