ADIOS2 Tests Failing with ADIOS2 2.9.0

Question

ADIOS2 Tests Failing with ADIOS2 2.9.0

willdunklin opened this issue a year ago · comments

From my CI work I've noticed that building SENSEI with ADIOS2 2.9.0 causes the following tests to fail:

- testADIOS2SST
- testPartitionersADIOS2SST

Both tests seem to be segfaulting with 2.9.0, but work fine with 2.8.3.

Let me know if logs of the errors would be helpful. I also have a docker test image that has ADIOS2 2.9.0 installed if that would help with reproducibility. You can find the docker image here, make sure to mount your SENSEI source to /sensei/src in the docker container.

Will Dunklin · Answer 1 · Mon Jun 19 2023 02:43:25 GMT+0800 (China Standard Time)

@burlen not sure if you've hit these issues before as well

Will Dunklin · Answer 2 · Tue Jun 20 2023 22:29:25 GMT+0800 (China Standard Time)

Just for reference, this is the output I'm getting when running ADIOS2SST test with ADIOS2 2.9.0 on the testing container.

root@9cb2fee5a080:/sensei/src/.github/ci# ctest -VV -S ctest_test.cmake -R ADIOS2SST
...
test 15
    Start 15: testADIOS2SST

15: Test command: /sensei/src/sensei/testing/testADIOS2.sh "/opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/openmpi-4.1.5-l7f7aw6einsykqbqqouclherjvxfmz5p/bin/mpiexec" "-n" "2" "/opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/bin/python3" "/sensei/src/sensei/testing" "testADIOS2SST.bp" "SST" "SST" "3" "--" "--allow-run-as-root"
15: Working Directory: /sensei/build/bin
15: Environment variables: 
15:  MPI_PROCESSES=2
15:  MPI_PROCESSES_HALF=1
15:  MPI_LAUNCHER=/opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/openmpi-4.1.5-l7f7aw6einsykqbqqouclherjvxfmz5p/bin/mpiexec -n 2 --allow-run-as-root
15:  MPIEXEC_POSTFLAGS=
15: Environment variable modifications: 
15:  PYTHONPATH=path_list_prepend:/sensei/build/lib/python3.10/site-packages/
15: Test timeout computed to be: 100
15: rm -rfv testADIOS2SST.bp
15: echo testing SST -> SST
15: testing SST -> SST
15: echo M=2 x N=1
15: M=2 x N=1
15: mpiexec --allow-run-as-root -n 2 /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/bin/python3 -m mpi4py /sensei/src/sensei/testing/testADIOS2Write.py SST testADIOS2SST.bp 0 3
15: writePid=5652
15: [[ SST == BP4 ]]
15: mpiexec --allow-run-as-root -n 1 /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/bin/python3 -m mpi4py /sensei/src/sensei/testing/testADIOS2Read.py SST testADIOS2SST.bp
15: STATUS[0] : initializing ADIOS2DataAdaptor file=testADIOS2SST.bp engine=SST
15: Errno was 111
15: [Tue Jun 20 13:59:03 2023] [ADIOS2 ERROR] <Helper> <adiosSystem> <ExceptionToError> : adios2_open: [Tue Jun 20 13:59:03 2023] [ADIOS2 EXCEPTION] <Engine> <SstReader> <SstReader> : SstReader did not find active Writer contact info in file "testADIOS2SST.bp.sst".  Timeout or non-current SST contact file?
15: 
15: ERROR: [0][/sensei/src/sensei/ADIOS2Schema.cxx:529][v4.1.0-41-ge8c2ae8]
15: ERROR: adios2_open "testADIOS2SST.bp" for reading failed
15: STATUS[0] : received step 0 time 0.0
15: [Tue Jun 20 13:59:03 2023] [ADIOS2 ERROR] <Helper> <adiosSystem> <ExceptionToError> : adios2_end_step: [Tue Jun 20 13:59:03 2023] [ADIOS2 EXCEPTION] <Helper> <adiosType> <CheckForNullPtr> : found null pointer for adios2_engine, in call to adios2_end_step
15: 
15: ERROR: [0][/sensei/src/sensei/ADIOS2Schema.cxx:591][v4.1.0-41-ge8c2ae8]
15: ERROR: adios2_end_step failed
15: STATUS[0] : closed stream after receiving 1 steps
15: STATUS[0] : initializing the SVTKDataAdaptor step 0 time 0.0
15: STATUS[0] : executing ADIOS2AnalysisAdaptor SST step 0 time 0.0
15: WARNING: [0][/sensei/src/sensei/ADIOS2AnalysisAdaptor.cxx:217][v4.1.0-41-ge8c2ae8]
15: WARNING: No subset specified. Writing all available data
15: rm -rfv testADIOS2SST.bp
15: exit 0
1/2 Test #15: testADIOS2SST ....................***Timeout 100.10 sec
test 17
    Start 17: testPartitionersADIOS2SST

17: Test command: /sensei/src/sensei/testing/testPartitionersDriver.sh "/opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/bin/python3" "/opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/openmpi-4.1.5-l7f7aw6einsykqbqqouclherjvxfmz5p/bin/mpiexec" "-n" "2" "1" "/sensei/src/sensei/testing" "/sensei/build/sensei/testing" "adios2_sst" "3" "3" "3" "1" "--" "--allow-run-as-root"
17: Working Directory: /sensei/build/bin
17: Environment variables: 
17:  MPI_PROCESSES=2
17:  MPI_PROCESSES_HALF=1
17:  MPI_LAUNCHER=/opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/openmpi-4.1.5-l7f7aw6einsykqbqqouclherjvxfmz5p/bin/mpiexec -n 2 --allow-run-as-root
17:  MPIEXEC_POSTFLAGS=
17: Environment variable modifications: 
17:  PYTHONPATH=path_list_prepend:/sensei/build/lib/python3.10/site-packages/
17: Test timeout computed to be: 400
17: find /sensei/src/sensei/testing -name read_adios2_sst*.xml -exec basename \{\} \; | sed s/read_adios2_sst_//g | cut -d. -f1 | sort
17: Testing adios2_sst M=2 N=1 part_M=block part_N=block ... 
17: file=test.bp rm -rf test.bp test.bp_writer_info.txt test.bp.sst rm -rf test_*.bp export PROFILER_ENABLE=2 PROFILER_LOG_FILE=WriterTimes.csv MEMPROF_LOG_FILE=WriterMemProf.csv mpiexec -n 2 /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/bin/python3 -m mpi4py /sensei/src/sensei/testing/testPartitionersWrite.py /sensei/src/sensei/testing/write_adios2_sst.xml 3 3 3 16 16 -6.2832 6.2832 -6.2832 6.2832 0 6.2832 writePid=5694 [[ 1 == 0 ]] [[ 1 == 1 ]] echo waiting for writer to start 1 waiting for writer to start 1 [[ -n True ]] [[ -e test.bp_writer_info.txt [[ 30 -le 0 ]] sleep 1s STATUS[0] : running nIts=3 with mesh parametersnblks=[3, 3] nx=[16, 16] dom=[-6.2832, 6.2832, -6.2832, 6.2832] dx=[0.2618, 0.2618] theta=[0.0, 6.2832] dt=3.1416 STATUS: [0][/sensei/src/sensei/ADIOS2AnalysisAdaptor.cxx:300][v4.1.0-41-ge8c2ae8] STATUS: Configured ADIOSAnalysisAdaptor filename="test.bp" engine=sst WARNING: [0][/sensei/src/sensei/ADIOS2AnalysisAdaptor.cxx:217][v4.1.0-41-ge8c2ae8] WARNING: No subset specified. Writing all available data let maxDelay=30-1 [[ -n True ]] [[ -e test.bp_writer_info.txt break export PROFILER_ENABLE=2 TIMER_LOG_FILE=ReaderTimes.csv MEMPROF_LOG_FILE=ReaderMemProf.csv mpiexec -n 1 /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/bin/python3 -m mpi4py /sensei/src/sensei/testing/testPartitionersRead.py /sensei/build/sensei/testing/catalyst_render_partition.xml /sensei/src/sensei/testing/read_adios2_sst_block.xml STATUS[0] : initializing the transport layer STATUS: [0][/sensei/src/sensei/Partitioner.h:39][v4.1.0-41-ge8c2ae8] STATUS: Configured BlockPartitioner STATUS: [0][/sensei/src/sensei/ConfigurableInTransitDataAdaptor.cxx:151][v4.1.0-41-ge8c2ae8] STATUS: Configured "ADIOS2DataAdaptor [9cb2fee5a080:05699] SENSEIEndPoint h5stream.n2 h5stream.n2_s0 h5stream.n2_s1 h5stream.n2_s2 h5stream.n2_s3 h5test.n2 hist_mesh_values_0.txt hist_mesh_values_1.txt hist_mesh_values_2.txt hist_mesh_values_3.txt hist_mesh_values_4.txt hist_ucdmesh_data_0.txt hist_ucdmesh_data_1.txt hist_ucdmesh_data_2.txt hist_ucdmesh_data_3.txt hist_ucdmesh_data_4.txt hist_ucdmesh_data_5.txt hist_ucdmesh_data_6.txt hist_ucdmesh_data_7.txt hist_ucdmesh_data_8.txt hist_ucdmesh_data_9.txt mandelbrot mesh_pvd_mandelbrot mesh_pvd_oscillator mesh_visit_mandelbrot mesh_visit_oscillator oscillator oscillator_iso_pipeline_mesh pcolor_0000.png pcolor_0001.png pcolor_0002.png pcolor_0003.png pcolor_0004.png pv_out sensei-env.sh simpleTestDriver test.bp.sst test.h5 testADIOS2BP4_00000.bp testADIOS2BP4_00001.bp testADIOS2SST.bp.sst testHDF5 testHistogram testProgrammableDataAdaptor visit_out Process received signal SENSEIEndPoint h5stream.n2 h5stream.n2_s0 h5stream.n2_s1 h5stream.n2_s2 h5stream.n2_s3 h5test.n2 hist_mesh_values_0.txt hist_mesh_values_1.txt hist_mesh_values_2.txt hist_mesh_values_3.txt hist_mesh_values_4.txt hist_ucdmesh_data_0.txt hist_ucdmesh_data_1.txt hist_ucdmesh_data_2.txt hist_ucdmesh_data_3.txt hist_ucdmesh_data_4.txt hist_ucdmesh_data_5.txt hist_ucdmesh_data_6.txt hist_ucdmesh_data_7.txt hist_ucdmesh_data_8.txt hist_ucdmesh_data_9.txt mandelbrot mesh_pvd_mandelbrot mesh_pvd_oscillator mesh_visit_mandelbrot mesh_visit_oscillator oscillator oscillator_iso_pipeline_mesh pcolor_0000.png pcolor_0001.png pcolor_0002.png pcolor_0003.png pcolor_0004.png pv_out sensei-env.sh simpleTestDriver test.bp.sst test.h5 testADIOS2BP4_00000.bp testADIOS2BP4_00001.bp testADIOS2SST.bp.sst testHDF5 testHistogram testProgrammableDataAdaptor visit_out [9cb2fee5a080:05699] Signal: Segmentation fault (11) [9cb2fee5a080:05699] Signal code: Address not mapped (1) [9cb2fee5a080:05699] Failing at address: (nil) [9cb2fee5a080:05700] SENSEIEndPoint h5stream.n2 h5stream.n2_s0 h5stream.n2_s1 h5stream.n2_s2 h5stream.n2_s3 h5test.n2 hist_mesh_values_0.txt hist_mesh_values_1.txt hist_mesh_values_2.txt hist_mesh_values_3.txt hist_mesh_values_4.txt hist_ucdmesh_data_0.txt hist_ucdmesh_data_1.txt hist_ucdmesh_data_2.txt hist_ucdmesh_data_3.txt hist_ucdmesh_data_4.txt hist_ucdmesh_data_5.txt hist_ucdmesh_data_6.txt hist_ucdmesh_data_7.txt hist_ucdmesh_data_8.txt hist_ucdmesh_data_9.txt mandelbrot mesh_pvd_mandelbrot mesh_pvd_oscillator mesh_visit_mandelbrot mesh_visit_oscillator oscillator oscillator_iso_pipeline_mesh pcolor_0000.png pcolor_0001.png pcolor_0002.png pcolor_0003.png pcolor_0004.png pv_out sensei-env.sh simpleTestDriver test.bp.sst test.h5 testADIOS2BP4_00000.bp testADIOS2BP4_00001.bp testADIOS2SST.bp.sst testHDF5 testHistogram testProgrammableDataAdaptor visit_out Process received signal SENSEIEndPoint h5stream.n2 h5stream.n2_s0 h5stream.n2_s1 h5stream.n2_s2 h5stream.n2_s3 h5test.n2 hist_mesh_values_0.txt hist_mesh_values_1.txt hist_mesh_values_2.txt hist_mesh_values_3.txt hist_mesh_values_4.txt hist_ucdmesh_data_0.txt hist_ucdmesh_data_1.txt hist_ucdmesh_data_2.txt hist_ucdmesh_data_3.txt hist_ucdmesh_data_4.txt hist_ucdmesh_data_5.txt hist_ucdmesh_data_6.txt hist_ucdmesh_data_7.txt hist_ucdmesh_data_8.txt hist_ucdmesh_data_9.txt mandelbrot mesh_pvd_mandelbrot mesh_pvd_oscillator mesh_visit_mandelbrot mesh_visit_oscillator oscillator oscillator_iso_pipeline_mesh pcolor_0000.png pcolor_0001.png pcolor_0002.png pcolor_0003.png pcolor_0004.png pv_out sensei-env.sh simpleTestDriver test.bp.sst test.h5 testADIOS2BP4_00000.bp testADIOS2BP4_00001.bp testADIOS2SST.bp.sst testHDF5 testHistogram testProgrammableDataAdaptor visit_out [9cb2fee5a080:05700] Signal: Segmentation fault (11) [9cb2fee5a080:05700] Signal code: Address not mapped (1) [9cb2fee5a080:05700] Failing at address: (nil) [9cb2fee5a080:05699] [ 0] [9cb2fee5a080:05700] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x43090)[0x7ff721f50090] [9cb2fee5a080:05699] [ 1] /lib/x86_64-linux-gnu/libc.so.6(+0x43090)[0x7f383b601090] [9cb2fee5a080:05700] [ 1] /lib/x86_64-linux-gnu/libc.so.6(+0x18b767)[0x7ff722098767] [9cb2fee5a080:05699] [ 2] /lib/x86_64-linux-gnu/libc.so.6(+0x18b767)[0x7f383b749767] [9cb2fee5a080:05700] [ 2] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/adios2-2.9.0-45plavvq7b3lnmhfao3r3xl74x5plswo/lib/../lib/libadios2_core.so.2(_ZN6adios26format13BP5Serializer23CollectFinalShapeValuesEv+0xc7)[0x7ff70c6be377] [9cb2fee5a080:05699] [ 3] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/adios2-2.9.0-45plavvq7b3lnmhfao3r3xl74x5plswo/lib/../lib/libadios2_core.so.2(_ZN6adios26format13BP5Serializer23CollectFinalShapeValuesEv+0xc7)[0x7f3825d10377] [9cb2fee5a080:05700] [ 3] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/adios2-2.9.0-45plavvq7b3lnmhfao3r3xl74x5plswo/lib/../lib/libadios2_core.so.2(_ZN6adios26format13BP5Serializer13CloseTimestepEib+0x30b)[0x7ff70c6c028b] [9cb2fee5a080:05699] [ 4] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/adios2-2.9.0-45plavvq7b3lnmhfao3r3xl74x5plswo/lib/../lib/libadios2_core.so.2(_ZN6adios26format13BP5Serializer13CloseTimestepEib+0x30b)[0x7f3825d1228b] [9cb2fee5a080:05700] [ 4] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/adios2-2.9.0-45plavvq7b3lnmhfao3r3xl74x5plswo/lib/../lib/libadios2_core.so.2(_ZN6adios24core6engine9SstWriter7EndStepEv+0x2ef)[0x7ff70c708a6f] [9cb2fee5a080:05699] [ 5] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/adios2-2.9.0-45plavvq7b3lnmhfao3r3xl74x5plswo/lib/libadios2_c.so.2(adios2_end_step+0xc0)[0x7ff718fe4cf0] [9cb2fee5a080:05699] [ 6] /sensei/build/lib/python3.10/site-packages/sensei/_senseiPython.so(_ZN6sensei21ADIOS2AnalysisAdaptor13WriteTimestepEmdRKSt6vectorISt10shared_ptrINS_12MeshMetadataEESaIS4_EERKS1_I16svtkSmartPointerI20svtkCompositeDataSetESaISB_EE+0x33c)[0x7ff71a7f6a4c] [9cb2fee5a080:05699] [ 7] /sensei/build/lib/python3.10/site-packages/sensei/_senseiPython.so(_ZN6sensei21ADIOS2AnalysisAdaptor7ExecuteEPNS_11DataAdaptorEPS2_+0x827)[0x7ff71a8115b7] [9cb2fee5a080:05699] [ 8] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/adios2-2.9.0-45plavvq7b3lnmhfao3r3xl74x5plswo/lib/../lib/libadios2_core.so.2(_ZN6adios24core6engine9SstWriter7EndStepEv+0x2ef)[0x7f3825d5aa6f] [9cb2fee5a080:05700] [ 5] /sensei/build/lib/python3.10/site-packages/sensei/_senseiPython.so(_ZN6sensei20ConfigurableAnalysis7ExecuteEPNS_11DataAdaptorEPS2_+0x86)[0x7ff71a793546] [9cb2fee5a080:05699] [ 9] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/adios2-2.9.0-45plavvq7b3lnmhfao3r3xl74x5plswo/lib/libadios2_c.so.2(adios2_end_step+0xc0)[0x7f38383eecf0] [9cb2fee5a080:05700] [ 6] /sensei/build/lib/python3.10/site-packages/sensei/_senseiPython.so(+0x18aa28)[0x7ff71a6e4a28] [9cb2fee5a080:05699] [10] /sensei/build/lib/python3.10/site-packages/sensei/_senseiPython.so(_ZN6sensei21ADIOS2AnalysisAdaptor13WriteTimestepEmdRKSt6vectorISt10shared_ptrINS_12MeshMetadataEESaIS4_EERKS1_I16svtkSmartPointerI20svtkCompositeDataSetESaISB_EE+0x33c)[0x7f38338bda4c] [9cb2fee5a080:05700] [ 7] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x112cf8)[0x7ff722216cf8] [9cb2fee5a080:05699] [11] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyObject_MakeTpCall+0x8c)[0x7ff7221cd57c] [9cb2fee5a080:05699] [12] /sensei/build/lib/python3.10/site-packages/sensei/_senseiPython.so(_ZN6sensei21ADIOS2AnalysisAdaptor7ExecuteEPNS_11DataAdaptorEPS2_+0x827)[0x7f38338d85b7] [9cb2fee5a080:05700] [ 8] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x8f84)[0x7ff722177e74] [9cb2fee5a080:05699] [13] /sensei/build/lib/python3.10/site-packages/sensei/_senseiPython.so(_ZN6sensei20ConfigurableAnalysis7ExecuteEPNS_11DataAdaptorEPS2_+0x86)[0x7f383385a546] [9cb2fee5a080:05700] [ 9] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7ff7222c0274] [9cb2fee5a080:05699] [14] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x5f7b)[0x7ff722174e6b] [9cb2fee5a080:05699] [15] /sensei/build/lib/python3.10/site-packages/sensei/_senseiPython.so(+0x18aa28)[0x7f38337aba28] [9cb2fee5a080:05700] [10] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7ff7222c0274] [9cb2fee5a080:05699] [16] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x112cf8)[0x7f383b8c7cf8] [9cb2fee5a080:05700] [11] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x749e)[0x7ff72217638e] [9cb2fee5a080:05699] [17] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(PyEval_EvalCode+0xd0)[0x7ff7222c00a0] [9cb2fee5a080:05699] [18] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyObject_MakeTpCall+0x8c)[0x7f383b87e57c] [9cb2fee5a080:05700] [12] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1b64bd)[0x7ff7222ba4bd] [9cb2fee5a080:05699] [19] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x8f84)[0x7f383b828e74] [9cb2fee5a080:05700] [13] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1134e3)[0x7ff7222174e3] [9cb2fee5a080:05699] [20] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7f383b971274] [9cb2fee5a080:05700] [14] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x749e)[0x7ff72217638e] [9cb2fee5a080:05699] [21] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x5f7b)[0x7f383b825e6b] [9cb2fee5a080:05700] [15] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7ff7222c0274] [9cb2fee5a080:05699] [22] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x749e)[0x7ff72217638e] [9cb2fee5a080:05699] [23] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7ff7222c0274] [9cb2fee5a080:05699] [24] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7f383b971274] [9cb2fee5a080:05700] [16] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x69b5)[0x7ff7221758a5] [9cb2fee5a080:05699] [25] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x749e)[0x7f383b82738e] [9cb2fee5a080:05700] [17] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7ff7222c0274] [9cb2fee5a080:05699] [26] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x69b5)[0x7ff7221758a5] [9cb2fee5a080:05699] [27] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(PyEval_EvalCode+0xd0)[0x7f383b9710a0] [9cb2fee5a080:05700] [18] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7ff7222c0274] [9cb2fee5a080:05699] [28] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x749e)[0x7ff72217638e] [9cb2fee5a080:05699] [29] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1b64bd)[0x7f383b96b4bd] [9cb2fee5a080:05700] [19] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7ff7222c0274] [9cb2fee5a080:05699] SENSEIEndPoint h5stream.n2 h5stream.n2_s0 h5stream.n2_s1 h5stream.n2_s2 h5stream.n2_s3 h5test.n2 hist_mesh_values_0.txt hist_mesh_values_1.txt hist_mesh_values_2.txt hist_mesh_values_3.txt hist_mesh_values_4.txt hist_ucdmesh_data_0.txt hist_ucdmesh_data_1.txt hist_ucdmesh_data_2.txt hist_ucdmesh_data_3.txt hist_ucdmesh_data_4.txt hist_ucdmesh_data_5.txt hist_ucdmesh_data_6.txt hist_ucdmesh_data_7.txt hist_ucdmesh_data_8.txt hist_ucdmesh_data_9.txt mandelbrot mesh_pvd_mandelbrot mesh_pvd_oscillator mesh_visit_mandelbrot mesh_visit_oscillator oscillator oscillator_iso_pipeline_mesh pcolor_0000.png pcolor_0001.png pcolor_0002.png pcolor_0003.png pcolor_0004.png pv_out sensei-env.sh simpleTestDriver test.bp.sst test.h5 testADIOS2BP4_00000.bp testADIOS2BP4_00001.bp testADIOS2SST.bp.sst testHDF5 testHistogram testProgrammableDataAdaptor visit_out End of error message SENSEIEndPoint h5stream.n2 h5stream.n2_s0 h5stream.n2_s1 h5stream.n2_s2 h5stream.n2_s3 h5test.n2 hist_mesh_values_0.txt hist_mesh_values_1.txt hist_mesh_values_2.txt hist_mesh_values_3.txt hist_mesh_values_4.txt hist_ucdmesh_data_0.txt hist_ucdmesh_data_1.txt hist_ucdmesh_data_2.txt hist_ucdmesh_data_3.txt hist_ucdmesh_data_4.txt hist_ucdmesh_data_5.txt hist_ucdmesh_data_6.txt hist_ucdmesh_data_7.txt hist_ucdmesh_data_8.txt hist_ucdmesh_data_9.txt mandelbrot mesh_pvd_mandelbrot mesh_pvd_oscillator mesh_visit_mandelbrot mesh_visit_oscillator oscillator oscillator_iso_pipeline_mesh pcolor_0000.png pcolor_0001.png pcolor_0002.png pcolor_0003.png pcolor_0004.png pv_out sensei-env.sh simpleTestDriver test.bp.sst test.h5 testADIOS2BP4_00000.bp testADIOS2BP4_00001.bp testADIOS2SST.bp.sst testHDF5 testHistogram testProgrammableDataAdaptor visit_out /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1134e3)[0x7f383b8c84e3] [9cb2fee5a080:05700] [20] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x749e)[0x7f383b82738e] [9cb2fee5a080:05700] [21] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7f383b971274] [9cb2fee5a080:05700] [22] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x749e)[0x7f383b82738e] [9cb2fee5a080:05700] [23] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7f383b971274] [9cb2fee5a080:05700] [24] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x69b5)[0x7f383b8268a5] [9cb2fee5a080:05700] [25] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7f383b971274] [9cb2fee5a080:05700] [26] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x69b5)[0x7f383b8268a5] [9cb2fee5a080:05700] [27] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7f383b971274] [9cb2fee5a080:05700] [28] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x749e)[0x7f383b82738e] [9cb2fee5a080:05700] [29] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7f383b971274] [9cb2fee5a080:05700] SENSEIEndPoint h5stream.n2 h5stream.n2_s0 h5stream.n2_s1 h5stream.n2_s2 h5stream.n2_s3 h5test.n2 hist_mesh_values_0.txt hist_mesh_values_1.txt hist_mesh_values_2.txt hist_mesh_values_3.txt hist_mesh_values_4.txt hist_ucdmesh_data_0.txt hist_ucdmesh_data_1.txt hist_ucdmesh_data_2.txt hist_ucdmesh_data_3.txt hist_ucdmesh_data_4.txt hist_ucdmesh_data_5.txt hist_ucdmesh_data_6.txt hist_ucdmesh_data_7.txt hist_ucdmesh_data_8.txt hist_ucdmesh_data_9.txt mandelbrot mesh_pvd_mandelbrot mesh_pvd_oscillator mesh_visit_mandelbrot mesh_visit_oscillator oscillator oscillator_iso_pipeline_mesh pcolor_0000.png pcolor_0001.png pcolor_0002.png pcolor_0003.png pcolor_0004.png pv_out sensei-env.sh simpleTestDriver test.bp.sst test.h5 testADIOS2BP4_00000.bp testADIOS2BP4_00001.bp testADIOS2SST.bp.sst testHDF5 testHistogram testProgrammableDataAdaptor visit_out End of error message SENSEIEndPoint h5stream.n2 h5stream.n2_s0 h5stream.n2_s1 h5stream.n2_s2 h5stream.n2_s3 h5test.n2 hist_mesh_values_0.txt hist_mesh_values_1.txt hist_mesh_values_2.txt hist_mesh_values_3.txt hist_mesh_values_4.txt hist_ucdmesh_data_0.txt hist_ucdmesh_data_1.txt hist_ucdmesh_data_2.txt hist_ucdmesh_data_3.txt hist_ucdmesh_data_4.txt hist_ucdmesh_data_5.txt hist_ucdmesh_data_6.txt hist_ucdmesh_data_7.txt hist_ucdmesh_data_8.txt hist_ucdmesh_data_9.txt mandelbrot mesh_pvd_mandelbrot mesh_pvd_oscillator mesh_visit_mandelbrot mesh_visit_oscillator oscillator oscillator_iso_pipeline_mesh pcolor_0000.png pcolor_0001.png pcolor_0002.png pcolor_0003.png pcolor_0004.png pv_out sensei-env.sh simpleTestDriver test.bp.sst test.h5 testADIOS2BP4_00000.bp testADIOS2BP4_00001.bp testADIOS2SST.bp.sst testHDF5 testHistogram testProgrammableDataAdaptor visit_out STATUS[0] : received mesh mesh with 2 arrays ('f_xyt', 'sin_x_sin_y') STATUS[0] : BlockIds=(0, 1, 2, 3, 4, 5, 6, 7, 8) STATUS[0] : SenderBlockOwner=(0, 0, 0, 0, 0, 1, 1, 1, 1) STATUS[0] : ReceiverBlockOwner=(0, 0, 0, 0, 0, 0, 0, 0, 0) -------------------------------------------------------------------------- Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted. -------------------------------------------------------------------------- STATUS: [0][/sensei/src/sensei/ConfigurableAnalysis.cxx:685][v4.1.0-41-ge8c2ae8] STATUS: Configured CatalystAnalysisAdaptor pythonscript /sensei/src/sensei/testing/catalyst_render_receiver_partition.py STATUS: [0][/sensei/src/sensei/ConfigurableAnalysis.cxx:685][v4.1.0-41-ge8c2ae8] STATUS: Configured CatalystAnalysisAdaptor pythonscript /sensei/src/sensei/testing/catalyst_render_sender_partition.py STATUS[0] : receiving data... ( 1.086s) [pvbatch ] v2_internals.py:171 WARN| Module 'catalyst_render_receiver_partition' missing Catalyst 'options', will use a default options object ( 1.088s) [pvbatch ]vtkCPPythonScriptV2Help:459 WARN| script may not depend on simulation data; is that expected? ( 1.090s) [pvbatch ] v2_internals.py:171 WARN| Module 'catalyst_render_sender_partition' missing Catalyst 'options', will use a default options object ( 1.090s) [pvbatch ]vtkCPPythonScriptV2Help:459 WARN| script may not depend on simulation data; is that expected? .[Tue Jun 20 14:00:45 2023] [ADIOS2 ERROR] <Helper> <adiosSystem> <ExceptionToError> : adios2_end_step: [Tue Jun 20 14:00:45 2023] [ADIOS2 EXCEPTION] <Engine> <SstReader> <EndStep> : EndStep() is called without a successful BeginStep()  ERROR: [0][/sensei/src/sensei/ADIOS2Schema.cxx:591][v4.1.0-41-ge8c2ae8] ERROR: adios2_end_step failed . STATUS[0] : completed after processing 2 steps test_stat=0 exit 0 -------------------------------------------------------------------------- mpiexec noticed that process rank 0 with PID 0 on node 9cb2fee5a080 exited on signal 11 (Segmentation fault). --------------------------------------------------------------------------
17: ERROR
17: 
17: file=test.bp rm -rf test.bp test.bp_writer_info.txt test.bp.sst rm -rf test_*.bp export PROFILER_ENABLE=2 PROFILER_LOG_FILE=WriterTimes.csv MEMPROF_LOG_FILE=WriterMemProf.csv mpiexec -n 2 /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/bin/python3 -m mpi4py /sensei/src/sensei/testing/testPartitionersWrite.py /sensei/src/sensei/testing/write_adios2_sst.xml 3 3 3 16 16 -6.2832 6.2832 -6.2832 6.2832 0 6.2832 writePid=5694 [[ 1 == 0 ]] [[ 1 == 1 ]] echo waiting for writer to start 1 waiting for writer to start 1 [[ -n True ]] [[ -e test.bp_writer_info.txt [[ 30 -le 0 ]] sleep 1s STATUS[0] : running nIts=3 with mesh parametersnblks=[3, 3] nx=[16, 16] dom=[-6.2832, 6.2832, -6.2832, 6.2832] dx=[0.2618, 0.2618] theta=[0.0, 6.2832] dt=3.1416 STATUS: [0][/sensei/src/sensei/ADIOS2AnalysisAdaptor.cxx:300][v4.1.0-41-ge8c2ae8] STATUS: Configured ADIOSAnalysisAdaptor filename="test.bp" engine=sst WARNING: [0][/sensei/src/sensei/ADIOS2AnalysisAdaptor.cxx:217][v4.1.0-41-ge8c2ae8] WARNING: No subset specified. Writing all available data let maxDelay=30-1 [[ -n True ]] [[ -e test.bp_writer_info.txt break export PROFILER_ENABLE=2 TIMER_LOG_FILE=ReaderTimes.csv MEMPROF_LOG_FILE=ReaderMemProf.csv mpiexec -n 1 /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/bin/python3 -m mpi4py /sensei/src/sensei/testing/testPartitionersRead.py /sensei/build/sensei/testing/catalyst_render_partition.xml /sensei/src/sensei/testing/read_adios2_sst_block.xml STATUS[0] : initializing the transport layer STATUS: [0][/sensei/src/sensei/Partitioner.h:39][v4.1.0-41-ge8c2ae8] STATUS: Configured BlockPartitioner STATUS: [0][/sensei/src/sensei/ConfigurableInTransitDataAdaptor.cxx:151][v4.1.0-41-ge8c2ae8] STATUS: Configured "ADIOS2DataAdaptor [9cb2fee5a080:05699] SENSEIEndPoint h5stream.n2 h5stream.n2_s0 h5stream.n2_s1 h5stream.n2_s2 h5stream.n2_s3 h5test.n2 hist_mesh_values_0.txt hist_mesh_values_1.txt hist_mesh_values_2.txt hist_mesh_values_3.txt hist_mesh_values_4.txt hist_ucdmesh_data_0.txt hist_ucdmesh_data_1.txt hist_ucdmesh_data_2.txt hist_ucdmesh_data_3.txt hist_ucdmesh_data_4.txt hist_ucdmesh_data_5.txt hist_ucdmesh_data_6.txt hist_ucdmesh_data_7.txt hist_ucdmesh_data_8.txt hist_ucdmesh_data_9.txt mandelbrot mesh_pvd_mandelbrot mesh_pvd_oscillator mesh_visit_mandelbrot mesh_visit_oscillator oscillator oscillator_iso_pipeline_mesh pcolor_0000.png pcolor_0001.png pcolor_0002.png pcolor_0003.png pcolor_0004.png pv_out sensei-env.sh simpleTestDriver test.bp.sst test.h5 testADIOS2BP4_00000.bp testADIOS2BP4_00001.bp testADIOS2SST.bp.sst testHDF5 testHistogram testProgrammableDataAdaptor visit_out Process received signal SENSEIEndPoint h5stream.n2 h5stream.n2_s0 h5stream.n2_s1 h5stream.n2_s2 h5stream.n2_s3 h5test.n2 hist_mesh_values_0.txt hist_mesh_values_1.txt hist_mesh_values_2.txt hist_mesh_values_3.txt hist_mesh_values_4.txt hist_ucdmesh_data_0.txt hist_ucdmesh_data_1.txt hist_ucdmesh_data_2.txt hist_ucdmesh_data_3.txt hist_ucdmesh_data_4.txt hist_ucdmesh_data_5.txt hist_ucdmesh_data_6.txt hist_ucdmesh_data_7.txt hist_ucdmesh_data_8.txt hist_ucdmesh_data_9.txt mandelbrot mesh_pvd_mandelbrot mesh_pvd_oscillator mesh_visit_mandelbrot mesh_visit_oscillator oscillator oscillator_iso_pipeline_mesh pcolor_0000.png pcolor_0001.png pcolor_0002.png pcolor_0003.png pcolor_0004.png pv_out sensei-env.sh simpleTestDriver test.bp.sst test.h5 testADIOS2BP4_00000.bp testADIOS2BP4_00001.bp testADIOS2SST.bp.sst testHDF5 testHistogram testProgrammableDataAdaptor visit_out [9cb2fee5a080:05699] Signal: Segmentation fault (11) [9cb2fee5a080:05699] Signal code: Address not mapped (1) [9cb2fee5a080:05699] Failing at address: (nil) [9cb2fee5a080:05700] SENSEIEndPoint h5stream.n2 h5stream.n2_s0 h5stream.n2_s1 h5stream.n2_s2 h5stream.n2_s3 h5test.n2 hist_mesh_values_0.txt hist_mesh_values_1.txt hist_mesh_values_2.txt hist_mesh_values_3.txt hist_mesh_values_4.txt hist_ucdmesh_data_0.txt hist_ucdmesh_data_1.txt hist_ucdmesh_data_2.txt hist_ucdmesh_data_3.txt hist_ucdmesh_data_4.txt hist_ucdmesh_data_5.txt hist_ucdmesh_data_6.txt hist_ucdmesh_data_7.txt hist_ucdmesh_data_8.txt hist_ucdmesh_data_9.txt mandelbrot mesh_pvd_mandelbrot mesh_pvd_oscillator mesh_visit_mandelbrot mesh_visit_oscillator oscillator oscillator_iso_pipeline_mesh pcolor_0000.png pcolor_0001.png pcolor_0002.png pcolor_0003.png pcolor_0004.png pv_out sensei-env.sh simpleTestDriver test.bp.sst test.h5 testADIOS2BP4_00000.bp testADIOS2BP4_00001.bp testADIOS2SST.bp.sst testHDF5 testHistogram testProgrammableDataAdaptor visit_out Process received signal SENSEIEndPoint h5stream.n2 h5stream.n2_s0 h5stream.n2_s1 h5stream.n2_s2 h5stream.n2_s3 h5test.n2 hist_mesh_values_0.txt hist_mesh_values_1.txt hist_mesh_values_2.txt hist_mesh_values_3.txt hist_mesh_values_4.txt hist_ucdmesh_data_0.txt hist_ucdmesh_data_1.txt hist_ucdmesh_data_2.txt hist_ucdmesh_data_3.txt hist_ucdmesh_data_4.txt hist_ucdmesh_data_5.txt hist_ucdmesh_data_6.txt hist_ucdmesh_data_7.txt hist_ucdmesh_data_8.txt hist_ucdmesh_data_9.txt mandelbrot mesh_pvd_mandelbrot mesh_pvd_oscillator mesh_visit_mandelbrot mesh_visit_oscillator oscillator oscillator_iso_pipeline_mesh pcolor_0000.png pcolor_0001.png pcolor_0002.png pcolor_0003.png pcolor_0004.png pv_out sensei-env.sh simpleTestDriver test.bp.sst test.h5 testADIOS2BP4_00000.bp testADIOS2BP4_00001.bp testADIOS2SST.bp.sst testHDF5 testHistogram testProgrammableDataAdaptor visit_out [9cb2fee5a080:05700] Signal: Segmentation fault (11) [9cb2fee5a080:05700] Signal code: Address not mapped (1) [9cb2fee5a080:05700] Failing at address: (nil) [9cb2fee5a080:05699] [ 0] [9cb2fee5a080:05700] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x43090)[0x7ff721f50090] [9cb2fee5a080:05699] [ 1] /lib/x86_64-linux-gnu/libc.so.6(+0x43090)[0x7f383b601090] [9cb2fee5a080:05700] [ 1] /lib/x86_64-linux-gnu/libc.so.6(+0x18b767)[0x7ff722098767] [9cb2fee5a080:05699] [ 2] /lib/x86_64-linux-gnu/libc.so.6(+0x18b767)[0x7f383b749767] [9cb2fee5a080:05700] [ 2] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/adios2-2.9.0-45plavvq7b3lnmhfao3r3xl74x5plswo/lib/../lib/libadios2_core.so.2(_ZN6adios26format13BP5Serializer23CollectFinalShapeValuesEv+0xc7)[0x7ff70c6be377] [9cb2fee5a080:05699] [ 3] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/adios2-2.9.0-45plavvq7b3lnmhfao3r3xl74x5plswo/lib/../lib/libadios2_core.so.2(_ZN6adios26format13BP5Serializer23CollectFinalShapeValuesEv+0xc7)[0x7f3825d10377] [9cb2fee5a080:05700] [ 3] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/adios2-2.9.0-45plavvq7b3lnmhfao3r3xl74x5plswo/lib/../lib/libadios2_core.so.2(_ZN6adios26format13BP5Serializer13CloseTimestepEib+0x30b)[0x7ff70c6c028b] [9cb2fee5a080:05699] [ 4] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/adios2-2.9.0-45plavvq7b3lnmhfao3r3xl74x5plswo/lib/../lib/libadios2_core.so.2(_ZN6adios26format13BP5Serializer13CloseTimestepEib+0x30b)[0x7f3825d1228b] [9cb2fee5a080:05700] [ 4] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/adios2-2.9.0-45plavvq7b3lnmhfao3r3xl74x5plswo/lib/../lib/libadios2_core.so.2(_ZN6adios24core6engine9SstWriter7EndStepEv+0x2ef)[0x7ff70c708a6f] [9cb2fee5a080:05699] [ 5] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/adios2-2.9.0-45plavvq7b3lnmhfao3r3xl74x5plswo/lib/libadios2_c.so.2(adios2_end_step+0xc0)[0x7ff718fe4cf0] [9cb2fee5a080:05699] [ 6] /sensei/build/lib/python3.10/site-packages/sensei/_senseiPython.so(_ZN6sensei21ADIOS2AnalysisAdaptor13WriteTimestepEmdRKSt6vectorISt10shared_ptrINS_12MeshMetadataEESaIS4_EERKS1_I16svtkSmartPointerI20svtkCompositeDataSetESaISB_EE+0x33c)[0x7ff71a7f6a4c] [9cb2fee5a080:05699] [ 7] /sensei/build/lib/python3.10/site-packages/sensei/_senseiPython.so(_ZN6sensei21ADIOS2AnalysisAdaptor7ExecuteEPNS_11DataAdaptorEPS2_+0x827)[0x7ff71a8115b7] [9cb2fee5a080:05699] [ 8] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/adios2-2.9.0-45plavvq7b3lnmhfao3r3xl74x5plswo/lib/../lib/libadios2_core.so.2(_ZN6adios24core6engine9SstWriter7EndStepEv+0x2ef)[0x7f3825d5aa6f] [9cb2fee5a080:05700] [ 5] /sensei/build/lib/python3.10/site-packages/sensei/_senseiPython.so(_ZN6sensei20ConfigurableAnalysis7ExecuteEPNS_11DataAdaptorEPS2_+0x86)[0x7ff71a793546] [9cb2fee5a080:05699] [ 9] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/adios2-2.9.0-45plavvq7b3lnmhfao3r3xl74x5plswo/lib/libadios2_c.so.2(adios2_end_step+0xc0)[0x7f38383eecf0] [9cb2fee5a080:05700] [ 6] /sensei/build/lib/python3.10/site-packages/sensei/_senseiPython.so(+0x18aa28)[0x7ff71a6e4a28] [9cb2fee5a080:05699] [10] /sensei/build/lib/python3.10/site-packages/sensei/_senseiPython.so(_ZN6sensei21ADIOS2AnalysisAdaptor13WriteTimestepEmdRKSt6vectorISt10shared_ptrINS_12MeshMetadataEESaIS4_EERKS1_I16svtkSmartPointerI20svtkCompositeDataSetESaISB_EE+0x33c)[0x7f38338bda4c] [9cb2fee5a080:05700] [ 7] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x112cf8)[0x7ff722216cf8] [9cb2fee5a080:05699] [11] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyObject_MakeTpCall+0x8c)[0x7ff7221cd57c] [9cb2fee5a080:05699] [12] /sensei/build/lib/python3.10/site-packages/sensei/_senseiPython.so(_ZN6sensei21ADIOS2AnalysisAdaptor7ExecuteEPNS_11DataAdaptorEPS2_+0x827)[0x7f38338d85b7] [9cb2fee5a080:05700] [ 8] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x8f84)[0x7ff722177e74] [9cb2fee5a080:05699] [13] /sensei/build/lib/python3.10/site-packages/sensei/_senseiPython.so(_ZN6sensei20ConfigurableAnalysis7ExecuteEPNS_11DataAdaptorEPS2_+0x86)[0x7f383385a546] [9cb2fee5a080:05700] [ 9] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7ff7222c0274] [9cb2fee5a080:05699] [14] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x5f7b)[0x7ff722174e6b] [9cb2fee5a080:05699] [15] /sensei/build/lib/python3.10/site-packages/sensei/_senseiPython.so(+0x18aa28)[0x7f38337aba28] [9cb2fee5a080:05700] [10] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7ff7222c0274] [9cb2fee5a080:05699] [16] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x112cf8)[0x7f383b8c7cf8] [9cb2fee5a080:05700] [11] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x749e)[0x7ff72217638e] [9cb2fee5a080:05699] [17] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(PyEval_EvalCode+0xd0)[0x7ff7222c00a0] [9cb2fee5a080:05699] [18] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyObject_MakeTpCall+0x8c)[0x7f383b87e57c] [9cb2fee5a080:05700] [12] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1b64bd)[0x7ff7222ba4bd] [9cb2fee5a080:05699] [19] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x8f84)[0x7f383b828e74] [9cb2fee5a080:05700] [13] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1134e3)[0x7ff7222174e3] [9cb2fee5a080:05699] [20] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7f383b971274] [9cb2fee5a080:05700] [14] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x749e)[0x7ff72217638e] [9cb2fee5a080:05699] [21] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x5f7b)[0x7f383b825e6b] [9cb2fee5a080:05700] [15] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7ff7222c0274] [9cb2fee5a080:05699] [22] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x749e)[0x7ff72217638e] [9cb2fee5a080:05699] [23] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7ff7222c0274] [9cb2fee5a080:05699] [24] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7f383b971274] [9cb2fee5a080:05700] [16] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x69b5)[0x7ff7221758a5] [9cb2fee5a080:05699] [25] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x749e)[0x7f383b82738e] [9cb2fee5a080:05700] [17] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7ff7222c0274] [9cb2fee5a080:05699] [26] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x69b5)[0x7ff7221758a5] [9cb2fee5a080:05699] [27] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(PyEval_EvalCode+0xd0)[0x7f383b9710a0] [9cb2fee5a080:05700] [18] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7ff7222c0274] [9cb2fee5a080:05699] [28] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x749e)[0x7ff72217638e] [9cb2fee5a080:05699] [29] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1b64bd)[0x7f383b96b4bd] [9cb2fee5a080:05700] [19] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7ff7222c0274] [9cb2fee5a080:05699] SENSEIEndPoint h5stream.n2 h5stream.n2_s0 h5stream.n2_s1 h5stream.n2_s2 h5stream.n2_s3 h5test.n2 hist_mesh_values_0.txt hist_mesh_values_1.txt hist_mesh_values_2.txt hist_mesh_values_3.txt hist_mesh_values_4.txt hist_ucdmesh_data_0.txt hist_ucdmesh_data_1.txt hist_ucdmesh_data_2.txt hist_ucdmesh_data_3.txt hist_ucdmesh_data_4.txt hist_ucdmesh_data_5.txt hist_ucdmesh_data_6.txt hist_ucdmesh_data_7.txt hist_ucdmesh_data_8.txt hist_ucdmesh_data_9.txt mandelbrot mesh_pvd_mandelbrot mesh_pvd_oscillator mesh_visit_mandelbrot mesh_visit_oscillator oscillator oscillator_iso_pipeline_mesh pcolor_0000.png pcolor_0001.png pcolor_0002.png pcolor_0003.png pcolor_0004.png pv_out sensei-env.sh simpleTestDriver test.bp.sst test.h5 testADIOS2BP4_00000.bp testADIOS2BP4_00001.bp testADIOS2SST.bp.sst testHDF5 testHistogram testProgrammableDataAdaptor visit_out End of error message SENSEIEndPoint h5stream.n2 h5stream.n2_s0 h5stream.n2_s1 h5stream.n2_s2 h5stream.n2_s3 h5test.n2 hist_mesh_values_0.txt hist_mesh_values_1.txt hist_mesh_values_2.txt hist_mesh_values_3.txt hist_mesh_values_4.txt hist_ucdmesh_data_0.txt hist_ucdmesh_data_1.txt hist_ucdmesh_data_2.txt hist_ucdmesh_data_3.txt hist_ucdmesh_data_4.txt hist_ucdmesh_data_5.txt hist_ucdmesh_data_6.txt hist_ucdmesh_data_7.txt hist_ucdmesh_data_8.txt hist_ucdmesh_data_9.txt mandelbrot mesh_pvd_mandelbrot mesh_pvd_oscillator mesh_visit_mandelbrot mesh_visit_oscillator oscillator oscillator_iso_pipeline_mesh pcolor_0000.png pcolor_0001.png pcolor_0002.png pcolor_0003.png pcolor_0004.png pv_out sensei-env.sh simpleTestDriver test.bp.sst test.h5 testADIOS2BP4_00000.bp testADIOS2BP4_00001.bp testADIOS2SST.bp.sst testHDF5 testHistogram testProgrammableDataAdaptor visit_out /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1134e3)[0x7f383b8c84e3] [9cb2fee5a080:05700] [20] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x749e)[0x7f383b82738e] [9cb2fee5a080:05700] [21] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7f383b971274] [9cb2fee5a080:05700] [22] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x749e)[0x7f383b82738e] [9cb2fee5a080:05700] [23] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7f383b971274] [9cb2fee5a080:05700] [24] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x69b5)[0x7f383b8268a5] [9cb2fee5a080:05700] [25] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7f383b971274] [9cb2fee5a080:05700] [26] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x69b5)[0x7f383b8268a5] [9cb2fee5a080:05700] [27] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7f383b971274] [9cb2fee5a080:05700] [28] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x749e)[0x7f383b82738e] [9cb2fee5a080:05700] [29] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7f383b971274] [9cb2fee5a080:05700] SENSEIEndPoint h5stream.n2 h5stream.n2_s0 h5stream.n2_s1 h5stream.n2_s2 h5stream.n2_s3 h5test.n2 hist_mesh_values_0.txt hist_mesh_values_1.txt hist_mesh_values_2.txt hist_mesh_values_3.txt hist_mesh_values_4.txt hist_ucdmesh_data_0.txt hist_ucdmesh_data_1.txt hist_ucdmesh_data_2.txt hist_ucdmesh_data_3.txt hist_ucdmesh_data_4.txt hist_ucdmesh_data_5.txt hist_ucdmesh_data_6.txt hist_ucdmesh_data_7.txt hist_ucdmesh_data_8.txt hist_ucdmesh_data_9.txt mandelbrot mesh_pvd_mandelbrot mesh_pvd_oscillator mesh_visit_mandelbrot mesh_visit_oscillator oscillator oscillator_iso_pipeline_mesh pcolor_0000.png pcolor_0001.png pcolor_0002.png pcolor_0003.png pcolor_0004.png pv_out sensei-env.sh simpleTestDriver test.bp.sst test.h5 testADIOS2BP4_00000.bp testADIOS2BP4_00001.bp testADIOS2SST.bp.sst testHDF5 testHistogram testProgrammableDataAdaptor visit_out End of error message SENSEIEndPoint h5stream.n2 h5stream.n2_s0 h5stream.n2_s1 h5stream.n2_s2 h5stream.n2_s3 h5test.n2 hist_mesh_values_0.txt hist_mesh_values_1.txt hist_mesh_values_2.txt hist_mesh_values_3.txt hist_mesh_values_4.txt hist_ucdmesh_data_0.txt hist_ucdmesh_data_1.txt hist_ucdmesh_data_2.txt hist_ucdmesh_data_3.txt hist_ucdmesh_data_4.txt hist_ucdmesh_data_5.txt hist_ucdmesh_data_6.txt hist_ucdmesh_data_7.txt hist_ucdmesh_data_8.txt hist_ucdmesh_data_9.txt mandelbrot mesh_pvd_mandelbrot mesh_pvd_oscillator mesh_visit_mandelbrot mesh_visit_oscillator oscillator oscillator_iso_pipeline_mesh pcolor_0000.png pcolor_0001.png pcolor_0002.png pcolor_0003.png pcolor_0004.png pv_out sensei-env.sh simpleTestDriver test.bp.sst test.h5 testADIOS2BP4_00000.bp testADIOS2BP4_00001.bp testADIOS2SST.bp.sst testHDF5 testHistogram testProgrammableDataAdaptor visit_out STATUS[0] : received mesh mesh with 2 arrays ('f_xyt', 'sin_x_sin_y') STATUS[0] : BlockIds=(0, 1, 2, 3, 4, 5, 6, 7, 8) STATUS[0] : SenderBlockOwner=(0, 0, 0, 0, 0, 1, 1, 1, 1) STATUS[0] : ReceiverBlockOwner=(0, 0, 0, 0, 0, 0, 0, 0, 0) -------------------------------------------------------------------------- Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted. -------------------------------------------------------------------------- STATUS: [0][/sensei/src/sensei/ConfigurableAnalysis.cxx:685][v4.1.0-41-ge8c2ae8] STATUS: Configured CatalystAnalysisAdaptor pythonscript /sensei/src/sensei/testing/catalyst_render_receiver_partition.py STATUS: [0][/sensei/src/sensei/ConfigurableAnalysis.cxx:685][v4.1.0-41-ge8c2ae8] STATUS: Configured CatalystAnalysisAdaptor pythonscript /sensei/src/sensei/testing/catalyst_render_sender_partition.py STATUS[0] : receiving data... ( 1.086s) [pvbatch ] v2_internals.py:171 WARN| Module 'catalyst_render_receiver_partition' missing Catalyst 'options', will use a default options object ( 1.088s) [pvbatch ]vtkCPPythonScriptV2Help:459 WARN| script may not depend on simulation data; is that expected? ( 1.090s) [pvbatch ] v2_internals.py:171 WARN| Module 'catalyst_render_sender_partition' missing Catalyst 'options', will use a default options object ( 1.090s) [pvbatch ]vtkCPPythonScriptV2Help:459 WARN| script may not depend on simulation data; is that expected? .[Tue Jun 20 14:00:45 2023] [ADIOS2 ERROR] <Helper> <adiosSystem> <ExceptionToError> : adios2_end_step: [Tue Jun 20 14:00:45 2023] [ADIOS2 EXCEPTION] <Engine> <SstReader> <EndStep> : EndStep() is called without a successful BeginStep()  ERROR: [0][/sensei/src/sensei/ADIOS2Schema.cxx:591][v4.1.0-41-ge8c2ae8] ERROR: adios2_end_step failed . STATUS[0] : completed after processing 2 steps test_stat=0 exit 0 -------------------------------------------------------------------------- mpiexec noticed that process rank 0 with PID 0 on node 9cb2fee5a080 exited on signal 11 (Segmentation fault). --------------------------------------------------------------------------
17: 
2/2 Test #17: testPartitionersADIOS2SST ........***Failed  Error regular expression found in output. Regex=[[Ee][Rr][Rr][Oo][Rr]]  3.99 sec

0% tests passed, 2 tests failed out of 2

Label Time Summary:
ADIOS2      = 208.18 sec*proc (2 tests)
CATALYST    =   7.98 sec*proc (1 test)
PARALLEL    = 208.18 sec*proc (2 tests)
PYTHON      = 208.18 sec*proc (2 tests)
SHELL       = 208.18 sec*proc (2 tests)

Total Test time (real) = 104.10 sec

The following tests FAILED:
	 15 - testADIOS2SST (Timeout)
	 17 - testPartitionersADIOS2SST (Failed)

Will Dunklin · Answer 3 · Wed Jun 21 2023 01:18:46 GMT+0800 (China Standard Time)

@vicentebolea does this failure look familiar? I know you were porting some ADIOS2 2.9 changes a while back

Vicente Bolea · Answer 4 · Wed Jun 21 2023 01:43:21 GMT+0800 (China Standard Time)

@willdunklin not to me, run those tests manually with the env variables set to SstVerbose=100 SstCPVerbose=100

Burlen Loring · Answer 5 · Fri Jun 23 2023 00:43:11 GMT+0800 (China Standard Time)

At the end of the error message

Timeout or non-current SST contact file?

could it be old file from a previous run is found, that we or adios changed the default time out for sst, or we need to increase the timeout on cloud test system?

Vicente Bolea · Answer 6 · Fri Jun 23 2023 01:02:14 GMT+0800 (China Standard Time)

At the end of the error message
Timeout or non-current SST contact file?
could it be old file from a previous run is found, that we or adios changed the default time out for sst, or we need to increase the timeout on cloud test system?

That looks like you are running the SST test in the same dir where that test was run before and it was not cleaned up.

Will Dunklin · Answer 7 · Fri Jun 23 2023 01:34:56 GMT+0800 (China Standard Time)

@vicentebolea Here's that more verbose output from the first of the two tests:

test 13
      Start 13: testADIOS2SST

13: Test command: /sensei/src/sensei/testing/testADIOS2.sh "/opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/openmpi-4.1.5-l7f7aw6einsykqbqqouclherjvxfmz5p/bin/mpiexec" "-n" "2" "/opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/bin/python3" "/sensei/src/sensei/testing" "testADIOS2SST.bp" "SST" "SST" "3" "--" "--allow-run-as-root"
13: Working Directory: /sensei/build/bin
13: Environment variables: 
13:  MPI_PROCESSES=2
13:  MPI_PROCESSES_HALF=1
13:  MPI_LAUNCHER=/opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/openmpi-4.1.5-l7f7aw6einsykqbqqouclherjvxfmz5p/bin/mpiexec -n 2 --allow-run-as-root
13:  MPIEXEC_POSTFLAGS=
13: Environment variable modifications: 
13:  PYTHONPATH=path_list_prepend:/sensei/build/lib/python3.10/site-packages/
13: Test timeout computed to be: 100
13: rm -rfv testADIOS2SST.bp
13: echo testing SST -> SST
13: testing SST -> SST
13: echo M=2 x N=1
13: M=2 x N=1
13: mpiexec --allow-run-as-root -n 2 /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/bin/python3 -m mpi4py /sensei/src/sensei/testing/testADIOS2Write.py SST testADIOS2SST.bp 0 3
13: writePid=4735
13: [[ SST == BP4 ]]
13: mpiexec --allow-run-as-root -n 1 /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/bin/python3 -m mpi4py /sensei/src/sensei/testing/testADIOS2Read.py SST testADIOS2SST.bp
13: STATUS[0] : initializing ADIOS2DataAdaptor file=testADIOS2SST.bp engine=SST
13: Sst set to use sockets as a Control Transport
13: STATUS[0] : initializing the SVTKDataAdaptor step 0 time 0.0
13: STATUS[0] : executing ADIOS2AnalysisAdaptor SST step 0 time 0.0
13: WARNING: [0][/sensei/src/sensei/ADIOS2AnalysisAdaptor.cxx:217][v4.1.0-41-ge8c2ae8]
13: WARNING: No subset specified. Writing all available data
13: Sst set to use sockets as a Control Transport
13: Sst set to use sockets as a Control Transport
13: Looking for writer contact in file testADIOS2SST.bp.sst, with timeout 60 secs
13: RDMA Dataplane could not find an RDMA-compatible fabric.
13: RDMA Dataplane evaluating viability, returning priority -1
13: Considering DataPlane "evpath" for possible use, priority is 1
13: Considering DataPlane "rdma" for possible use, priority is -1
13: RDMA Dataplane could not find an RDMA-compatible fabric.
13: RDMA Dataplane evaluating viability, returning priority -1
13: RDMA Dataplane unloading
13: Selecting DataPlane "evpath", priority 1 for use
13: RDMA Dataplane unloading
13: Stream "testADIOS2SST.bp" waiting for 1 readers
13: Opening Stream "testADIOS2SST.bp"
13: Writer stream params are:
13: Param -   RegistrationMethod=File
13: Param -   RendezvousReaderCount=1
13: Param -   QueueLimit=0 (unlimited)
13: Param -   QueueFullPolicy=Block
13: Param -   StepDistributionMode=StepsAllToAll
13: Param -   DataTransport=evpath
13: Param -   ControlTransport=sockets
13: Param -   NetworkInterface=(default)
13: Param -   ControlInterface=(default to NetworkInterface if applicable)
13: Param -   DataInterface=(default to NetworkInterface if applicable)
13: Param -   CompressionMethod=None
13: Param -   CPCommPattern=Min
13: Param -   MarshalMethod=BP5
13: Param -   FirstTimestepPrecious=False
13: Param -   IsRowMajor=1  (not user settable) 
13: Param -   OpenTimeoutSecs=60 (seconds)
13: Param -   SpeculativePreloadMode=Auto
13: Param -   SpecAutoNodeThreshold=1
13: Param -   ControlModule=select
13: Stream "testADIOS2SST.bp" waiting for 1 readers
13: Waiting for writer DPResponse message in SstReadOpen("testADIOS2SST.bp")
13: finished wait writer DPresponse message in read_open, WRITER is using "evpath" DataPlane
13: RDMA Dataplane could not find an RDMA-compatible fabric.
13: RDMA Dataplane evaluating viability, returning priority -1
13: Prefered dataplane name is "evpath"
13: Considering DataPlane "evpath" for possible use, priority is 1
13: Selecting DataPlane "evpath" (preferred) for use
13: RDMA Dataplane unloading
13: Waiting for writer response message in SstReadOpen("testADIOS2SST.bp")
13: Beginning writer-side reader open protocol
13: Beginning writer-side reader open protocol
13: Received contact info "AAIAAJTJ8o097QAAATkCmAMAEaw=", RD_Stream 0x559ade7891a0 for Reader Rank 0
13: Setting SpeculativePreload ON for new reader
13: Received contact info "AAIAAJTJ8o097QAAATkCmAMAEaw=", RD_Stream 0x559ade7891a0 for Reader Rank 0
13: Setting SpeculativePreload ON for new reader
13: My oldest timestep was 0, global oldest timestep was 0
13: My oldest timestep was 0, global oldest timestep was 0
13: Finish writer-side reader open protocol for reader 0x555c813aa820, reader ready response pending
13: Finish writer-side reader open protocol for reader 0x5583b70dbb00, reader ready response pending
13: (PID 1288, TID 7f841daf05c0) Waiting for Reader ready on WSR 0x5583b70dbb00.
13: finished wait writer response message in read_open
13: Opening Reader Stream.
13: Writer stream params are:
13: Param -   RegistrationMethod=File
13: Param -   RendezvousReaderCount=1
13: Param -   QueueLimit=0 (unlimited)
13: Param -   QueueFullPolicy=Block
13: Param -   StepDistributionMode=StepsAllToAll
13: Param -   DataTransport=evpath
13: Param -   ControlTransport=sockets
13: Param -   NetworkInterface=(default)
13: Param -   ControlInterface=(default to NetworkInterface if applicable)
13: Param -   DataInterface=(default to NetworkInterface if applicable)
13: Param -   CompressionMethod=None
13: Param -   CPCommPattern=Min
13: Param -   MarshalMethod=BP5
13: Param -   FirstTimestepPrecious=False
13: Param -   IsRowMajor=1  (not user settable) 
13: Param -   OpenTimeoutSecs=60 (seconds)
13: Param -   SpeculativePreloadMode=Auto
13: Param -   SpecAutoNodeThreshold=1
13: Param -   ControlModule=select
13: Reader stream params are:
13: Param -   RegistrationMethod=File
13: Param -   DataTransport=evpath
13: Param -   ControlTransport=sockets
13: Param -   NetworkInterface=(default)
13: Param -   ControlInterface=(default to NetworkInterface if applicable)
13: Param -   DataInterface=(default to NetworkInterface if applicable)
13: Param -   AlwaysProvideLatestTimestep=False
13: Param -   OpenTimeoutSecs=60 (seconds)
13: Param -   SpeculativePreloadMode=Auto
13: Param -   SpecAutoNodeThreshold=1
13: Param -   ControlModule=select
13: Writer is using Minimum Connection Communication pattern (min)
13: Received contact info "Writer Rank 0, test contact", WS_stream 0x5583b70d6480 for WSR Rank 0
13: Received contact info "Writer Rank 1, test contact", WS_stream 0x555c813a51a0 for WSR Rank 1
13: Sending Reader Activate messages to writer
13: Finish opening Stream "testADIOS2SST.bp", starting with Step number 0
13: Wait for next metadata after last timestep -1
13: Waiting for metadata for a Timestep later than TS -1
13: (PID 1285, TID 7f80abe0b5c0) Stream status is Established
13: Reader Activate message received for Stream 0x5583b70dbb00.  Setting state to Established.
13: Parent stream reader count is now 1.
13: Reader ready on WSR 0x5583b70dbb00, Stream established, Starting 0 LastProvided 0.
13: Finish opening Stream "testADIOS2SST.bp"
13: Finish opening Stream "testADIOS2SST.bp"
13: ProvideTimestep, registering timestep 0, data 0x555c813c1b10, fprint 939050d2103150d
13: ProvideTimestep, registering timestep 0, data 0x5583b70f5280, fprint 939050d2103150d
13: Reader 0 status Established has last released 4294967295, last sent 0
13: QueueMaintenance, smallest last released = -1, count = 1
13: Removing dead entries
13: QueueMaintenance complete
13: Sending TimestepMetadata for timestep 0 (ref count 1), one to each reader
13: Sent timestep 0 to reader cohort 0
13: ADDING timestep 0 to sent list for reader cohort 0, READER 0x5583b70dbb00, reference count is now 2
13: PRELOADMODE for timestep 0 non-default for reader , active at timestep 0, mode 1
13: Per reader registration for timestep 0, preload mode 1
13: Sending Speculative Preload messages, reader 0x5583b70d6480, timestep 0
13: Reader 0 status Established has last released 4294967295, last sent 0
13: QueueMaintenance, smallest last released = -1, count = 1
13: Removing dead entries
13: QueueMaintenance complete
13: Sending TimestepMetadata for timestep 0 (ref count 1), one to each reader
13: Sent timestep 0 to reader cohort 0
13: ADDING timestep 0 to sent list for reader cohort 0, READER 0x555c813aa820, reference count is now 2
13: PRELOADMODE for timestep 0 non-default for reader , active at timestep 0, mode 1
13: Per reader registration for timestep 0, preload mode 1
13: Sending Speculative Preload messages, reader 0x555c813a51a0, timestep 0
13: Sending a message to reader 0 (0x559ade180270)
13: SubRef : Writer-side Timestep 0 now has reference count 1, expired 0, precious 0
13: Reader 0 status Established has last released 4294967295, last sent 0
13: QueueMaintenance, smallest last released = -1, count = 1
13: Removing dead entries
13: QueueMaintenance complete
13: STATUS: [0][/sensei/src/sensei/ADIOS2AnalysisAdaptor.cxx:242][v4.1.0-41-ge8c2ae8]
13: STATUS: ADIOS2AnalysisAdaptor wrote step 0
13: STATUS[0] : initializing the SVTKDataAdaptor step 1 time 1.0
13: SubRef : Writer-side Timestep 0 now has reference count 1, expired 0, precious 0
13: Reader 0 status Established has last released 4294967295, last sent 0
13: QueueMaintenance, smallest last released = -1, count = 1
13: Removing dead entries
13: QueueMaintenance complete
13: STATUS[0] : executing ADIOS2AnalysisAdaptor SST step 1 time 1.0
13: [9fc96ed74fcf:04744] *** Process received signal ***
13: [9fc96ed74fcf:04745] *** Process received signal ***
13: [9fc96ed74fcf:04744] Signal: Segmentation fault (11)
13: [9fc96ed74fcf:04744] Signal code: Address not mapped (1)
13: [9fc96ed74fcf:04744] Failing at address: (nil)
13: [9fc96ed74fcf:04745] Signal: Segmentation fault (11)
13: [9fc96ed74fcf:04745] Signal code: Address not mapped (1)
13: [9fc96ed74fcf:04745] Failing at address: (nil)
13: [9fc96ed74fcf:04744] [ 0] [9fc96ed74fcf:04745] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x43090)[0x7f841ddcf090]
13: [9fc96ed74fcf:04744] [ 1] /lib/x86_64-linux-gnu/libc.so.6(+0x43090)[0x7fd782ad3090]
13: [9fc96ed74fcf:04745] [ 1] /lib/x86_64-linux-gnu/libc.so.6(+0x18b767)[0x7f841df17767]
13: [9fc96ed74fcf:04744] [ 2] /lib/x86_64-linux-gnu/libc.so.6(+0x18b767)[0x7fd782c1b767]
13: [9fc96ed74fcf:04745] [ 2] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/adios2-2.9.0-45plavvq7b3lnmhfao3r3xl74x5plswo/lib/../lib/libadios2_core.so.2(_ZN6adios26format13BP5Serializer23CollectFinalShapeValuesEv+0xc7)[0x7fd76d243377]
13: [9fc96ed74fcf:04745] [ 3] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/adios2-2.9.0-45plavvq7b3lnmhfao3r3xl74x5plswo/lib/../lib/libadios2_core.so.2(_ZN6adios26format13BP5Serializer23CollectFinalShapeValuesEv+0xc7)[0x7f840854d377]
13: [9fc96ed74fcf:04744] [ 3] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/adios2-2.9.0-45plavvq7b3lnmhfao3r3xl74x5plswo/lib/../lib/libadios2_core.so.2(_ZN6adios26format13BP5Serializer13CloseTimestepEib+0x30b)[0x7fd76d24528b]
13: [9fc96ed74fcf:04745] [ 4] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/adios2-2.9.0-45plavvq7b3lnmhfao3r3xl74x5plswo/lib/../lib/libadios2_core.so.2(_ZN6adios26format13BP5Serializer13CloseTimestepEib+0x30b)[0x7f840854f28b]
13: [9fc96ed74fcf:04744] [ 4] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/adios2-2.9.0-45plavvq7b3lnmhfao3r3xl74x5plswo/lib/../lib/libadios2_core.so.2(_ZN6adios24core6engine9SstWriter7EndStepEv+0x2ef)[0x7fd76d28da6f]
13: [9fc96ed74fcf:04745] [ 5] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/adios2-2.9.0-45plavvq7b3lnmhfao3r3xl74x5plswo/lib/libadios2_c.so.2(adios2_end_step+0xc0)[0x7fd779b61cf0]
13: [9fc96ed74fcf:04745] [ 6] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/adios2-2.9.0-45plavvq7b3lnmhfao3r3xl74x5plswo/lib/../lib/libadios2_core.so.2(_ZN6adios24core6engine9SstWriter7EndStepEv+0x2ef)[0x7f8408597a6f]
13: [9fc96ed74fcf:04744] [ 5] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/adios2-2.9.0-45plavvq7b3lnmhfao3r3xl74x5plswo/lib/libadios2_c.so.2(adios2_end_step+0xc0)[0x7f8414e60cf0]
13: [9fc96ed74fcf:04744] [ 6] /sensei/build/lib/python3.10/site-packages/sensei/_senseiPython.so(_ZN6sensei21ADIOS2AnalysisAdaptor13WriteTimestepEmdRKSt6vectorISt10shared_ptrINS_12MeshMetadataEESaIS4_EERKS1_I16svtkSmartPointerI20svtkCompositeDataSetESaISB_EE+0x33c)[0x7fd77b373a4c]
13: [9fc96ed74fcf:04745] [ 7] /sensei/build/lib/python3.10/site-packages/sensei/_senseiPython.so(_ZN6sensei21ADIOS2AnalysisAdaptor13WriteTimestepEmdRKSt6vectorISt10shared_ptrINS_12MeshMetadataEESaIS4_EERKS1_I16svtkSmartPointerI20svtkCompositeDataSetESaISB_EE+0x33c)[0x7f8416672a4c]
13: [9fc96ed74fcf:04744] [ 7] /sensei/build/lib/python3.10/site-packages/sensei/_senseiPython.so(_ZN6sensei21ADIOS2AnalysisAdaptor7ExecuteEPNS_11DataAdaptorEPS2_+0x827)[0x7fd77b38e5b7]
13: [9fc96ed74fcf:04745] [ 8] /sensei/build/lib/python3.10/site-packages/sensei/_senseiPython.so(_ZN6sensei21ADIOS2AnalysisAdaptor7ExecuteEPNS_11DataAdaptorEPS2_+0x827)[0x7f841668d5b7]
13: [9fc96ed74fcf:04744] [ 8] /sensei/build/lib/python3.10/site-packages/sensei/_senseiPython.so(+0x18b088)[0x7fd77b262088]
13: [9fc96ed74fcf:04745] [ 9] /sensei/build/lib/python3.10/site-packages/sensei/_senseiPython.so(+0x18b088)[0x7f8416561088]
13: [9fc96ed74fcf:04744] [ 9] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x112cf8)[0x7fd782d99cf8]
13: [9fc96ed74fcf:04745] [10] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyObject_MakeTpCall+0x8c)[0x7fd782d5057c]
13: [9fc96ed74fcf:04745] [11] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x112cf8)[0x7f841e095cf8]
13: [9fc96ed74fcf:04744] [10] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x8f84)[0x7fd782cfae74]
13: [9fc96ed74fcf:04745] [12] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyObject_MakeTpCall+0x8c)[0x7f841e04c57c]
13: [9fc96ed74fcf:04744] [11] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x8f84)[0x7f841dff6e74]
13: [9fc96ed74fcf:04744] [12] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7fd782e43274]
13: [9fc96ed74fcf:04745] [13] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x5f7b)[0x7fd782cf7e6b]
13: [9fc96ed74fcf:04745] [14] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7f841e13f274]
13: [9fc96ed74fcf:04744] [13] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7fd782e43274]
13: [9fc96ed74fcf:04745] [15] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x5f7b)[0x7f841dff3e6b]
13: [9fc96ed74fcf:04744] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x749e)[0x7fd782cf938e]
13: [9fc96ed74fcf:04745] [16] [14] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(PyEval_EvalCode+0xd0)[0x7fd782e430a0]
13: [9fc96ed74fcf:04745] [17] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7f841e13f274]
13: [9fc96ed74fcf:04744] [15] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x749e)[0x7f841dff538e]
13: [9fc96ed74fcf:04744] [16] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1b64bd)[0x7fd782e3d4bd]
13: [9fc96ed74fcf:04745] [18] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(PyEval_EvalCode+0xd0)[0x7f841e13f0a0]
13: [9fc96ed74fcf:04744] [17] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1134e3)[0x7fd782d9a4e3]
13: [9fc96ed74fcf:04745] [19] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1b64bd)[0x7f841e1394bd]
13: [9fc96ed74fcf:04744] [18] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x749e)[0x7fd782cf938e]
13: [9fc96ed74fcf:04745] [20] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7fd782e43274]
13: [9fc96ed74fcf:04745] [21] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1134e3)[0x7f841e0964e3]
13: [9fc96ed74fcf:04744] [19] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x749e)[0x7fd782cf938e]
13: [9fc96ed74fcf:04745] [22] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x749e)[0x7f841dff538e]
13: [9fc96ed74fcf:04744] [20] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7fd782e43274]
13: [9fc96ed74fcf:04745] [23] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x69b5)[0x7fd782cf88a5]
13: [9fc96ed74fcf:04745] [24] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7f841e13f274]
13: [9fc96ed74fcf:04744] [21] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7fd782e43274]
13: [9fc96ed74fcf:04745] [25] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x749e)[0x7f841dff538e]
13: [9fc96ed74fcf:04744] [22] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x69b5)[0x7fd782cf88a5]
13: [9fc96ed74fcf:04745] [26] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7fd782e43274]
13: [9fc96ed74fcf:04745] [27] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7f841e13f274]
13: [9fc96ed74fcf:04744] [23] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x749e)[0x7fd782cf938e]
13: [9fc96ed74fcf:04745] [28] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x69b5)[0x7f841dff48a5]
13: [9fc96ed74fcf:04744] [24] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7fd782e43274]
13: [9fc96ed74fcf:04745] [29] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x749e)[0x7fd782cf938e]
13: [9fc96ed74fcf:04745] *** End of error message ***
13: /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7f841e13f274]
13: [9fc96ed74fcf:04744] [25] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x69b5)[0x7f841dff48a5]
13: [9fc96ed74fcf:04744] [26] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7f841e13f274]
13: [9fc96ed74fcf:04744] [27] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x749e)[0x7f841dff538e]
13: [9fc96ed74fcf:04744] [28] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(+0x1bc274)[0x7f841e13f274]
13: [9fc96ed74fcf:04744] [29] /opt/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/python-3.10.10-qkgsudypzqi5xikftwylbiis5xh7aa3x/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x749e)[0x7f841dff538e]
13: [9fc96ed74fcf:04744] *** End of error message ***
13: Got a preload message from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Received a Timestep metadata message for timestep 0, signaling condition
13: Examining metadata for Timestep 0
13: Returning metadata for Timestep 0
13: Setting TSmsg to Rootentry value
13: Got a preload message from writer rank 1 for timestep 0, fprint 939050d2103150d
13: EVPATH registering reader arrival of TS 0 metadata, preload mode 1
13: SstAdvanceStep returning Success on timestep 0
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: STATUS[0] : received step 0 time 0.0
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 0 for timestep 0, fprint 939050d2103150d
13: Satisfying remote memory read with preload from writer rank 1 for timestep 0, fprint 939050d2103150d
13: Sending ReleaseTimestep message for timestep 0, one to each writer
13: Wait for next metadata after last timestep 0
13: Waiting for metadata for a Timestep later than TS 0
13: (PID 1285, TID 7f80abe0b5c0) Stream status is Established
13: --------------------------------------------------------------------------
13: Primary job  terminated normally, but 1 process returned
13: a non-zero exit code. Per user-direction, the job has been aborted.
13: --------------------------------------------------------------------------
13: Reader-side close handler invoked
13: Reader-side Rank received a connection-close event during normal operations, peer likely failed
13: The close was for connection to writer peer 0, notifying DP
13: received notification that writer peer 0 has failed, failing any pending requests
13: Fail pending requests to rank 0 on stream 0x559ade7891a0
13: Done Failing requests to writer 0 from stream 0x559ade7891a0
13: Stream Final Timestep is 2147483647, last timestep was 0
13: Wait for next metadata returning NULL because the connection failed before final timestep notification
13: SstAdvanceStepMin rank 0 returning FatalError at timestep 0
13: Setting TSmsg to NULL
13: SstAdvanceStep returning FAILURE
13: STATUS[0] : received step 0 time 0.0
13: [Thu Jun 22 17:31:25 2023] [ADIOS2 ERROR] <Helper> <adiosSystem> <ExceptionToError> : adios2_end_step: [Thu Jun 22 17:31:25 2023] [ADIOS2 EXCEPTION] <Engine> <SstReader> <EndStep> : EndStep() is called without a successful BeginStep()
13: 
13: ERROR: [0][/sensei/src/sensei/ADIOS2Schema.cxx:591][v4.1.0-41-ge8c2ae8]
13: ERROR: adios2_end_step failed
13: 
13: Stream "testADIOS2SST.bp" (0x559ade180270) summary info:
13: 	Duration (secs) = 0.188318
13: 	Timestep Metadata Received = 1
13: 	Timesteps Consumed = 1
13: 	MetadataBytesReceived = 5552 (5.4 kB)
13: 	DataBytesReceived = 23296 (22.8 kB)
13: 	PreloadBytesReceived = 23296 (22.8 kB)
13: 	PreloadTimestepsReceived = 1
13: 	AverageReadRankFanIn = 2.0
13: 
13: Destroying stream 0x559ade180270, name testADIOS2SST.bp
13: Discarding prior, TS 0, data 0x7f8088010ce8, fprint 939050d2103150d
13: Discarding prior, TS 0, data 0x7f8088006b88, fprint 939050d2103150d
13: Reference count now zero, Destroying process SST info cache
13: Freeing LastCallList
13: SstStreamDestroy successful, returning
13: STATUS[0] : closed stream after receiving 2 steps
13: rm -rfv testADIOS2SST.bp
13: exit 0
13: --------------------------------------------------------------------------
13: mpiexec noticed that process rank 1 with PID 0 on node 9fc96ed74fcf exited on signal 11 (Segmentation fault).
13: --------------------------------------------------------------------------
13/41 Test #13: testADIOS2SST ............................***Failed  Error regular expression found in output. Regex=[[Ee][Rr][Rr][Oo][Rr]]  2.18 sec

Vicente Bolea · Answer 8 · Tue Jul 04 2023 01:23:11 GMT+0800 (China Standard Time)

running this with GDB will probably tell us the reason of this.