[Fabtest] tests are getting fial while running with ./runfabtests.sh but works when running them menually
danielap1996 opened this issue · comments
The fubtest has a script called ./runfabtests.sh
when running this script - the tests are getting fail:
$> ./runfabtests.sh tcp 127.0.1.1 127.0.1.1
# Test Result
# ------------------------------------------------------------------------------
fi_getinfo_test -s 127.0.1.1 127.0.1.1 -p "tcp": Fail
fi_av_test -g 127.0.1.1 -n 1 -s 127.0.1.1 -e rdm -p "tcp": Fail
fi_av_test -g 127.0.1.1 -n 1 -s 127.0.1.1 -e dgram -p "tcp": Fail
fi_dom_test -n 2 -p "tcp": Fail
fi_eq_test -p "tcp": Fail
fi_cq_test -p "tcp": Fail
fi_mr_test -p "tcp": Fail
fi_cntr_test -p "tcp": Fail
fi_setopt_test -p "tcp": Fail
fi_dgram g00n13s -p "tcp": Fail
fi_rdm g00n13s -p "tcp": Fail
fi_msg g00n13s -p "tcp": Fail
fi_av_xfer -e rdm -p "tcp": Fail
but when I copy the tests commands and run them menually - they pass:
./fi_getinfo_test -s 127.0.1.1 127.0.1.1 -p "tcp"
Running getinfo_no_hints1 [Test with no node, service, flags or hints]...PASS!
Running getinfo_no_hints2 [Test with node, no service, FI_SOURCE flag, no hints]...PASS!
Running getinfo_no_hints3 [Test with service, FI_SOURCE flag, no node, no hints]...PASS!
Running getinfo_no_hints4 [Test with node, service, FI_SOURCE flag, no hints]...PASS!
Running getinfo_no_hints5 [Test with node, service, no hints]...PASS!
Running getinfo_src1 [Test with no node, service, or flags]...PASS!
Running getinfo_src2 [Test with node, no service, FI_SOURCE flag]...PASS!
Running getinfo_src3 [Test with service, FI_SOURCE flag, no node]...PASS!
Running getinfo_src4 [Test with node, service, FI_SOURCE flag]...PASS!
Running getinfo_src_dest1 [Test with node, service]...PASS!
Running getinfo_src_dest2 [Test API version]...PASS!
Running getinfo_msg_ordering1 [Test tx ordering bits supported are set]...(passed)(skipped) (128)(896)/1024 combinations
PASS!
Running getinfo_msg_ordering2 [Test rx ordering bits supported are set]...(passed)(skipped) (128)(896)/1024 combinations
PASS!
Running getinfo_raw_ordering1 [Test rma RAW ordering size is set]...PASS!
Running getinfo_raw_ordering2 [Test rma RAW ordering size is set to hints]...PASS!
Running getinfo_war_ordering1 [Test rma WAR ordering size is set]...skipped because: fi_getinfo returned 61 - No data available
Running getinfo_war_ordering2 [Test rma WAR ordering size is set to hints]...skipped because: fi_getinfo returned 61 - No data available
Running getinfo_waw_ordering1 [Test rma WAW ordering size is set]...PASS!
Running getinfo_waw_ordering2 [Test rma WAW ordering size is set to hints]...PASS!
Running getinfo_bad_raw_ordering1 [Test invalid rma RAW ordering size]...PASS!
Running getinfo_bad_war_ordering1 [Test invalid rma WAR ordering size]...skipped because: fi_getinfo returned 61 - No data available
Running getinfo_bad_waw_ordering1 [Test invalid rma WAW ordering size]...PASS!
Running getinfo_neg1 [Test with non-existent domain name]...PASS!
Running getinfo_mr_mode1 [Test FI_MR_BASIC]...PASS!
Running getinfo_mr_mode2 [Test FI_MR_SCALABLE]...PASS!
Running getinfo_mr_mode3 [Test FI_MR_UNSPEC (v1.0)]...PASS!
Running getinfo_mr_mode4 [Test FI_MR_BASIC (v1.0)]...PASS!
Running getinfo_mr_mode5 [Test FI_MR_SCALABLE (v1.0)]...PASS!
Running getinfo_mr_mode6 [Test mr_mode bits]...(passed)(skipped) (256)(0)/256 combinations
PASS!
Running getinfo_progress1 [Test data manual progress]...PASS!
Running getinfo_progress2 [Test data auto progress]...PASS!
Running getinfo_progress3 [Test ctrl manual progress]...PASS!
Running getinfo_progress4 [Test ctrl auto progress]...PASS!
Running getinfo_caps1 [Test capability bits supported are set]...(passed)(skipped) (64)(960)/1024 combinations
PASS!
Running getinfo_caps2 [Test capability with no hints]...PASS!
Running getinfo_caps3 [Test domain capabilities]...(passed)(skipped) (4)(4)/8 combinations
PASS!
Running getinfo_caps4 [Test for capability bit regression]...PASS!
Running getinfo_caps5 [Test if either FI_LOCAL_COMM or FI_REMOTE_COMM is set]...PASS!
Summary: all tests passed
./fi_av_test -g 127.0.1.1 -n 1 -s 127.0.1.1 -e rdm -p "tcp"
Testing AVs on fabric tcp
Testing with type = FI_AV_MAP
Running av_open_close [Test open and close AVs of varying sizes]...PASS!
Running av_good_sync [Test sync AV insert with good address]...PASS!
Running av_null_fi_addr [Test AV insert without specifying fi_addr]...skipped because: test not valid for AV type FI_AV_MAP
Running av_good_vector_async [Test async AV insert with vector of good addresses]...PASS!
Running av_zero_async [Test async insert AV insert of zero addresses]...PASS!
Running av_good_2vector_async [Test async AV inserts with two address vectors]...PASS!
Running av_insert_stages [Test AV insert at various stages]...PASS!
Testing with invalid address
Running av_bad_sync [Test sync AV insert of bad address]...PASS!
Running av_goodbad_vector_sync [Test sync AV insert of 1 good and 1 bad address]...PASS!
Running av_goodbad_vector_async [Test async AV insert with good and bad address]...PASS!
Running av_goodbad_vector_sync_err [Test AV insert of 1 good, 1 bad address using FI_SYNC_ERR]...skipped because: test not valid for AV type FI_AV_MAP
Testing with type = FI_AV_TABLE
Running av_open_close [Test open and close AVs of varying sizes]...PASS!
Running av_good_sync [Test sync AV insert with good address]...PASS!
Running av_null_fi_addr [Test AV insert without specifying fi_addr]...PASS!
Running av_good_vector_async [Test async AV insert with vector of good addresses]...PASS!
Running av_zero_async [Test async insert AV insert of zero addresses]...PASS!
Running av_good_2vector_async [Test async AV inserts with two address vectors]...PASS!
Running av_insert_stages [Test AV insert at various stages]...PASS!
Testing with invalid address
Running av_bad_sync [Test sync AV insert of bad address]...PASS!
Running av_goodbad_vector_sync [Test sync AV insert of 1 good and 1 bad address]...PASS!
Running av_goodbad_vector_async [Test async AV insert with good and bad address]...PASS!
Running av_goodbad_vector_sync_err [Test AV insert of 1 good, 1 bad address using FI_SYNC_ERR]...PASS!
Summary: all tests passed
@danielap1996 My first guess is most likely an ssh issue. runfabtests.sh works by ssh'ing into the arguments (127.0.1.1) and then running the command. If you're seeing this behavior, verify that ssh'ing into 127.0.1.1 works without having to ask you for a password (you might have to setup some new ssh keys). If that is working fine, try running runfabtests.sh -vvv (there are 3 verbosity levels: -v, -vv, -vvv) and we can go from there.
The runfabtests.sh script uses ssh to run the commands and usually the current working directory is not the same as the working directory of the ssh target. As a result, ssh can't find the test binaries. This can be discovered with the -v option. Please add the -p $PWD option to solve the issue, i.e.,
./runfabtests.sh -p $PWD tcp 127.0.1.1 127.0.1.1