Can not find `/usr/bin/ib_read_lat` on remote server (but it did exist)

[sinkinben]

I want to run benchmark test on our machine. And I use the same machine as both server and client.

The config.json is:

{
    "_comment": "A configuration file providing all information describing the system which are necessary to run ./rpma_fio_bench.sh and ./ib_read.sh.",
    "platform_generation": "Cascade Lake",
    "server_ip": "XX.XX.208.36",
    "JOB_NUMA": 0,
    "AUX_PARAMS": "-d mlx5_1 -R",
    "IB_PATH": "/usr/bin/",
    "FIO_PATH": "/usr/local/bin/",
    "REMOTE_USER": "XXXX",
    "REMOTE_PASS": "123X456",
    "REMOTE_JOB_NUMA": 0,
    "REMOTE_AUX_PARAMS": "-d mlx5_1 -R",
    "REMOTE_IB_PATH": "/usr/bin/",
    "REMOTE_DIRECT_WRITE_TO_PMEM": false,
    "REMOTE_FIO_PATH": "/usr/local/bin/",
    "REMOTE_JOB_PATH": "/home/jianbang/workspace/rpma/tools/perf/fio_jobs/",
    "REMOTE_JOB_MEM_PATH": "/dev/dax1.1"
}

However, when I run:

sudo ./report_bench.py run --config config.json --figures figures/read.json figures/write.json --result_dir results

It outputs:

This tool is EXPERIMENTAL
WARNING: sudo (called on the remote side) will prompt for password!
         Toggling DDIO will be skipped!

         In order to change it:
           1) set permissions of sudo to NOPASSWD in '/etc/sudoers' and
           2) set REMOTE_SUDO_NOPASSWD=1

STARTING benchmark for MODE=lat IP=10.93.208.36 ...
Performance results: ib_read_lat_21-12-08-145453.csv
Output and errors (both sides): /dev/shm/ib_read_lat_21-12-08-145453-errors.log

Name of this run: ib_read_lat_21-12-08-145453_bs256
Error: wrong remote path (REMOTE_IB_PATH) to the "ib_read_lat" tool -  "/usr/bin/ib_read_lat" does not exist
Traceback (most recent call last):
  File "report_bench.py", line 86, in <module>
    main()
  File "report_bench.py", line 80, in main
    if bench.run():
  File "/home/jianbang/workspace/rpma/tools/perf/lib/bench.py", line 258, in run
    req.benchmarks_run(self, config, self.__result_dir)
  File "/home/jianbang/workspace/rpma/tools/perf/lib/Requirement.py", line 176, in benchmarks_run
    b.run(config, result_dir)
  File "/home/jianbang/workspace/rpma/tools/perf/lib/benchmark/base.py", line 199, in run
    runner.run(self, config, idfile)
  File "/home/jianbang/workspace/rpma/tools/perf/lib/benchmark/runner/bash.py", line 100, in run
    subprocess.run(args, env=env, check=True)
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['./ib_read.sh', '10.93.208.36', 'lat']' returned non-zero exit status 1.

[sinkinben]

The content of error log file /dev/shm/ib_read_lat_21-12-08-145453-errors.log:

SSHPASS searching for password prompt using match "assword"
SSHPASS read: 
[e-mail].208.36's password: 
SSHPASS detected prompt. Sending password.
SSHPASS read: 

bash: /dev/shm/ib_read_lat_21-12-08-145453-errors.log: Permission denied

I am sure that the user123 has the sudo privilege.

And I simulate the steps of the script via paramiko library, it can access the /usr/bin/ib_read_lat.