Error when running the local experiment: subprocess.CalledProcessError
wyunc opened this issue · comments
An error occurred while running the local experiment, The command I used was:
PYTHONPATH=. python3 experiment/run_experiment.py \
--experiment-config test1.yaml \
--benchmarks libpng_libpng_read_fuzzer libxml2_xml \
--experiment-name $EXPERIMENT_NAME \
--fuzzers afl libfuzzer
Docker version 23.0.5
The error message is as follows:
INFO:root:Starting experiment.
ERROR:root:Executed command: "make -j base-image worker" returned: 2. Extras: {'output': 'python3.10 -m venv .venv || python3 -m venv .venv\nsource .venv/bin/activate && python3 -m pip install --upgrade pip setuptools && python3 -m pip install -r requirements.txt\nRequirement already satisfied: pip in ./.venv/lib/python3.10/site-packages (22.2.2)\nWARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/pip/\nCollecting pip\n Downloading pip-23.1.2-py3-none-any.whl (2.1 MB)\n ━━━ 0.2/2.1 MB 21.3 kB/s eta 0:01:29\nERROR: Exception:\nTraceback (most recent call last):\n File "/work/src/.venv/lib/python3.10/site-packages/pip/_vendor/urllib3/response.py", line 435, in _error_catcher\n yield\n File "/work/src/.venv/lib/python3.10/site-packages/pip/_vendor/urllib3/response.py", line 516, in read\n data = self._fp.read(amt) if not fp_closed else b""\n File "/work/src/.venv/lib/python3.10/site-packages/pip/_vendor/cachecontrol/filewrapper.py", line 90, in read\n data = self.__fp.read(amt)\n File "/usr/local/lib/python3.10/http/client.py", line 465, in read\n s = self.fp.read(amt)\n File "/usr/local/lib/python3.10/socket.py", line 705, in readinto\n return self._sock.recv_into(b)\n File "/usr/local/lib/python3.10/ssl.py", line 1274, in recv_into\n return self.read(nbytes, buffer)\n File "/usr/local/lib/python3.10/ssl.py", line 1130, in read\n return self._sslobj.read(len, buffer)\nTimeoutError: The read operation timed out\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/work/src/.venv/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper\n status = run_func(*args)\n File "/work/src/.venv/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 247, in wrapper\n return func(self, options, args)\n File "/work/src/.venv/lib/python3.10/site-packages/pip/_internal/commands/install.py", line 369, in run\n requirement_set = resolver.resolve(\n File "/work/src/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 92, in resolve\n result = self._result = resolver.resolve(\n File "/work/src/.venv/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 481, in resolve\n state = resolution.resolve(requirements, max_rounds=max_rounds)\n File "/work/src/.venv/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 348, in resolve\n self._add_to_criteria(self.state.criteria, r, parent=None)\n File "/work/src/.venv/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 172, in _add_to_criteria\n if not criterion.candidates:\n File "/work/src/.venv/lib/python3.10/site-packages/pip/_vendor/resolvelib/structs.py", line 151, in bool\n return bool(self._sequence)\n File "/work/src/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in bool\n return any(self)\n File "/work/src/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in \n return (c for c in iterator if id(c) not in self._incompatible_ids)\n File "/work/src/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 97, in _iter_built_with_inserted\n candidate = func()\n File "/work/src/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 206, in _make_candidate_from_link\n self._link_candidate_cache[link] = LinkCandidate(\n File "/work/src/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 297, in init\n super().init(\n File "/work/src/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 162, in init\n self.dist = self._prepare()\n File "/work/src/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 231, in _prepare\n dist = self._prepare_distribution()\n File "/work/src/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 308, in _prepare_distribution\n return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)\n File "/work/src/.venv/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 438, in prepare_linked_requirement\n return self._prepare_linked_requirement(req, parallel_builds)\n File "/work/src/.venv/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 483, in _prepare_linked_requirement\n local_file = unpack_url(\n File "/work/src/.venv/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 165, in unpack_url\n file = get_http_url(\n File "/work/src/.venv/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 106, in get_http_url\n from_path, content_type = download(link, temp_dir.path)\n File "/work/src/.venv/lib/python3.10/site-packages/pip/_internal/network/download.py", line 147, in call\n for chunk in chunks:\n File "/work/src/.venv/lib/python3.10/site-packages/pip/_internal/cli/progress_bars.py", line 53, in _rich_progress_bar\n for chunk in iterable:\n File "/work/src/.venv/lib/python3.10/site-packages/pip/_internal/network/utils.py", line 63, in response_chunks\n for chunk in response.raw.stream(\n File "/work/src/.venv/lib/python3.10/site-packages/pip/_vendor/urllib3/response.py", line 573, in stream\n data = self.read(amt=amt, decode_content=decode_content)\n File "/work/src/.venv/lib/python3.10/site-packages/pip/_vendor/urllib3/response.py", line 509, in read\n with self._error_catcher():\n File "/usr/local/lib/python3.10/contextlib.py", line 153, in exit\n self.gen.throw(typ, value, traceback)\n File "/work/src/.venv/lib/python3.10/site-packages/pip/_vendor/urllib3/response.py", line 440, in _error_catcher\n raise ReadTimeoutError(self._pool, None, "Read timed out.")\npip._vendor.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out.\n\n[notice] A new release of pip available: 22.2.2 -> 23.1.2\n[notice] To update, run: pip install --upgrade pip\nmake: *** [Makefile:52: .venv/bin/activate] Error 2\n'}
ERROR:root:Error conducting experiment. Extras: {'traceback': 'Traceback (most recent call last):\n File "/work/src/experiment/dispatcher.py", line 194, in main\n dispatcher_main()\n File "/work/src/experiment/dispatcher.py", line 148, in dispatcher_main\n trials = build_images_for_trials(experiment.fuzzers, experiment.benchmarks,\n File "/work/src/experiment/dispatcher.py", line 114, in build_images_for_trials\n builder.build_base_images()\n File "/work/src/experiment/build/builder.py", line 94, in build_base_images\n return buildlib.build_base_images()\n File "/work/src/experiment/build/local_build.py", line 38, in build_base_images\n return make(['base-image', 'worker'])\n File "/work/src/experiment/build/local_build.py", line 33, in make\n return new_process.execute(command, cwd=utils.ROOT_DIR)\n File "/work/src/common/new_process.py", line 125, in execute\n raise subprocess.CalledProcessError(retcode, command)\nsubprocess.CalledProcessError: Command '['make', '-j', 'base-image', 'worker']' returned non-zero exit status 2.\n'}
Traceback (most recent call last):
File "/work/src/experiment/dispatcher.py", line 212, in
sys.exit(main())
File "/work/src/experiment/dispatcher.py", line 197, in main
raise error
File "/work/src/experiment/dispatcher.py", line 194, in main
dispatcher_main()
File "/work/src/experiment/dispatcher.py", line 148, in dispatcher_main
trials = build_images_for_trials(experiment.fuzzers, experiment.benchmarks,
File "/work/src/experiment/dispatcher.py", line 114, in build_images_for_trials
builder.build_base_images()
File "/work/src/experiment/build/builder.py", line 94, in build_base_images
return buildlib.build_base_images()
File "/work/src/experiment/build/local_build.py", line 38, in build_base_images
return make(['base-image', 'worker'])
File "/work/src/experiment/build/local_build.py", line 33, in make
return new_process.execute(command, cwd=utils.ROOT_DIR)
File "/work/src/common/new_process.py", line 125, in execute
raise subprocess.CalledProcessError(retcode, command)
subprocess.CalledProcessError: Command '['make', '-j', 'base-image', 'worker']' returned non-zero exit status 2.
I've tried many things, but nothing seems to work. How do I solve this problem? :(
Does your machine have access to the internet? It sounds like in this case it could not connect to pypi to install packages.
Separately, the command that fails here is either make base-image, or make worker which you should also be able to just run locally in the root fuzzbench directory. In general I would recommend making all of the images outside of the run_experiment.py first, to see issues early and in an environment where they are easier to debug (outside of the experiment container)
So, in your example first make sure that all of the below commands build correctly and produce what looks like fuzzer output
make test-run-afl-libpng_libpng_read_fuzzer
make test-run-libfuzzer-libpng_libpng_read_fuzzer
make test-run-afl-libxml2_xml
make test-run-afl-libpng_libpng_read_fuzzer
I recommend following this guide to make sure everything works before jumping into running the actual experiment.
So, in your example first make sure that all of the below commands build correctly and produce what looks like fuzzer output
make test-run-afl-libpng_libpng_read_fuzzer make test-run-libfuzzer-libpng_libpng_read_fuzzer make test-run-afl-libxml2_xml make test-run-afl-libpng_libpng_read_fuzzerI recommend following this guide to make sure everything works before jumping into running the actual experiment.
Awesome, thank you so much! I just follow the steps in "Prerequisites" &"Running a local experiment ". It's really a good idea to "make test-xxx" separately. Thanks again!! I'll try to fix it