Unable to run fuzzbench with docker rootless
ericspring08 opened this issue · comments
I am running fuzzbench with this command:
PYTHONPATH=. python3 experiment/run_experiment.py \
--experiment-config config.yaml \
--benchmarks sqlite3_ossfuzz \
--experiment-name experiment \
--fuzzers aflplusplus
After the experiment starts this error appears and the experiment terminates:
ERROR:root:Executed command: "make -j base-image worker" returned: 2. Extras: {'output': "ngine_Qt5-5.15.2-py3-none-manylinux2014_x86_64.whl (67.5 MB)\n ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 67.5/67.5 MB 13.4 MB/s eta 0:00:00\nCollecting charset-normalizer<3,>=2\n Downloading charset_normalizer-2.1.0-py3-none-any.whl (39 kB)\nCollecting urllib3<1.27,>=1.21.1\n Downloading urllib3-1.26.11-py2.py3-none-any.whl (139 kB)\n ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 139.9/139.9 kB 15.9 MB/s eta 0:00:00\nCollecting idna<4,>=2.5\n Downloading idna-3.3-py3-none-any.whl (61 kB)\n ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.2/61.2 kB 23.7 MB/s eta 0:00:00\nCollecting threadpoolctl>=2.0.0\n Downloading threadpoolctl-3.1.0-py3-none-any.whl (14 kB)\nCollecting requests-oauthlib>=0.7.0\n Downloading requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)\nCollecting google-cloud-datastore\n Downloading google_cloud_datastore-1.15.5-py2.py3-none-any.whl (134 kB)\n ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 134.2/134.2 kB 40.8 MB/s eta 0:00:00\nCollecting pymemcache\n Downloading pymemcache-3.5.2-py2.py3-none-any.whl (56 kB)\n ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.0/57.0 kB 28.2 MB/s eta 0:00:00\nCollecting google-cloud-storage\n Downloading google_cloud_storage-2.3.0-py2.py3-none-any.whl (107 kB)\n ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 107.1/107.1 kB 40.3 MB/s eta 0:00:00\n Downloading google_cloud_storage-2.2.1-py2.py3-none-any.whl (107 kB)\n ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 107.1/107.1 kB 44.7 MB/s eta 0:00:00\nCollecting google-resumable-media>=2.3.2\n Downloading google_resumable_media-2.3.3-py2.py3-none-any.whl (76 kB)\n ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 76.9/76.9 kB 8.4 MB/s eta 0:00:00\nCollecting jeepney>=0.4.2\n Downloading jeepney-0.8.0-py3-none-any.whl (48 kB)\n ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.4/48.4 kB 5.2 MB/s eta 0:00:00\nCollecting importlib-metadata>=3.6\n Downloading importlib_metadata-4.12.0-py3-none-any.whl (21 kB)\nCollecting SecretStorage>=3.2\n Downloading SecretStorage-3.3.2-py3-none-any.whl (15 kB)\nCollecting mozinfo\n Using cached mozinfo-1.2.2.tar.gz (5.8 kB)\n Preparing metadata (setup.py): started\n Preparing metadata (setup.py): finished with status 'done'\nCollecting et-xmlfile\n Downloading et_xmlfile-1.1.0-py3-none-any.whl (4.7 kB)\nCollecting patsy>=0.5.2\n Downloading patsy-0.5.2-py2.py3-none-any.whl (233 kB)\n ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 233.7/233.7 kB 17.5 MB/s eta 0:00:00\nCollecting google-crc32c<2.0dev,>=1.0\n Downloading google_crc32c-1.3.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (37 kB)\nCollecting zipp>=0.5\n Downloading zipp-3.8.1-py3-none-any.whl (5.6 kB)\nCollecting oauthlib>=3.0.0\n Downloading oauthlib-3.2.0-py3-none-any.whl (151 kB)\n ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 151.5/151.5 kB 44.9 MB/s eta 0:00:00\nCollecting cryptography>=2.0\n Downloading cryptography-37.0.4-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.2 MB)\n ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.2/4.2 MB 29.9 MB/s eta 0:00:00\nCollecting mypy-extensions>=0.3.0\n Using cached mypy_extensions-0.4.3-py2.py3-none-any.whl (4.5 kB)\nCollecting msgpack>=0.5.2\n Downloading msgpack-1.0.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (322 kB)\n ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 322.5/322.5 kB 14.2 MB/s eta 0:00:00\nCollecting lockfile>=0.9\n Using cached lockfile-0.12.2-py2.py3-none-any.whl (13 kB)\nCollecting distro>=1.4.0\n Downloading distro-1.7.0-py3-none-any.whl (20 kB)\nCollecting mozfile>=0.12\n Using cached mozfile-2.1.0.tar.gz (5.9 kB)\n Preparing metadata (setup.py): started\n Preparing metadata (setup.py): finished with status 'done'\nCollecting cffi>=1.12\n Downloading cffi-1.15.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (442 kB)\n ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 442.7/442.7 kB 24.8 MB/s eta 0:00:00\nCollecting pycparser\n Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB)\n ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 118.7/118.7 kB 39.6 MB/s eta 0:00:00\nUsing legacy 'setup.py install' for alembic, since package 'wheel' is not installed.\nUsing legacy 'setup.py install' for importlab, since package 'wheel' is not installed.\nUsing legacy 'setup.py install' for python-louvain, since package 'wheel' is not installed.\nUsing legacy 'setup.py install' for google-cloud-profiler, since package 'wheel' is not installed.\nUsing legacy 'setup.py install' for serverfiles, since package 'wheel' is not installed.\nUsing legacy 'setup.py install' for wrapt, since package 'wheel' is not installed.\nUsing legacy 'setup.py install' for mozinfo, since package 'wheel' is not installed.\nUsing legacy 'setup.py install' for mozfile, since package 'wheel' is not installed.\nInstalling collected packages: yapf, wrapt, rfc3986, pytz, python-editor, PyQtWebEngine-Qt5, PyQt5-Qt5, pyfakefs, pyasn1, ninja, mypy-extensions, msgpack, mccabe, lockfile, iniconfig, dictdiffer, commonmark, zipp, xlsxwriter, xlrd, urllib3, uritemplate, typing-extensions, typed-ast, toml, threadpoolctl, sqlalchemy, sniffio, six, rsa, redis, qasync, PyYAML, PyQt5-sip, pyparsing, pycparser, pyasn1-modules, py, psycopg2-binary, psutil, protobuf, pluggy, pillow, oauthlib, numpy, networkx, MarkupSafe, lazy-object-proxy, kiwisolver, keyrings.alt, joblib, jeepney, isort, idna, h11, google-crc32c, fonttools, et-xmlfile, docutils, distro, cycler, click, charset-normalizer, chardet, certifi, cachetools, attrs, AnyQt, typing-inspect, scipy, rq, requests, python-louvain, python-dateutil, pyqtgraph, PyQt5, pymemcache, proto-plus, patsy, packaging, openpyxl, mozfile, Mako, Jinja2, importlib-metadata, importlab, httplib2, httpcore, grpcio, googleapis-common-protos, google-resumable-media, google-auth, cffi, bottleneck, astroid, serverfiles, scikit-learn, requests-oauthlib, pytype, pytest, PyQtWebEngine, pylint, pandas, oauth2client, mozinfo, matplotlib, libcst, httpx, google-auth-httplib2, google-api-core, cryptography, cachecontrol, alembic, statsmodels, SecretStorage, seaborn, openTSNE, mozprocess, grpc-google-iam-v1, google-cloud-core, google-auth-oauthlib, google-api-python-client, baycomp, scikit-posthocs, orange-canvas-core, keyring, google-cloud-storage, google-cloud-secret-manager, google-cloud-profiler, google-cloud-monitoring, google-cloud-logging, google-cloud-datastore, orange-widget-base, google-cloud-ndb, google-cloud-error-reporting, Orange3, clusterfuzz\n Running setup.py install for wrapt: started\n Running setup.py install for wrapt: finished with status 'done'\n Running setup.py install for python-louvain: started\n Running setup.py install for python-louvain: finished with status 'done'\n Running setup.py install for mozfile: started\n Running setup.py install for mozfile: finished with status 'done'\n Running setup.py install for importlab: started\n Running setup.py install for importlab: finished with status 'done'\n Running setup.py install for serverfiles: started\n Running setup.py install for serverfiles: finished with status 'done'\n Running setup.py install for mozinfo: started\n Running setup.py install for mozinfo: finished with status 'done'\n Running setup.py install for alembic: started\n Running setup.py install for alembic: finished with status 'done'\n Running setup.py install for google-cloud-profiler: started\n Running setup.py install for google-cloud-profiler: finished with status 'done'\nSuccessfully installed AnyQt-0.1.1 Jinja2-2.11.3 Mako-1.2.1 MarkupSafe-2.1.1 Orange3-3.28.0 PyQt5-5.15.7 PyQt5-Qt5-5.15.2 PyQt5-sip-12.11.0 PyQtWebEngine-5.15.6 PyQtWebEngine-Qt5-5.15.2 PyYAML-5.4 SecretStorage-3.3.2 alembic-1.4.0 astroid-2.6.6 attrs-22.1.0 baycomp-1.0.2 bottleneck-1.3.5 cachecontrol-0.12.11 cachetools-4.2.4 certifi-2022.6.15 cffi-1.15.1 chardet-5.0.0 charset-normalizer-2.1.0 click-8.1.3 clusterfuzz-0.0.1a0 commonmark-0.9.1 cryptography-37.0.4 cycler-0.11.0 dictdiffer-0.9.0 distro-1.7.0 docutils-0.19 et-xmlfile-1.1.0 fonttools-4.34.4 google-api-core-1.32.0 google-api-python-client-2.5.0 google-auth-1.30.1 google-auth-httplib2-0.1.0 google-auth-oauthlib-0.5.2 google-cloud-core-1.7.3 google-cloud-datastore-1.15.5 google-cloud-error-reporting-1.1.2 google-cloud-logging-1.15.1 google-cloud-monitoring-2.10.1 google-cloud-ndb-1.11.1 google-cloud-profiler-4.0.0 google-cloud-secret-manager-2.4.0 google-cloud-storage-2.2.1 google-crc32c-1.3.0 google-resumable-media-2.3.3 googleapis-common-protos-1.56.4 grpc-google-iam-v1-0.12.4 grpcio-1.48.0 h11-0.13.0 httpcore-0.12.3 httplib2-0.20.4 httpx-0.16.1 idna-3.3 importlab-0.7 importlib-metadata-4.12.0 iniconfig-1.1.1 isort-5.10.1 jeepney-0.8.0 joblib-1.1.0 keyring-23.7.0 keyrings.alt-4.1.1 kiwisolver-1.4.4 lazy-object-proxy-1.7.1 libcst-0.4.7 lockfile-0.12.2 matplotlib-3.5.2 mccabe-0.6.1 mozfile-2.1.0 mozinfo-1.2.2 mozprocess-1.3.0 msgpack-1.0.4 mypy-extensions-0.4.3 networkx-2.8.5 ninja-1.10.2.3 numpy-1.18.1 oauth2client-4.1.3 oauthlib-3.2.0 openTSNE-0.6.2 openpyxl-3.0.10 orange-canvas-core-0.1.27 orange-widget-base-4.17.0 packaging-21.3 pandas-1.2.4 patsy-0.5.2 pillow-9.2.0 pluggy-0.13.1 proto-plus-1.20.6 protobuf-3.20.1 psutil-5.9.1 psycopg2-binary-2.8.4 py-1.11.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.21 pyfakefs-3.7.1 pylint-2.7.4 pymemcache-3.5.2 pyparsing-3.0.9 pyqtgraph-0.12.4 pytest-6.1.2 python-dateutil-2.8.1 python-editor-1.0.4 python-louvain-0.16 pytype-2021.4.15 pytz-2019.3 qasync-0.23.0 redis-3.5.3 requests-2.28.1 requests-oauthlib-1.3.1 rfc3986-1.5.0 rq-1.4.3 rsa-4.9 scikit-learn-1.1.1 scikit-posthocs-0.6.2 scipy-1.6.2 seaborn-0.11.1 serverfiles-0.3.1 six-1.16.0 sniffio-1.2.0 sqlalchemy-1.3.19 statsmodels-0.13.2 threadpoolctl-3.1.0 toml-0.10.2 typed-ast-1.5.4 typing-extensions-4.3.0 typing-inspect-0.7.1 uritemplate-3.0.1 urllib3-1.26.11 wrapt-1.12.1 xlrd-2.0.1 xlsxwriter-3.0.3 yapf-0.30.0 zipp-3.8.1\nsource .venv/bin/activate && PYTHONPATH=. python3 docker/generate_makefile.py docker/generated.mk\ndocker pull ubuntu:xenial\nGot permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.39/images/create?fromImage=ubuntu&tag=xenial: dial unix /var/run/docker.sock: connect: permission denied\ndocker/generated.mk:102: recipe for target 'base-image' failed\nmake: *** [base-image] Error 1\n"}
ERROR:root:Error conducting experiment. Extras: {'traceback': 'Traceback (most recent call last):\n File "/work/src/experiment/dispatcher.py", line 204, in main\n dispatcher_main()\n File "/work/src/experiment/dispatcher.py", line 157, in dispatcher_main\n trials = build_images_for_trials(experiment.fuzzers, experiment.benchmarks,\n File "/work/src/experiment/dispatcher.py", line 117, in build_images_for_trials\n builder.build_base_images()\n File "/work/src/experiment/build/builder.py", line 77, in build_base_images\n return buildlib.build_base_images()\n File "/work/src/experiment/build/local_build.py", line 38, in build_base_images\n return make([\'base-image\', \'worker\'])\n File "/work/src/experiment/build/local_build.py", line 33, in make\n return new_process.execute(command, cwd=utils.ROOT_DIR)\n File "/work/src/common/new_process.py", line 124, in execute\n raise subprocess.CalledProcessError(retcode, command)\nsubprocess.CalledProcessError: Command \'[\'make\', \'-j\', \'base-image\', \'worker\']\' returned non-zero exit status 2.\n'}
Traceback (most recent call last):
File "/work/src/experiment/dispatcher.py", line 222, in <module>
sys.exit(main())
File "/work/src/experiment/dispatcher.py", line 207, in main
raise error
File "/work/src/experiment/dispatcher.py", line 204, in main
dispatcher_main()
File "/work/src/experiment/dispatcher.py", line 157, in dispatcher_main
trials = build_images_for_trials(experiment.fuzzers, experiment.benchmarks,
File "/work/src/experiment/dispatcher.py", line 117, in build_images_for_trials
builder.build_base_images()
File "/work/src/experiment/build/builder.py", line 77, in build_base_images
return buildlib.build_base_images()
File "/work/src/experiment/build/local_build.py", line 38, in build_base_images
return make(['base-image', 'worker'])
File "/work/src/experiment/build/local_build.py", line 33, in make
return new_process.execute(command, cwd=utils.ROOT_DIR)
File "/work/src/common/new_process.py", line 124, in execute
raise subprocess.CalledProcessError(retcode, command)
subprocess.CalledProcessError: Command '['make', '-j', 'base-image', 'worker']' returned non-zero exit status 2.
It's pretty hard to see what the error is from what you posted but I think it's this:
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.39/images/create?fromImage=ubuntu&tag=xenial: dial unix /var/run/docker.sock: connect: permission denied\ndocker/[generated.mk:102](http://generated.mk:102/): recipe for target 'base-image' failed\nmake: *** [base-image] Error 1\n"}
What do you mean docker rootless? I don't run docker with root, I added the user I run fuzzbench with to the docker group. Can you do the same?
In my current situation I cannot be added to the docker group, however I believe the issue may be fuzzbench hard coding the docker path when dispatching the docker container.
We are. Do you have an alternate suggestion?
I guess probably running docker-in-docker which I don't want to do as it will hurt performance for everyone else.
We are. Do you have an alternate suggestion?
I am working on a way for fuzzbench to dynamically access the docker socket. Thank should fix this issue.