make docker-build fails
erikr opened this issue · comments
Error description
I freshly cloned this repo on a new Ubuntu 20.04 VM, installed miniconda3, and followed the instructions at Run BioPhi dev server through Docker Compose.
Running make docker-build throws an error:
$ make docker-build
docker-compose build
redis uses an image, skipping
mongo uses an image, skipping
Building worker
Sending build context to Docker daemon 29.63MB
Step 1/12 : FROM continuumio/miniconda
latest: Pulling from continuumio/miniconda
Digest: sha256:fee1354ae2435522b9a8a79c5f1c406facc07ec5c44d730d8053600b37c924f0
Status: Downloaded newer image for continuumio/miniconda:latest
---> b8ea69b5c41c
Step 2/12 : RUN apt-get update -y && apt-get install -y --no-install-recommends build-essential
---> Using cache
---> 9a6c9d2094d4
Step 3/12 : RUN conda config --add channels bioconda && conda config --add channels conda-forge
---> Using cache
---> c3292b2e0f58
Step 4/12 : WORKDIR /opt/biophi
---> Using cache
---> 188452c8dc8c
Step 5/12 : COPY environment.yml .
---> 40000c92604e
Step 6/12 : COPY Makefile .
---> 35a9f24f85cf
Step 7/12 : RUN make env-update ENV_NAME=base
---> Running in 74fb699b3d87
conda env update -n base -f environment.yml
Collecting package metadata (repodata.json): ...working... make: *** [Makefile:12: env-upda
te] Killed
The command '/bin/sh -c make env-update ENV_NAME=base' returned a non-zero code: 2
ERROR: Service 'worker' failed to build : Build failed
make: *** [Makefile:18: docker-build] Error 1So I ran make env-create:
base $ make env-create
conda env create -n biophi-dev -f environment.yml
Collecting package metadata (repodata.json): done
Solving environment: failed
ResolvePackageNotFound:
- abnumber==0.2.7
make: *** [Makefile:9: env-create] Error 1So I added the env name and channels to environment.yml:
name: biophi-test
channels:
- defaults
- bioconda
- anaconda
- conda-forge
dependencies:
- python = 3.8
- pip
- abnumber == 0.2.7
- pip:
- click >= 7
- pandas
- sqlalchemy
- flask
- redis
- celery
- biopython
- pytest
- requests
- tqdm
- xlsxwriter
- humanize
- fairseq == 0.10.2
and re-ran make env-create, which now gives a new error:
base $ make env-create
conda env create -n biophi-dev -f environment.yml
Collecting package metadata (repodata.json): done
Solving environment: done
Downloading and Extracting Packages
pip-21.2.2 | 1.8 MB | ################################################ | 100%
python-dateutil-2.8. | 233 KB | ################################################ | 100%
pytz-2021.1 | 181 KB | ################################################ | 100%
mkl_fft-1.3.0 | 180 KB | ################################################ | 100%
bottleneck-1.3.2 | 125 KB | ################################################ | 100%
pandas-1.3.1 | 9.6 MB | ################################################ | 100%
python-3.8.11 | 18.2 MB | ################################################ | 100%
biopython-1.78 | 2.1 MB | ################################################ | 100%
mkl-2021.3.0 | 141.2 MB | ################################################ | 100%
anarci-2021.02.04 | 1.1 MB | ################################################ | 100%
blas-1.0 | 6 KB | ################################################ | 100%
numpy-base-1.20.3 | 4.5 MB | ################################################ | 100%
mkl-service-2.4.0 | 59 KB | ################################################ | 100%
hmmer-3.3.2 | 9.6 MB | ################################################ | 100%
numpy-1.20.3 | 23 KB | ################################################ | 100%
numexpr-2.7.3 | 188 KB | ################################################ | 100%
mkl_random-1.2.2 | 308 KB | ################################################ | 100%
abnumber-0.2.7 | 33 KB | ################################################ | 100%
intel-openmp-2021.3. | 1.4 MB | ################################################ | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Installing pip dependencies: - Ran pip subprocess with arguments:
['/home/ubuntu/miniconda3/envs/biophi-dev/bin/python', '-m', 'pip', 'install', '-U', '-r', '/home/ubuntu/BioPhi/condaenv.4ik288eg.requirements.txt']
Pip subprocess output:
Collecting click>=7
Using cached click-8.0.1-py3-none-any.whl (97 kB)
Requirement already satisfied: pandas in /home/ubuntu/miniconda3/envs/biophi-dev/lib/python3.8/site-packages (from -r /home/ubuntu/BioPhi/condaenv.4ik288eg.requirements.txt (line 2)) (1.3.1)
Collecting sqlalchemy
Using cached SQLAlchemy-1.4.22-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB)
Collecting flask
Using cached Flask-2.0.1-py3-none-any.whl (94 kB)
Collecting redis
Using cached redis-3.5.3-py2.py3-none-any.whl (72 kB)
Collecting celery
Using cached celery-5.1.2-py3-none-any.whl (401 kB)
Requirement already satisfied: biopython in /home/ubuntu/miniconda3/envs/biophi-dev/lib/python3.8/site-packages (from -r /home/ubuntu/BioPhi/condaenv.4ik288eg.requirements.txt (line 7)) (1.78)
Collecting biopython
Using cached biopython-1.79-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (2.3 MB)
Collecting pytest
Using cached pytest-6.2.4-py3-none-any.whl (280 kB)
Collecting requests
Using cached requests-2.26.0-py2.py3-none-any.whl (62 kB)
Collecting tqdm
Using cached tqdm-4.62.0-py2.py3-none-any.whl (76 kB)
Collecting xlsxwriter
Using cached XlsxWriter-3.0.1-py3-none-any.whl (148 kB)
Collecting humanize
Using cached humanize-3.11.0-py3-none-any.whl (90 kB)
Collecting fairseq==0.10.2
Using cached fairseq-0.10.2-cp38-cp38-manylinux1_x86_64.whl (1.7 MB)
Collecting dataclasses
Using cached dataclasses-0.6-py3-none-any.whl (14 kB)
Collecting cython
Using cached Cython-0.29.24-cp38-cp38-manylinux1_x86_64.whl (1.9 MB)
Collecting regex
Using cached regex-2021.8.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (738 kB)
Collecting sacrebleu>=1.4.12
Using cached sacrebleu-2.0.0-py3-none-any.whl (90 kB)
Collecting hydra-core
Using cached hydra_core-1.1.0-py3-none-any.whl (144 kB)
Collecting torch
Pip subprocess error:
/home/ubuntu/miniconda3/envs/biophi-dev/.tmp4b21tyj7: line 3: 32406 Killed /home/ubuntu/miniconda3/envs/biophi-dev/bin/python -m pip install -U -r /home/ubuntu/BioPhi/condaenv.4ik288eg.requirements.txt
failed
CondaEnvException: Pip failed
make: *** [Makefile:9: env-create] Error 1Expected behavior
make env-create does not throw an error. Strangely this worked last night but the errors began around 10 pm EST.
Hi @erikr thanks for the detailed report. It's definitely a good idea to add the channels at the top of the yml file, can you create a pull request? The required channels are added as one of the steps in the dockerfile, but they will be useful if people try to install it locally just as you did.
As for the crashes, it appears that the workers are killed for an unrelated reason. Could it be a memory issue?
Hi @prhoda, thanks for the input. Yes, will fork and open PR.
Previously I used a VM with 4 GB of RAM. Now I am using 32 GB RAM and cannot reproduce the error. It would be ideal to have a clearer understanding of minimum memory requirements, e.g., if the README says "If running BioPhi on a local machine or server, we recommend having at least x GB of RAM".
(now that I can successfully run docker-build, I now have a positive control to assess if my PR breaks setup)
I will close this issue now since the problem disappeared, but if there is interest to hunt down the root cause, this issue can always be re-opened.
@erikr FYI We set up the university server on Ubuntu without Docker. Feel free to use this repo as a guide: https://github.com/lich-uct/biophi.dichlab.org
I also added this link to the README.
Also FYI, I removed the mongodb and beat services from the docker-compose file because these are actually not needed anymore.