Can't run the tests (several errors) on Windows
olivierdalang opened this issue · comments
Hi,
Following the README.md, I was able to setup the vagrant VM, but the test commands don't work.
I downloaded the box manually from https://www.djangoproject.com/m/vms/djangocore-box-1.1.box (I have an unstable and slow connection, and stupidly, atlas.hashicorp doesn't support resuming downloads). I run Windows 10 64 bits, Virtualbox 5.0.20.
Disclaimer : I never used vagrant boxes before, so maybe I missed an implicit step.
~$ runtests2.7-mysql
fails with
Please install test dependencies first:
$ pip install -r requirements/py2.txt
Then
~$ pip install -r /django/tests/requirements/py2.txt
fails with
Exception:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 139, in main
status = self.run(options, args)
File "/usr/local/lib/python2.7/dist-packages/pip/commands/install.py", line 240, in run
for req in parse_requirements(filename, finder=finder, options=options):
File "/usr/local/lib/python2.7/dist-packages/pip/req.py", line 1314, in parse_requirements
for item in parse_requirements(req_url, finder, comes_from=filename, options=options):
File "/usr/local/lib/python2.7/dist-packages/pip/req.py", line 1356, in parse_requirements
req = InstallRequirement.from_line(line, comes_from)
File "/usr/local/lib/python2.7/dist-packages/pip/req.py", line 118, in from_line
return cls(req, comes_from, url=url)
File "/usr/local/lib/python2.7/dist-packages/pip/req.py", line 43, in __init__
req = pkg_resources.Requirement.parse(req)
File "/usr/local/lib/python2.7/dist-packages/distribute-0.6.35-py2.7.egg/pkg_resources.py", line 2680, in parse
reqs = list(parse_requirements(s))
File "/usr/local/lib/python2.7/dist-packages/distribute-0.6.35-py2.7.egg/pkg_resources.py", line 2605, in parse_requirements
line, p, specs = scan_list(VERSION,LINE_END,line,p,(1,2),"version spec")
File "/usr/local/lib/python2.7/dist-packages/distribute-0.6.35-py2.7.egg/pkg_resources.py", line 2573, in scan_list
raise ValueError("Expected "+item_name+" in",line,"at",line[p:])
ValueError: ('Expected version spec in', "pylibmc; sys.platform != 'win32'", 'at', "; sys.platform != 'win32'")
Python3.3 tests have another problem :
~$ runtests3.3-mysql
fails with
Traceback (most recent call last):
File "/django/tests/runtests.py", line 13, in <module>
from django.apps import apps
File "/django/django/apps/__init__.py", line 1, in <module>
from .config import AppConfig
File "/django/django/apps/config.py", line 6, in <module>
from django.utils.module_loading import module_has_submodule
File "/django/django/utils/module_loading.py", line 67, in <module>
from importlib.util import find_spec as importlib_find
ImportError: cannot import name find_spec
What version of pip is installed? Our requirements files require pip 8+.
The Python 3.3 failure is expected since master doesn't support Python 3. See https://docs.djangoproject.com/en/dev/faq/install/#what-python-version-can-i-use-with-django.
Oops I just saw I downloaded https://www.djangoproject.com/m/vms/djangocore-box-1.1.box , I misread the link and didn't make the difference between 1.1 and 1.11... Sorry for the noise !
I think I'll remove that image since it's obsolete. Is there a pointer to it that we could remove?
No I found the link on the mailing list thread.
But is the last version hosted somewhere else than on hashicorp's server ? They don't support resuming downloads (!!) making getting the boxes a challenge form here (I'm in Senegal, with a slow and unreliable connection).
I don't think so. Maybe you could contact them and ask if there's a reason it's not supported.
@olivierdalang you could try using the commands to build the box from scratch which only involves an initial 500mb (or so) to get the base ubuntu box. The provisioner will then download the rest of the software needed.
Alternatively, you can download the fully compiled box directly from hashicorp with the following url:
https://atlas.hashicorp.com/djangoproject/boxes/django-box-1.11/versions/1.11.0/virtualbox.box
Ok it works better with the correct version, but I still run into some errors.
(vm) ~$ runtests35-mysql
returned
ERROR: py35-mysql: InvocationError: /usr/bin/python3 -m virtualenv --python /usr/bin/python3.5 py35-mysql (see /django/.tox/py35-mysql/log/py35-mysql-0.log)
The log file displays
OSError: [Errno 71] Protocol error: '/usr/lib/python3.5/config-3.5m-x86_64-linux-gnu' -> '/django/.tox/py35-mysql/lib/python3.5/config-3.5m-x86_64-linux-gnu'
I think that's a Windows-specific error, where VirtualBox shared folders don't support symbolic links by default. I thought this should be fixable with this :
(host) "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" setextradata django-box_default_1474373922214_44815 VBoxInternal2/SharedFoldersEnableSymlinksCreate/django 1
But it didn't work.
I had more success by changing the tox working directory so that it's not on the shared folder anymore. In /django/tox.ini, add toxworkdir=/tmp/toxwork.
The annoying thing is that tox.ini is versioned. I didn't find a way to override the toxworkdir variable otherwise. Isn't there a way to set it using something like (vm) ~$ runtests35-mysql --toxworkdir=/tmp/toxwork ?
After this
(vm) ~$ runtests35-mysql
returns
py35-mysql: could not install deps [-rtests/requirements/py3.txt, -rtests/requirements/mysql.txt]; v = InvocationError('/tmp/toxwork/py35-mysql/bin/pip install -rtests/requirements/py3.txt -rtests/requirements/mysql.txt (see /tmp/toxwork/py35-mysql/log/py35-mysql-1.log)', 1)
The log file displays
actionid: py35-mysql
msg: getenv
cmdargs: [local('/tmp/toxwork/py35-mysql/bin/pip'), 'install', '-rtests/requirements/py3.txt', '-rtests/requirements/mysql.txt']
env: {'VIRTUAL_ENV': '/tmp/toxwork/py35-mysql', 'SSH_CONNECTION': '10.0.2.2 53363 10.0.2.15 22', 'PWD': '/home/vagrant', 'HOME': '/home/vagrant', 'PYTHONHASHSEED': '614178105', 'XDG_SESSION_ID': '2', 'SHELL': '/bin/bash', 'LOGNAME': 'vagrant', '_': '/usr/local/bin/tox', 'PATH': '/tmp/toxwork/py35-mysql/bin:/home/vagrant/bin:/home/vagrant/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games', 'LC_ALL': 'en_US.UTF-8', 'TERM': 'cygwin', 'LANG': 'en_US.UTF-8', 'DISPLAY': ':99', 'PYTHONPATH': '/home/vagrant/djangodata/', 'SSH_CLIENT': '10.0.2.2 53363 22', 'MAIL': '/var/mail/vagrant', 'XDG_RUNTIME_DIR': '/run/user/900', 'SSH_TTY': '/dev/pts/0', 'USER': 'vagrant', 'LANGUAGE': 'en_US.UTF-8', 'PYTHONDONTWRITEBYTECODE': '1', 'LESSCLOSE': '/usr/bin/lesspipe %s %s', 'SHLVL': '1', 'LESSOPEN': '| /usr/bin/lesspipe %s', 'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:'}
Collecting argon2-cffi>=16.1.0 (from -r tests/requirements/base.txt (line 1))
Using cached argon2_cffi-16.2.0.tar.gz
Collecting bcrypt (from -r tests/requirements/base.txt (line 2))
Using cached bcrypt-3.1.1-cp35-cp35m-manylinux1_x86_64.whl
Collecting docutils (from -r tests/requirements/base.txt (line 3))
Using cached docutils-0.12-py3-none-any.whl
Collecting geoip2 (from -r tests/requirements/base.txt (line 4))
Using cached geoip2-2.4.0-py2.py3-none-any.whl
Collecting jinja2>=2.7 (from -r tests/requirements/base.txt (line 5))
Using cached Jinja2-2.8-py2.py3-none-any.whl
Collecting numpy (from -r tests/requirements/base.txt (line 6))
Downloading numpy-1.11.1-cp35-cp35m-manylinux1_x86_64.whl (15.6MB)
Collecting Pillow (from -r tests/requirements/base.txt (line 7))
Downloading Pillow-3.3.1-cp35-cp35m-manylinux1_x86_64.whl (5.6MB)
Collecting PyYAML (from -r tests/requirements/base.txt (line 8))
Downloading PyYAML-3.12.tar.gz (253kB)
Collecting pylibmc (from -r tests/requirements/base.txt (line 10))
Downloading pylibmc-1.5.1.tar.gz (59kB)
Collecting pytz>dev (from -r tests/requirements/base.txt (line 11))
Downloading pytz-2016.6.1-py2.py3-none-any.whl (481kB)
Collecting selenium (from -r tests/requirements/base.txt (line 12))
Downloading selenium-2.53.6-py2.py3-none-any.whl (884kB)
Collecting sqlparse (from -r tests/requirements/base.txt (line 13))
Downloading sqlparse-0.2.1.tar.gz (53kB)
Collecting tblib (from -r tests/requirements/base.txt (line 14))
Downloading tblib-1.3.0-py2.py3-none-any.whl
Collecting python3-memcached (from -r tests/requirements/py3.txt (line 2))
Downloading python3-memcached-1.51.tar.gz
Collecting mysqlclient>=1.3.7 (from -r tests/requirements/mysql.txt (line 2))
Downloading mysqlclient-1.3.7.tar.gz (79kB)
Collecting six (from argon2-cffi>=16.1.0->-r tests/requirements/base.txt (line 1))
Downloading six-1.10.0-py2.py3-none-any.whl
Collecting cffi>=1.0.0 (from argon2-cffi>=16.1.0->-r tests/requirements/base.txt (line 1))
Downloading cffi-1.8.3-cp35-cp35m-manylinux1_x86_64.whl (397kB)
Collecting maxminddb>=1.2.1 (from geoip2->-r tests/requirements/base.txt (line 4))
Downloading maxminddb-1.2.1.tar.gz (1.3MB)
Collecting requests>=2.9 (from geoip2->-r tests/requirements/base.txt (line 4))
Downloading requests-2.11.1-py2.py3-none-any.whl (514kB)
Collecting MarkupSafe (from jinja2>=2.7->-r tests/requirements/base.txt (line 5))
Downloading MarkupSafe-0.23.tar.gz
Collecting pycparser (from cffi>=1.0.0->argon2-cffi>=16.1.0->-r tests/requirements/base.txt (line 1))
Downloading pycparser-2.14.tar.gz (223kB)
Building wheels for collected packages: argon2-cffi, PyYAML, pylibmc, sqlparse, python3-memcached, mysqlclient, maxminddb, MarkupSafe, pycparser
Running setup.py bdist_wheel for argon2-cffi: started
Running setup.py bdist_wheel for argon2-cffi: finished with status 'done'
Stored in directory: /home/vagrant/.cache/pip/wheels/f8/f2/af/213f3e49e3b54c3f2bd4ec2da7ca7cbfd8d0262a0d1db22eaa
Running setup.py bdist_wheel for PyYAML: started
Running setup.py bdist_wheel for PyYAML: finished with status 'done'
Stored in directory: /home/vagrant/.cache/pip/wheels/2c/f7/79/13f3a12cd723892437c0cfbde1230ab4d82947ff7b3839a4fc
Running setup.py bdist_wheel for pylibmc: started
Running setup.py bdist_wheel for pylibmc: finished with status 'error'
Complete output from command /tmp/toxwork/py35-mysql/bin/python3.5 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-zuota6lu/pylibmc/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" bdist_wheel -d /tmp/tmpbfuzimcopip-wheel- --python-tag cp35:
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.5
creating build/lib.linux-x86_64-3.5/pylibmc
copying src/pylibmc/__main__.py -> build/lib.linux-x86_64-3.5/pylibmc
copying src/pylibmc/test.py -> build/lib.linux-x86_64-3.5/pylibmc
copying src/pylibmc/consts.py -> build/lib.linux-x86_64-3.5/pylibmc
copying src/pylibmc/pools.py -> build/lib.linux-x86_64-3.5/pylibmc
copying src/pylibmc/client.py -> build/lib.linux-x86_64-3.5/pylibmc
copying src/pylibmc/__init__.py -> build/lib.linux-x86_64-3.5/pylibmc
warning: build_py: byte-compiling is disabled, skipping.
running build_ext
building '_pylibmc' extension
creating build/temp.linux-x86_64-3.5
creating build/temp.linux-x86_64-3.5/src
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DUSE_ZLIB -I/usr/include/python3.5m -I/tmp/toxwork/py35-mysql/include/python3.5m -c src/_pylibmcmodule.c -o build/temp.linux-x86_64-3.5/src/_pylibmcmodule.o -fno-strict-aliasing -std=c99
In file included from src/_pylibmcmodule.c:34:0:
src/_pylibmcmodule.h:42:36: fatal error: libmemcached/memcached.h: No such file or directory
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------------------------
Failed building wheel for pylibmc
Running setup.py clean for pylibmc
Running setup.py bdist_wheel for sqlparse: started
Running setup.py bdist_wheel for sqlparse: finished with status 'done'
Stored in directory: /home/vagrant/.cache/pip/wheels/e2/82/57/0a192624b6d4faa1d64d2b1f0efae857496b4f35fb7e86aba1
Running setup.py bdist_wheel for python3-memcached: started
Running setup.py bdist_wheel for python3-memcached: finished with status 'done'
Stored in directory: /home/vagrant/.cache/pip/wheels/be/3c/13/413aea983af79f9a4b4c2e737a6c8e7802c9cf0d69a7673b60
Running setup.py bdist_wheel for mysqlclient: started
Running setup.py bdist_wheel for mysqlclient: finished with status 'done'
Stored in directory: /home/vagrant/.cache/pip/wheels/09/64/5e/8f66e5b5ddef002da639a9b16b32ecedd4d56c51e6358e4e0b
Running setup.py bdist_wheel for maxminddb: started
Running setup.py bdist_wheel for maxminddb: finished with status 'done'
Stored in directory: /home/vagrant/.cache/pip/wheels/d3/5f/0b/8744068f80d2b3cf0772b529475a1617fa27d50f7bc818d8c3
Running setup.py bdist_wheel for MarkupSafe: started
Running setup.py bdist_wheel for MarkupSafe: finished with status 'done'
Stored in directory: /home/vagrant/.cache/pip/wheels/a3/fa/dc/0198eed9ad95489b8a4f45d14dd5d2aee3f8984e46862c5748
Running setup.py bdist_wheel for pycparser: started
Running setup.py bdist_wheel for pycparser: finished with status 'done'
Stored in directory: /home/vagrant/.cache/pip/wheels/9b/f4/2e/d03e949a551719a1ffcb659f2c63d8444f4df12e994ce52112
Successfully built argon2-cffi PyYAML sqlparse python3-memcached mysqlclient maxminddb MarkupSafe pycparser
Failed to build pylibmc
Installing collected packages: six, pycparser, cffi, argon2-cffi, bcrypt, docutils, maxminddb, requests, geoip2, MarkupSafe, jinja2, numpy, Pillow, PyYAML, pylibmc, pytz, selenium, sqlparse, tblib, python3-memcached, mysqlclient
Running setup.py install for pylibmc: started
Running setup.py install for pylibmc: finished with status 'error'
Complete output from command /tmp/toxwork/py35-mysql/bin/python3.5 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-zuota6lu/pylibmc/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-l0xc1do4-record/install-record.txt --single-version-externally-managed --compile --install-headers /tmp/toxwork/py35-mysql/include/site/python3.5/pylibmc:
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.5
creating build/lib.linux-x86_64-3.5/pylibmc
copying src/pylibmc/__main__.py -> build/lib.linux-x86_64-3.5/pylibmc
copying src/pylibmc/test.py -> build/lib.linux-x86_64-3.5/pylibmc
copying src/pylibmc/consts.py -> build/lib.linux-x86_64-3.5/pylibmc
copying src/pylibmc/pools.py -> build/lib.linux-x86_64-3.5/pylibmc
copying src/pylibmc/client.py -> build/lib.linux-x86_64-3.5/pylibmc
copying src/pylibmc/__init__.py -> build/lib.linux-x86_64-3.5/pylibmc
warning: build_py: byte-compiling is disabled, skipping.
running build_ext
building '_pylibmc' extension
creating build/temp.linux-x86_64-3.5
creating build/temp.linux-x86_64-3.5/src
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DUSE_ZLIB -I/usr/include/python3.5m -I/tmp/toxwork/py35-mysql/include/python3.5m -c src/_pylibmcmodule.c -o build/temp.linux-x86_64-3.5/src/_pylibmcmodule.o -fno-strict-aliasing -std=c99
In file included from src/_pylibmcmodule.c:34:0:
src/_pylibmcmodule.h:42:36: fatal error: libmemcached/memcached.h: No such file or directory
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------------------------
Command "/tmp/toxwork/py35-mysql/bin/python3.5 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-zuota6lu/pylibmc/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-l0xc1do4-record/install-record.txt --single-version-externally-managed --compile --install-headers /tmp/toxwork/py35-mysql/include/site/python3.5/pylibmc" failed with error code 1 in /tmp/pip-build-zuota6lu/pylibmc/vagrant@djangobox:~$
Here it looks there are missing dependencies. I didn't investigate further.
Are you using the version released yesterday? #14 (comment)
No. It looks it's the same error on that other issue, I didn't see it.
I'll test again when I have a chance to download the new box.
Hi,
Testing again with the last version, it now works with the following fixes :
- enable symlink for the virtual machine (replace django-box_default_1474373922214_44815 by the VM name in VirtualBox) :
(host) "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" setextradata django-box_default_1474373922214_44815 VBoxInternal2/SharedFoldersEnableSymlinksCreate/django 1 - make sure to run vagrant as an administrator
Thanks for the support ! I'll make a PR for the readme to add these instructions.
Well sorry I just thought about it again, I think this VBoxInternal2/SharedFoldersEnableSymlinksCreate/django setting could be set to true in the .box file, right ? If so, the only needed step for a windows user would be to run vagrant as an admin.
Thanks for all of this @olivierdalang . I'm acknowledging that I've got this on my radar, it's just been a busy couple of weeks, and will be a busy couple more.
Have you figured out if it's possible to allow symlinks from within the Vagrantfile? Ideally users shouldn't have to mess around with the virtualbox app directly.
Looks like you can control this option from the vagrant file. From http://stackoverflow.com/a/24365338/10583
config.vm.provider "virtualbox" do |v|
v.customize ["setextradata", :id, "VBoxInternal2/SharedFoldersEnableSymlinksCreate/v-root", "1"]
end
Great, that should do. With this, the only step one has to remember is to run vagrant as an administrator.
I tried and tried and tried to make this work, but couldn't. I even resorted to calling SharedFOldersEnableSymlinksCreate passing the value of the machine to no avail. I was trying under Windows 10, with VirtualBox 5.1.8 and Vagrant 1.8.6. Everything I tried was from an elevated shell.
Should the SharedFolders command be run against a running virtual machine or a provisioned but halted one?
@olivierdalang are you able to try modifying the Vagrantfile and adding the following block:
config.vm.provider "virtualbox" do |v|
v.customize ["setextradata", :id, "VBoxInternal2/SharedFoldersEnableSymlinksCreate/django", "1"]
end
Then run vagrant up from an elevated shell and let me know if it provisions correctly?
Acoording to this blog post: https://blog.puphpet.com/blog/2015/06/25/windows-symlinks/, linked from hashicorp/vagrant#6024 (comment) there are three things needed to get sysliks to work on a Linux VirtuaBox guest under a Vagrant host:
- Setting
"setextradata", :id, "VBoxInternal2/SharedFoldersEnableSymlinksCreate/for the folder(s) where this is needed - Running Vagrant from a shell executed ad an administrator
- Adding the Windows user to the users/groups listed in
SeCreateSymbolicLinkPrivilege(SE_CREATE_SYMBOLIC_LINK_NAME) using a policy editor and then rebooting the system
It seems we've already figured out the two former ones.
Unfortunately I'm not able to confirm the this right now; posting this in case someone can test it.