Azure Web Apps fail to install Azure-Storage 0.33.0

Question

Azure Web Apps fail to install Azure-Storage 0.33.0

4c74356b41 opened this issue 8 years ago · comments

Hello, issue is easily reproduceable:
Create a flask web app add azure-storage to requirements.txt and try to push it to the Web App... It will fail with 2.7 and 3.4. See attached log file.

Running setup.py install for cryptography
BLA-BLA-BLA
remote: distutils.errors.DistutilsError: Setup script exited with error: Unable to find vcvarsall.bat
remote:
remote: ----------------------------------------
remote: Cleaning up...
remote: .........................
remote: Command D:\home\site\wwwroot\env\Scripts\python.exe -c "import setuptools, tokenize;__file__='D:\\home\\site\\wwwroot\\env\\build\\cryptography\\setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record D:\local\Temp\pip-1tk19zep-record\install-record.txt --single-version-externally-managed --compile --install-headers D:\home\site\wwwroot\env\include\site\python3.4 failed with error code 1 in D:\home\site\wwwroot\env\build\cryptography
remote: Storing debug log for failure in D:\home\pip\pip.log
remote: An error has occurred during web site deployment.

You owe me a couple of days of my life :(
So it works for any version without that ~~stupid~~ broken cryptography.

fail.txt

Ram B · Answer 1 · Wed Sep 07 2016 03:52:11 GMT+0800 (China Standard Time)

Hi @4c74356b41,
Sorry you are having trouble with this. The reason you're getting this error is because the cryptography package needs to be built natively, and it is looking for a compiler (VS) to start the build process.

If have Visual Studio/C++ already installed,
Type set in command prompt, it may be the case these environmental vars are not populated (depending on your version):

VS110COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\Tools\
VS120COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\Tools\
VS140COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\Tools\

You can set them to your VS installation path manually. However, the setuptools we use should be able to autodetect your compiler, so first verify you are using the latest version (and if not, retry the deployment):
pip install --upgrade setuptools

If you still have trouble deploying our storage library with cryptography or don't have a compiler installed, installing the following should take care of it:
https://www.microsoft.com/en-us/download/details.aspx?id=44266

Gleb Boushev · Answer 2 · Wed Sep 07 2016 05:06:32 GMT+0800 (China Standard Time)

Hello Rambo (sounds extremely cool), I'm talking about Azure WebApps only, this stuff works on my PC no issues. Only when I push my code to Azure Web Apps it fails to install it.

Laurent Mazuel · Answer 3 · Wed Sep 07 2016 05:25:35 GMT+0800 (China Standard Time)

Hi @4c74356b41

For some reason, pip on WebApp didn't choose to download the wheel file for cryptography on PyPI. I guess the introspection of pip wasn't able to get the "win32" or "win_amd64" tags on this system.

I suggest you to try the options on this document:
https://azure.microsoft.com/en-us/documentation/articles/web-sites-python-configure/#troubleshooting---package-installation

IOW, either download cryptography-1.5-cp34-cp34m-win32.whl, put it in a "wheelhouse" folder and use --find-links or a deployment script using easy_install

As the previous document says, you will not be able to compile cryptography directly on the WebApp server, so you need a way to force the install of the wheel file.

I hope it helps,

Laurent Mazuel · Answer 4 · Wed Sep 07 2016 05:46:28 GMT+0800 (China Standard Time)

I think you will find this article interesting too:
https://blogs.msdn.microsoft.com/pythonengineering/2016/08/04/upgrading-python-on-azure-app-service/

Hugues Valois · Answer 5 · Wed Sep 07 2016 06:25:44 GMT+0800 (China Standard Time)

We've had customers that hit a bug in pip where it can't use/find wheels even when they are available. It's fixed in the latest version of pip, but unfortunately the pip on webapps is not the latest.

Not sure if that's what's happening with the crypto package or not.

Gleb Boushev · Answer 6 · Wed Sep 07 2016 15:19:00 GMT+0800 (China Standard Time)

Look folks, I've worked around that, I don't really need Azure Storage 0.33.0 exactly. I'm generally telling you to forward this ISSUE to the web app team, because right now this thing is broken (as is).
But! I really thank you for all those links, I'll read through them!

oh yes, that article is truly epic, thanks!

Ram B · Answer 7 · Sat Sep 17 2016 09:04:30 GMT+0800 (China Standard Time)

Thanks! I'll consider this resolved, pending updates from the WebApps team.

Gleb Boushev · Answer 8 · Sat Sep 17 2016 18:30:58 GMT+0800 (China Standard Time)

works for me, thanks!

Mary Wahl · Answer 9 · Fri Sep 30 2016 22:45:13 GMT+0800 (China Standard Time)

FWIW, I've found a workaround for installing the cryptography package on Azure WebJobs.

pip will only install wheels with an ABI code of "none" during Web App deployment:

import pip
print(pip.pep425tags.get_supported())
[09/28/2016 17:08:30 > 775106: INFO] [('cp34', 'none', 'win32'), ('cp34', 'none', 'any'), ('cp3', 'none', 'any'), ('cp33', 'none', 'any'), ('cp32', 'none', 'any'), ('cp31', 'none', 'any'), ('cp30', 'none', 'any'), ('py34', 'none', 'any'), ('py3', 'none', 'any'), ('py33', 'none', 'any'), ('py32', 'none', 'any'), ('py31', 'none', 'any'), ('py30', 'none', 'any')]

Many modern packages in PyPI -- e.g. cryptography==1.5.2 -- don't provide wheels with this ABI tag. Fortunately, it appears all will be well if the wheel file is renamed to have the "none" ABI tag.

I'll assume that you are already deploying your Web App from a git repo. Here are the steps to get cryptography installed:

Download the appropriate wheel for your Web App's Python version and the win32 platform from the cryptography page on PyPI into your Web App's repo.
Rename the file to change the ABI tag to "none", e.g.
cryptography-1.5.2-cp34-cp34m-win32.whl =>cryptography-1.5.2-cp34-none-win32.whl
Repeat steps 1 and 2 for the cffi package.
Add entries at the top of the requirements,txt file containing the filenames of the wheels.
Deploy via git.

I believe that'd be sufficient, though there are a few more steps I have taken (generating and including wheels for azure-common, azure-nspkg, and azure-storage via pip wheel <package name>, including other dependencies as wheels) that might also be necessary.

Ahmet · Answer 10 · Tue Nov 08 2016 04:50:23 GMT+0800 (China Standard Time)

Couldn't get any of this to work with 64bit, 3.4. Downgraded azure-storage to 0.32.0 and all is good. Hope to see this fixed soon.

michal_s · Answer 11 · Wed Jan 04 2017 00:53:05 GMT+0800 (China Standard Time)

It is not resolved and it should not be closed. It is an issue between Storage SDK for Python team and Web Apps team (Web Apps PG?). It should be fixed as soon as possible because it is real production blocker. If help is needed, just inform us how to help or where to ask directly.

Steve Dower · Answer 12 · Fri Jan 06 2017 02:02:20 GMT+0800 (China Standard Time)

The best solution is to upgrade the version of Python on app service by following the instructions at https://aka.ms/PythonOnAppService.

The version that is preinstalled is very old, but for backwards-compatibility reasons we cannot upgrade it, so instead we are deprecating use of the preinstalled version and recommending everyone use the site extensions described at the link above.

And yes, we intend to improve this process so it is simpler in the future, but until the App Service team is able to help us out (right now we have to fix it with only publicly-available APIs/features), there are going to be a few extra manual steps involved. Luckily, most of those just make the process more transparent, so it isn't necessarily a bad thing, but it does complicate the process somewhat.

tadashiee · Answer 13 · Fri Jan 06 2017 17:44:30 GMT+0800 (China Standard Time)

I encountered this problem, too. But, I found the workarround of this problem .
I describe our environment, problem, primary cause, and workarround below:
[[ Environment ]]
** Azure Web Apps **
python 3.4.1 (32bit)
** My development PC **

Windows 7 (32bit)
python 3.4.1 (32bit)
pip (1.5.6)
setuptools (2.1)
wheel packages in the 'wheelhouse' directory
azure_common-1.1.4-py2.py3-none-any.whl
azure_nspkg-1.0.0-py2.py3-none-any.whl
azure_storage-0.33.0-py3-none-any.whl
cffi-1.9.1-cp34-cp34m-win32.whl
cryptography-1.7.1-cp34-cp34m-win32.whl
idna-2.2-py2.py3-none-any.whl
pyasn1-0.1.9-py2.py3-none-any.whl
pycparser-2.17-py2.py3-none-any.whl
python_dateutil-2.6.0-py2.py3-none-any.whl
requests-2.12.4-py2.py3-none-any.whl
setuptools-32.3.1-py2.py3-none-any.whl
six-1.10.0-py2.py3-none-any.whl
description of requirements.txt
+-------------------------------------------
|--no-index --find-links wheelhouse
|azure-nspkg==1.0.0
|azure-common==1.1.4
| :
| (omit)
| :
+-------------------------------------------

[[ Problem]]
The wheel package registered in the git repository fails in the installation on the Azure environment.

[[ Primary cause ]]
on Python 3.4 of Azure Web Apps, version of pip is 1.5.6 .
pip (1.5.6) is not able to recognize '--find-links' option which was written in the top of requirements.txt file.

[[ Workarround ]]
Upgrade pip on Azure Web Apps, using 'deploy.cmd' file.

[[ Detail ]]
At first, please refer to following URL.
https://docs.microsoft.com/en-us/azure/app-service-web/web-sites-python-configure

I got default '.deployment' file and 'deploy.cmd' file from Azure Web Apps Kudu Service.
** How to get these files **

Login to azure dashboard
Select Web Apps instance
Select [Advanced Tools] and view new page which title is 'Kudu Services'
Select [Tools] at menu bar, and choose [Download deployment script]
--> Save 'deploymentscript.zip' file into local.
Unzip deploymentscript.zip

and, please edit the vicinity of the following lines in 'deploy.cmd' .
(before)
echo Pip install requirements.
env\scripts\pip install -r requirements.txt
(after)
echo Upgrade pip package.
env\scripts\python -m pip install pip --upgrade
IF !ERRORLEVEL! NEQ 0 goto error
echo Pip install requirements.
env\scripts\pip install -r requirements.txt

(* I also upgraded setuptools )

finally, please copy '.deployment' and 'deploy.cmd' file onto top folder of GIT project,
then commit/push it.

By this workarrownd, I successfully installed wheel format packages into Azure Web Apps.
It would be greatly appreciated if this workarround is useful for you.
(Please pardon a poor English expression)

Ze Qian Zhang · Answer 14 · Mon Jun 19 2017 15:55:13 GMT+0800 (China Standard Time)

I'm closing this since no further action is required from the storage team. Please let me know if there's anything else. 😉