upload to untrusted hosts with --trusted option
glenfant opened this issue · comments
Hi,
Twine denies uploading to untrusted https PyPI clones. This triggers an SSL error caused by an unknown CA certificate. I need the ability to upload packages to untrusted private PyPI servers since
Backgroung : I'm in process to add PyPI and NPM support to a Nexus server that's published in production with regular CA signed certificates. But can't obtain such certificates for the Nexus test site.
I made a small quick'n'dirty patch setting verify=False option when posting a tarball or wheel file here.
https://github.com/pypa/twine/blob/master/twine/repository.py#L152
I know that the requests lib issues a warning in such situations but I don't care.
Thanks
Would you accept a PR with a new option "trusted-host" option that does the same job as for pip ?
$ pip help install
...
--trusted-host <hostname> Mark this host as trusted, even though it does
not have valid or any HTTPS.
...
That seems like a good way to name it and go about it, @glenfant .
Hello there,
Was this ever implemented?
We need this badly as setup.py upload command does not seem to be working anymore with Python 3.7.x versions. when uploading to private PyPi repositories(gives 400 Bad Request)
Sorry I didn't comment earlier, but I wanted to say that I think I'm -1 on making this change.
I'm a bit concerned that if we add this flag, it will proliferate through comments, stack overflow answers, etc as an "easy" way to fix SSL errors that should actually be fixed by producing and obtaining valid certificates for third-party indices. Basically, this shouldn't be necessary and feels like an easy way out.
I'm especially concerned about the possibility of this being used with --trusted-host pypi.org which should basically never happen. At the very least, I'd like to see this being implemented in a way that prevents this flag from ever being used with that host.
(cc @ewdurbin for thoughts here as well)
I agree, it's preferable and technically easy to add a real CA approved certificate to the private registry server (Nexus) of my customer's company, but it's a political nightmare. I can understand your opinion, but the only workaround I found is to add to every workstation, CI container (etc.) a script that add the private CA cert to the standard keyring :/
@di If I understand correctly, could you accept a change that ignores --trusted on *.pypi.org target registry?
Sorry I didn't comment earlier, but I wanted to say that I think I'm -1 on making this change.
I'm a bit concerned that if we add this flag, it will proliferate through comments, stack overflow answers, etc as an "easy" way to fix SSL errors that should actually be fixed by producing and obtaining valid certificates for third-party indices. Basically, this shouldn't be necessary and feels like an easy way out.
I'm especially concerned about the possibility of this being used with
--trusted-host pypi.orgwhich should basically never happen. At the very least, I'd like to see this being implemented in a way that prevents this flag from ever being used with that host.(cc @ewdurbin for thoughts here as well)
It's not always possible to work the "right" way, that's why pip, conda install and npm gives the option to add trusted-host\verify-ssl flags. Not as the recommended way, but as a possiblity for users. The default should obviously be to verify whenever its a possibility.
In my case, I just can't demand from the Artifactory repository providers to add a real CA approved certificate and I have no choise but use verify=False when uploading and downloading packages from there. and there are no alternatives in the lan...
Is there a better option here? Is it possible without disabling SSL to still trust/allow it? That is, is it possible to just download the certificate and allow it? Something like:
import ssl
import sys
import urllib.parse
url, = sys.argv[1:]
if '/' not in url:
url = '//' + url
parsed = urllib.parse.urlparse(url)
with open('cert.pem', 'w') as f:
f.write(ssl.get_server_certificate((parsed.hostname, parsed.port or 443)))
to download the certificate and then
REQUESTS_CA_BUNDLE=cert.pem twine upload ...
According to this answer, I'd expect that to work, but it didn't work for me when I tested against self-signed.badssl.com.
@jaraco this is what --cert is supposed to provide. If you have the certificates as a PEM file which can be generated trivially by searching StackOverflow, then they should be able to pass that to --cert iirc instead of having --trusted. That solves only the one case that @glenfant described though.
There's still the issue of "what about an implementation without TLS?" I think the answer there is "maybe that works today already." I don't think we validate for https:// as a scheme or change that forcefully. If we do, then we'd need an option along the lines of --yes-i-know-i-should-use-tls but that wouldn't be fixed by --trusted because we'd otherwise still try connecting over :443 and the server may not even listen on that port.
I can say with confidence that upload requests over http work, because twine now has integration tests against a local devpi instance over plaintext HTTP.
+1 for the issue, I'd really like to have this option in
The solution to this problem is threefold:
- insecure requests over plain HTTP are allowed
- where SSL is used with an otherwise untrusted certificate, the
--certparameter should be used to pass the trusted certificate - if the above solutions aren't available, implement a hack (such as this) at your own risk to disable certificate checking.
This last option is intentionally ugly and inconvenient to limit the proliferation of that discouraged approach.
If none of these approaches are suitable, please feel free to elaborate on why.
This issue really shouldn't be closed . You are not special for not allowing "insecure" requests on secured isolated networks
pip, conda install and npm gives the option to add trusted-host\verify-ssl flags
[twine is] not special for not allowing "insecure" requests
Maybe twine is special, because it's responsible for publishing whereas the other tools mentioned are primarily responsible for consuming. There's a higher standard of security demanded from tools that publish content.
Does npm also allow the same flags for publishing? Are there other examples of tools for publishing work that allow bypassing critical security checks.
I don't think anyone has yet proposed a technique that would allow disabling of the security checks when they're not needed but would disallow the same where the checks should be enforced. As a general rule, if the server is only exposing HTTPS, then it's the client's responsibility to validate the authenticity of the server. Twine would need a way to prevent casual bypass of the check. How can twine discriminate between legitimate bypass and casual bypass?
[supplying the correct cert] is what
--certis supposed to provide. If you have the certificates as a PEM file... then they should be able to pass that to--cert.
I tried this approach this time I was able to make it work using the script above as get-cert.py.
draft @ py -m get-cert https://self-signed.badssl.com
draft @ http -q --download https://files.pythonhosted.org/packages/c9/c9/6122a974f5b611b16396c918722ea75945db7dcd3069c477a7c608a405a0/example-0.1.
0.tar.gz
draft @ twine upload --password foo --repository-url https://self-signed.badssl.com/upload/ example-0.1.0.tar.gz
Uploading distributions to https://self-signed.badssl.com/upload/
Uploading example-0.1.0.tar.gz
WARNING Retrying (Retry(total=9, connect=5, read=None, redirect=None, status=None)) after connection broken by
'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate
(_ssl.c:1002)'))': /upload/
...
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='self-signed.badssl.com', port=443): Max retries exceeded with url: /upload/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1002)')))
draft @ twine upload --password foo --cert cert.pem --repository-url https://self-signed.badssl.com/upload/ example-0.1.0.tar.gz
Uploading distributions to https://self-signed.badssl.com/upload/
Uploading example-0.1.0.tar.gz
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.6/3.6 kB • 00:00 • ?
WARNING Error during upload. Retry with the --verbose option for more details.
ERROR HTTPError: 404 Not Found from https://self-signed.badssl.com/upload/
Not Found
In the first attempt, I don't pass the certificate and the request fails with an SSL error. In the second attempt, I pass the downloaded certificate and pass it to twine, authorizing that specific server (or the man in the middle attacker), and it fails with a 404 error, presumably after the TLS handshake has succeeded.
That technique should work for any isolated networks. I'll amend the recommendations.
The recommendations to address this problem:
- switch to plain HTTP
- where SSL is used with an otherwise untrusted certificate, pass a trusted certificate using the
--certparameter. The trusted certificate may be acquired from the server.
If neither of these approaches are suitable, please feel free to elaborate on why.