1.6.0: pytest is failing in `pooch/tests/test_core.py::test_check_availability_on_ftp` unit
kloczek opened this issue · comments
I'm packaging your module as an rpm package so I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.
python3 -sBm build -w --no-isolation- because I'm calling
buildwith--no-isolationI'm using during all processes only locally installed modules - install .whl file in </install/prefix>
- run pytest with PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>
Here is pytest output:
+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-pooch-1.6.0-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-pooch-1.6.0-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.16, pytest-7.2.0, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/pooch-1.6.0, configfile: pyproject.toml
plugins: anyio-3.6.2, localftpserver-1.1.3
collected 126 items
pooch/tests/test_core.py .........................F...... [ 25%]
pooch/tests/test_downloaders.py sss........ss....ssss. [ 42%]
pooch/tests/test_hashes.py ......sssss...sssss....... [ 63%]
pooch/tests/test_integration.py . [ 64%]
pooch/tests/test_processors.py ................................ [ 89%]
pooch/tests/test_utils.py ............ [ 99%]
pooch/tests/test_version.py . [100%]
================================================================================= FAILURES =================================================================================
______________________________________________________________________ test_check_availability_on_ftp ______________________________________________________________________
@pytest.mark.network
def test_check_availability_on_ftp():
"Should correctly check availability of existing and non existing files"
# Check available remote file on FTP server
pup = Pooch(
path=DATA_DIR,
base_url="ftp://data-out.unavco.org/pub/products/velocity/rel_201712/",
registry={
"pbo.final_igs08.20171202.vel": "md5:0b75d4049dedd0e179615f4b5e956156",
"doesnot_exist.zip": "jdjdjdjdflld",
},
)
> assert pup.is_available("pbo.final_igs08.20171202.vel")
pooch/tests/test_core.py:488:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pooch/core.py:699: in is_available
ftp.connect(host=parsed_url["netloc"])
/usr/lib64/python3.8/ftplib.py:154: in connect
self.sock = socket.create_connection((self.host, self.port), self.timeout,
/usr/lib64/python3.8/socket.py:808: in create_connection
raise err
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
address = ('data-out.unavco.org', 21), timeout = <object object at 0x7ff5548303b0>, source_address = None
def create_connection(address, timeout=_GLOBAL_DEFAULT_TIMEOUT,
source_address=None):
"""Connect to *address* and return the socket object.
Convenience function. Connect to *address* (a 2-tuple ``(host,
port)``) and return the socket object. Passing the optional
*timeout* parameter will set the timeout on the socket instance
before attempting to connect. If no *timeout* is supplied, the
global default timeout setting returned by :func:`getdefaulttimeout`
is used. If *source_address* is set it must be a tuple of (host, port)
for the socket to bind as a source address before making the connection.
A host of '' or port 0 tells the OS to use the default.
"""
host, port = address
err = None
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
af, socktype, proto, canonname, sa = res
sock = None
try:
sock = socket(af, socktype, proto)
if timeout is not _GLOBAL_DEFAULT_TIMEOUT:
sock.settimeout(timeout)
if source_address:
sock.bind(source_address)
> sock.connect(sa)
E ConnectionRefusedError: [Errno 111] Connection refused
/usr/lib64/python3.8/socket.py:796: ConnectionRefusedError
========================================================================= short test summary info ==========================================================================
SKIPPED [2] pooch/tests/test_downloaders.py:51: requires tqdm
SKIPPED [1] pooch/tests/test_downloaders.py:65: requires paramiko
SKIPPED [1] pooch/tests/test_downloaders.py:137: requires paramiko to run SFTP
SKIPPED [1] pooch/tests/test_downloaders.py:149: requires paramiko to run SFTP
SKIPPED [2] pooch/tests/test_downloaders.py:179: requires tqdm
SKIPPED [1] pooch/tests/test_downloaders.py:208: requires tqdm
SKIPPED [1] pooch/tests/test_downloaders.py:233: requires paramiko
SKIPPED [5] pooch/tests/test_hashes.py:126: requires xxhash
SKIPPED [5] pooch/tests/test_hashes.py:144: requires xxhash
FAILED pooch/tests/test_core.py::test_check_availability_on_ftp - ConnectionRefusedError: [Errno 111] Connection refused
================================================================ 1 failed, 106 passed, 19 skipped in 43.10s ================================================================
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:79: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-e59b8484-0a67-4761-b803-e36562acd70d
<class 'OSError'>: [Errno 39] Directory not empty: '/tmp/pytest-of-tkloczko/garbage-e59b8484-0a67-4761-b803-e36562acd70d'
warnings.warn(
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:79: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-de25aea0-37a6-45b7-b898-161b93a777ba/test_rmtree_errorhandler_rerai0
<class 'OSError'>: [Errno 39] Directory not empty: 'test_rmtree_errorhandler_rerai0'
warnings.warn(
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:79: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-de25aea0-37a6-45b7-b898-161b93a777ba/test_rmtree_errorhandler_reado0
<class 'OSError'>: [Errno 39] Directory not empty: 'test_rmtree_errorhandler_reado0'
warnings.warn(
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:79: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-de25aea0-37a6-45b7-b898-161b93a777ba/test_safe_delete_no_perms0
<class 'OSError'>: [Errno 39] Directory not empty: 'test_safe_delete_no_perms0'
warnings.warn(
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:79: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-de25aea0-37a6-45b7-b898-161b93a777ba/test_safe_set_no_perms0
<class 'OSError'>: [Errno 39] Directory not empty: 'test_safe_set_no_perms0'
warnings.warn(
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:79: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-de25aea0-37a6-45b7-b898-161b93a777ba/test_safe_get_no_perms0
<class 'OSError'>: [Errno 39] Directory not empty: 'test_safe_get_no_perms0'
warnings.warn(
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:79: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-de25aea0-37a6-45b7-b898-161b93a777ba
<class 'OSError'>: [Errno 39] Directory not empty: '/tmp/pytest-of-tkloczko/garbage-de25aea0-37a6-45b7-b898-161b93a777ba'
warnings.warn(
Here is list of installed modules in build env
Package Version
----------------------------- -----------------
alabaster 0.7.12
anyio 3.6.2
appdirs 1.4.4
async-generator 1.10
attrs 22.1.0
Babel 2.11.0
beautifulsoup4 4.11.1
Brlapi 0.8.3
build 0.9.0
certifi 2022.9.24
cffi 1.15.1
charset-normalizer 3.0.1
click 8.1.3
cryptography 38.0.4
cssselect 1.1.0
distro 1.8.0
dnspython 2.2.1
docutils 0.19
editables 0.3
exceptiongroup 1.0.0
extras 1.0.0
fixtures 4.0.0
gpg 1.17.1-unknown
h11 0.13.0
hatchling 1.11.1
hpack 4.0.0
hyperframe 6.0.1
idna 3.4
imagesize 1.4.1
importlib-metadata 5.1.0
iniconfig 1.1.1
Jinja2 3.1.2
libcomps 0.1.19
louis 3.24.0
lxml 4.9.1
MarkupSafe 2.1.1
mccabe 0.7.0
mypy-extensions 0.4.3
numpy 1.23.1
outcome 1.1.0
packaging 21.3
pathspec 0.10.2
pbr 5.9.0
pep517 0.13.0
pip 22.3.1
platformdirs 2.5.2
pluggy 1.0.0
ply 3.11
pycodestyle 2.9.1
pycparser 2.21
pydata-sphinx-theme 0.12.0
pyflakes 2.5.0
pyftpdlib 1.5.6
Pygments 2.13.0
PyGObject 3.42.2
pyOpenSSL 22.0.0
pyparsing 3.0.9
pytest 7.2.0
pytest-localftpserver 1.1.3
python-dateutil 2.8.2
pytz 2022.4
PyYAML 6.0
requests 2.28.1
rpm 4.17.0
scour 0.38.2
setuptools 65.6.3
setuptools-scm 7.0.5
six 1.16.0
sniffio 1.2.0
snowballstemmer 2.2.0
sortedcontainers 2.4.0
soupsieve 2.3.2.post1
Sphinx 5.3.0
sphinx-book-theme 0.3.3
sphinx-panels 0.6.0
sphinxcontrib-applehelp 1.0.2.dev20221204
sphinxcontrib-devhelp 1.0.2.dev20221204
sphinxcontrib-htmlhelp 2.0.0
sphinxcontrib-jsmath 1.0.1.dev20221204
sphinxcontrib-qthelp 1.0.3.dev20221204
sphinxcontrib-serializinghtml 1.1.5
testtools 2.5.0
tomli 2.0.1
trio 0.21.0+dev
typing_extensions 4.4.0
urllib3 1.26.12
wheel 0.38.4
zipp 3.11.0
Hi @kloczek. Thanks for opening this issue and sorry for the delayed reply.
The test function you are trying to run will fail because the UNAVCO ftp server was decommissioned. We already merged #322 to solve it.
I imagine that you are packaging the latest release of Pooch and this is blocking you from doing it. If you are in a rush, I would recommend ignoring this error: it's not due to a bug in our code as pointed out in that PR. If you want to have full tests pass, I think we are about to release a Pooch version soon, considering that we fixed a few bugs recently.
Units which requires public network acces should be marked by @pytest.mark.network mark to allow skipp such units on systems without access to the public network by running pytest with -m "not network" like in many other modules.
Oher thing is that probably in tis case better would be use https://github.com/oz123/pytest-localftpserver/
Thanks for the tip. Pooch currently uses the network pytest mark on some test functions, but I suspect some test functions that require connectivity aren't decorated yet. Would you like to open an issue describing the problem and providing possible solutions to it?
BTW, Pooch v.1.7.0 have been just released, which includes the fixes introduced in #322. I think now you shouldn't have any problem running the test locally (while connected to internet).
I've opened #367 to double check that all network-accessing tests are properly marked. That seems to be the real issue here so closing this one.