Giters

rcarriga / vim-ultest

The ultimate testing plugin for (Neo)Vim

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Ultest incorrectly marking failing tests as "passed"

grepinsight opened this issue · comments

commented

Describe the bug

Screen Shot 2021-11-29 at 10 05 02 PM

When using pytest.paramterize in python (which enables code reuse to test a list of input),ultest incorrectly gives "false negative", meaning it says tests are passing when one of the tests are failing. In the screenshots above, test_parametrize should be marked as failed as opposed to passed

Logs

22:07:58 | INFO | MainThread | logging.py:create_logger:101 | Logger created
22:07:58 | INFO | MainThread | __init__.py:__init__:26 | Using local threaded child watcher
22:07:58 | DEBUG | MainThread | __init__.py:__init__:44 | Handler created
22:07:58 | INFO | MainThread | __init__.py:get_attach_script:227 | Creating script to attach to process 
22:07:58 | INFO | MainThread | __init__.py:run_nearest:128 | Running nearest test in tests/test_ultest.py at line 0
22:07:58 | INFO | MainThread | __init__.py:run_nearest:133 | No tests found for tests/test_ultest.py, rerunning after processing positions
22:07:58 | INFO | MainThread | tracker.py:_init_test_file:97 | Initialising test file /user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py
22:07:58 | DEBUG | Thread-1 | __init__.py:_handle_coroutine:62 | Starting job with group update_positions
22:07:58 | INFO | Thread-1 | tracker.py:_async_update:57 | Updating positions in /user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py
22:07:58 | DEBUG | Thread-1 | file.py:parse_file_structure:25 | Converted pattern {'test': ['\\v^\\s*%(async )?def (test_\\w+)'], 'namespace': ['\\v^\\s*class (\\w+)']} to {'test': [re.compile('^\\s*(?:async )?def (test_\\w+)')], 'namespace': [re.compile('^\\s*class (\\w+)')]}
22:07:58 | DEBUG | Thread-1 | tracker.py:_async_update:84 | New test /user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py found in /user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py
22:07:58 | DEBUG | Thread-1 | tracker.py:_async_update:84 | New test test_success-4911137721759419562 found in /user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py
22:07:58 | DEBUG | Thread-1 | tracker.py:_async_update:84 | New test test_failure-4911137721759419562 found in /user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py
22:07:58 | DEBUG | Thread-1 | tracker.py:_async_update:84 | New test test_parametrize-4911137721759419562 found in /user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py
22:07:58 | DEBUG | Thread-1 | tracker.py:_remove_old_positions:129 | No tests removed
22:07:58 | DEBUG | Thread-1 | __init__.py:_handle_coroutine:62 | Starting job with group update_positions
22:07:58 | DEBUG | Thread-1 | __init__.py:_handle_coroutine:82 | Finished job with group update_positions
22:07:58 | INFO | Thread-1 | tracker.py:_async_update:57 | Updating positions in /user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py
22:07:58 | DEBUG | Thread-1 | file.py:parse_file_structure:25 | Converted pattern {'test': ['\\v^\\s*%(async )?def (test_\\w+)'], 'namespace': ['\\v^\\s*class (\\w+)']} to {'test': [re.compile('^\\s*(?:async )?def (test_\\w+)')], 'namespace': [re.compile('^\\s*class (\\w+)')]}
22:07:58 | DEBUG | Thread-1 | tracker.py:_remove_old_positions:129 | No tests removed
22:07:58 | DEBUG | Thread-1 | __init__.py:_handle_coroutine:82 | Finished job with group update_positions
22:07:58 | INFO | MainThread | __init__.py:run_nearest:128 | Running nearest test in tests/test_ultest.py at line 0
22:07:58 | DEBUG | MainThread | __init__.py:_register_started:288 | Registering /user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py as started
22:07:58 | DEBUG | MainThread | __init__.py:_register_started:288 | Registering test_success-4911137721759419562 as started
22:07:58 | DEBUG | MainThread | __init__.py:_register_started:288 | Registering test_failure-4911137721759419562 as started
22:07:58 | DEBUG | MainThread | __init__.py:_register_started:288 | Registering test_parametrize-4911137721759419562 as started
22:07:58 | DEBUG | Thread-1 | __init__.py:_handle_coroutine:62 | Starting job with group /user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py
22:07:58 | DEBUG | Thread-1 | processes.py:run:51 | Starting test process /user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py with command ['poetry', 'run', 'pytest', '-s', '-v', 'tests/test_ultest.py'], cwd = /user/home/allee/feature/allee/batch_id_proper, env = None
22:07:59 | DEBUG | Thread-1 | processes.py:run:77 | Process /user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py complete with exit code: 1
22:07:59 | INFO | Thread-1 | output.py:parse_failed:71 | Found failed test in output test_failure in namespaces None of runner python#pytest
22:07:59 | INFO | Thread-1 | output.py:parse_failed:71 | Found failed test in output test_parametrize[5] in namespaces None of runner python#pytest
22:07:59 | DEBUG | Thread-1 | __init__.py:_register_result:299 | Registering /user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py as exited with result {"id": "/user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py", "file": "/user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py", "code": 1, "output": "/tmp/ultestlct3tsj2/__user__home__allee__feature__allee__batch_id_proper__tests__test_ultest_py/__user__home__allee__feature__allee__batch_id_proper__tests__test_ultest_py_out"}
22:07:59 | DEBUG | Thread-1 | __init__.py:_register_result:299 | Registering test_success-4911137721759419562 as exited with result {"id": "test_success-4911137721759419562", "file": "/user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py", "code": 0, "output": "/tmp/ultestlct3tsj2/__user__home__allee__feature__allee__batch_id_proper__tests__test_ultest_py/__user__home__allee__feature__allee__batch_id_proper__tests__test_ultest_py_out"}
22:07:59 | DEBUG | Thread-1 | __init__.py:_register_result:299 | Registering test_failure-4911137721759419562 as exited with result {"id": "test_failure-4911137721759419562", "file": "/user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py", "code": 1, "output": "/tmp/ultestlct3tsj2/__user__home__allee__feature__allee__batch_id_proper__tests__test_ultest_py/__user__home__allee__feature__allee__batch_id_proper__tests__test_ultest_py_out"}
22:07:59 | DEBUG | Thread-1 | __init__.py:_register_result:299 | Registering test_parametrize-4911137721759419562 as exited with result {"id": "test_parametrize-4911137721759419562", "file": "/user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py", "code": 0, "output": "/tmp/ultestlct3tsj2/__user__home__allee__feature__allee__batch_id_proper__tests__test_ultest_py/__user__home__allee__feature__allee__batch_id_proper__tests__test_ultest_py_out"}
22:07:59 | DEBUG | Thread-1 | __init__.py:_handle_coroutine:82 | Finished job with group /user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py

To Reproduce
Steps to reproduce the behavior:

  • Use the code
"""Ultest bug reprex"""
import pytest


def test_success():
    """Normal case."""
    a = 3
    assert a == 3


def test_failure():
    """Normal case."""
    assert 3 == 5


@pytest.mark.parametrize("answer", [3, 5])
def test_parametrize(answer):
    """pytest parameterize"""
    assert answer == 3

Expected behavior

test_parametrize should be marked as failed as opposed to passed

Screenshots

Attached

Additional context

I love this plugin!

commented

Thanks for the great report 😁 I've accounted for the parameters in the output parsing so you should now see the correct results. Let me know if there any issues, I'm not too familiar with pytest parameters and pytest output is a pain to parse consistently.

commented

Wow. Thanks for the quick response.

However, when I tried efd96a0 with the same test input above, I still get the same output

commented

Actually that branch is unrelated, the fix was pushed to master. You can just pull latest master to try

commented

Aha I see.
I verified that f925585 returns the correct result. Thank you!!

However, the latest commit on master 7f56232 seem to break ultest altogether. Basically it turns FAILED for all three cases above.

commented

Oh that is odd. Can you create a new log please?

commented

Here's the log message

00:18:27 | INFO | MainThread | logging.py:create_logger:101 | Logger created
00:18:27 | DEBUG | MainThread | __init__.py:__init__:44 | Handler created
00:18:27 | INFO | MainThread | __init__.py:get_attach_script:227 | Creating script to attach to process 
00:18:27 | INFO | MainThread | __init__.py:run_nearest:128 | Running nearest test in tests/test_blah.py at line 0
00:18:27 | INFO | MainThread | __init__.py:run_nearest:133 | No tests found for tests/test_blah.py, rerunning after processing positions
00:18:27 | INFO | MainThread | tracker.py:_init_test_file:97 | Initialising test file /user/home/allee/feature/allee/blah/tests/test_blah.py
00:18:27 | DEBUG | MainThread | __init__.py:_handle_coroutine:49 | Starting job with group update_positions
00:18:27 | INFO | MainThread | tracker.py:_async_update:57 | Updating positions in /user/home/allee/feature/allee/blah/tests/test_blah.py
00:18:27 | DEBUG | MainThread | file.py:parse_file_structure:25 | Converted pattern {'test': ['\\v^\\s*%(async )?def (test_\\w+)'], 'namespace': ['\\v^\\s*class (\\w+)']} to {'test': [re.compile('^\\s*(?:async )?def (test_\\w+)')], 'namespace': [re.compile('^\\s*class (\\w+)')]}
00:18:27 | DEBUG | MainThread | tracker.py:_async_update:84 | New test /user/home/allee/feature/allee/blah/tests/test_blah.py found in /user/home/allee/feature/allee/blah/tests/test_blah.py
00:18:27 | DEBUG | MainThread | tracker.py:_async_update:84 | New test test_success7822109570511667405 found in /user/home/allee/feature/allee/blah/tests/test_blah.py
00:18:27 | DEBUG | MainThread | tracker.py:_async_update:84 | New test test_failure7822109570511667405 found in /user/home/allee/feature/allee/blah/tests/test_blah.py
00:18:27 | DEBUG | MainThread | tracker.py:_async_update:84 | New test test_parametrize7822109570511667405 found in /user/home/allee/feature/allee/blah/tests/test_blah.py
00:18:27 | DEBUG | MainThread | tracker.py:_remove_old_positions:129 | No tests removed
00:18:27 | DEBUG | MainThread | __init__.py:_handle_coroutine:69 | Finished job with group update_positions
00:18:27 | DEBUG | MainThread | __init__.py:_handle_coroutine:49 | Starting job with group update_positions
00:18:27 | INFO | MainThread | tracker.py:_async_update:57 | Updating positions in /user/home/allee/feature/allee/blah/tests/test_blah.py
00:18:27 | DEBUG | MainThread | file.py:parse_file_structure:25 | Converted pattern {'test': ['\\v^\\s*%(async )?def (test_\\w+)'], 'namespace': ['\\v^\\s*class (\\w+)']} to {'test': [re.compile('^\\s*(?:async )?def (test_\\w+)')], 'namespace': [re.compile('^\\s*class (\\w+)')]}
00:18:27 | DEBUG | MainThread | tracker.py:_remove_old_positions:129 | No tests removed
00:18:27 | INFO | MainThread | __init__.py:run_nearest:128 | Running nearest test in tests/test_blah.py at line 0
00:18:27 | DEBUG | MainThread | __init__.py:_handle_coroutine:69 | Finished job with group update_positions
00:18:27 | DEBUG | MainThread | __init__.py:_register_started:288 | Registering /user/home/allee/feature/allee/blah/tests/test_blah.py as started
00:18:27 | DEBUG | MainThread | __init__.py:_register_started:288 | Registering test_success7822109570511667405 as started
00:18:27 | DEBUG | MainThread | __init__.py:_register_started:288 | Registering test_failure7822109570511667405 as started
00:18:27 | DEBUG | MainThread | __init__.py:_register_started:288 | Registering test_parametrize7822109570511667405 as started
00:18:27 | DEBUG | MainThread | __init__.py:_handle_coroutine:49 | Starting job with group /user/home/allee/feature/allee/blah/tests/test_blah.py
00:18:27 | DEBUG | MainThread | processes.py:run:51 | Starting test process /user/home/allee/feature/allee/blah/tests/test_blah.py with command ['poetry', 'run', 'pytest', '-s', '-v', 'tests/test_blah.py'], cwd = /user/home/allee/feature/allee/blah, env = None
00:18:27 | WARNING | MainThread | processes.py:run:71 | An exception was thrown when starting process /user/home/allee/feature/allee/blah/tests/test_blah.py with command: ['poetry', 'run', 'pytest', '-s', '-v', 'tests/test_blah.py']
Traceback (most recent call last):
  File "/ghdevhome/home/allee/.local/share/nvim/site/pack/packer/opt/vim-ultest/rplugin/python3/ultest/handler/runner/processes.py", line 64, in run
    close_fds=True,
  File "/user/home/miniconda3/envs/myenv/lib/python3.7/asyncio/subprocess.py", line 217, in create_subprocess_exec
    stderr=stderr, **kwds)
  File "/user/home/miniconda3/envs/myenv/lib/python3.7/asyncio/base_events.py", line 1533, in subprocess_exec
    bufsize, **kwargs)
  File "/user/home/miniconda3/envs/myenv/lib/python3.7/asyncio/unix_events.py", line 193, in _make_subprocess_transport
    self._child_watcher_callback, transp)
  File "/user/home/miniconda3/envs/myenv/lib/python3.7/asyncio/unix_events.py", line 924, in add_child_handler
    "Cannot add child handler, "
RuntimeError: Cannot add child handler, the child watcher does not have a loop attached
00:18:27 | DEBUG | MainThread | processes.py:run:77 | Process /user/home/allee/feature/allee/blah/tests/test_blah.py complete with exit code: 1
00:18:27 | DEBUG | MainThread | __init__.py:_register_result:299 | Registering /user/home/allee/feature/allee/blah/tests/test_blah.py as exited with result {"id": "/user/home/allee/feature/allee/blah/tests/test_blah.py", "file": "/user/home/allee/feature/allee/blah/tests/test_blah.py", "code": 1, "output": "/tmp/ultestb370zjtv/__user__home__allee__feature__allee__blah__tests__test_blah_py/__user__home__allee__feature__allee__blah__tests__test_blah_py_out"}
00:18:27 | DEBUG | MainThread | __init__.py:_register_result:299 | Registering test_success7822109570511667405 as exited with result {"id": "test_success7822109570511667405", "file": "/user/home/allee/feature/allee/blah/tests/test_blah.py", "code": 1, "output": "/tmp/ultestb370zjtv/__user__home__allee__feature__allee__blah__tests__test_blah_py/__user__home__allee__feature__allee__blah__tests__test_blah_py_out"}
00:18:27 | DEBUG | MainThread | __init__.py:_register_result:299 | Registering test_failure7822109570511667405 as exited with result {"id": "test_failure7822109570511667405", "file": "/user/home/allee/feature/allee/blah/tests/test_blah.py", "code": 1, "output": "/tmp/ultestb370zjtv/__user__home__allee__feature__allee__blah__tests__test_blah_py/__user__home__allee__feature__allee__blah__tests__test_blah_py_out"}
00:18:27 | DEBUG | MainThread | __init__.py:_register_result:299 | Registering test_parametrize7822109570511667405 as exited with result {"id": "test_parametrize7822109570511667405", "file": "/user/home/allee/feature/allee/blah/tests/test_blah.py", "code": 1, "output": "/tmp/ultestb370zjtv/__user__home__allee__feature__allee__blah__tests__test_blah_py/__user__home__allee__feature__allee__blah__tests__test_blah_py_out"}
00:18:27 | DEBUG | MainThread | __init__.py:_handle_coroutine:69 | Finished job with group /user/home/allee/feature/allee/blah/tests/test_blah.py
commented

By the way I am using Python 3.7.3. It seems that the failure is happening in asyncio module in python

commented

Ah it's due to a python bug https://bugs.python.org/issue35621 which I had removed the workaround for due to no longer using a separate loop. Thought I had tested but must have misconfigured something. Should be fine in latest commit, sorry for the trouble! 3.7 is a bit troublesome 😅

commented

Thank you! The issue is now fixed with the recent commit.