facebookresearch / labgraph

🐛 Bug

When running the test suite with python -m pytest --pyargs labgraph, pytest crashes at test_process_manager.py. Strangely, the tests complete successfully when test_process_manager.py is run independently.

To Reproduce

Steps to reproduce the behavior:

Run the full test suite: python -m pytest --pyargs labgraph. Here's the output on my system:

====================================================== test session starts ======================================================
platform linux -- Python 3.6.8, pytest-3.10.1, py-1.10.0, pluggy-0.13.1
rootdir: /home/snel, inifile:
plugins: mock-2.0.0
collected 115 items                                                                                                             

miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/_cthulhu/tests/test_clock.py ...                            [  2%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/_cthulhu/tests/test_cthulhu.py ..                           [  4%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/events/tests/test_event_generator.py ............           [ 14%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/events/tests/test_event_generator_node.py .....             [ 19%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/graphs/tests/test_config.py ...                             [ 21%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/graphs/tests/test_graph.py ..                               [ 23%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/graphs/tests/test_group.py ................                 [ 37%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/graphs/tests/test_harness.py .....                          [ 41%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/graphs/tests/test_module.py ..                              [ 43%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/graphs/tests/test_node.py ....                              [ 46%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/graphs/tests/test_publisher.py ..                           [ 48%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/graphs/tests/test_subscriber.py ..                          [ 50%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/loggers/tests/test_loggers.py .                             [ 51%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/messages/tests/test_message.py ....................         [ 68%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_aligner.py ss                            [ 70%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_cpp.py .                                 [ 71%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_exception.py .........                   [ 79%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_launch.py .                              [ 80%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_process_manager.py ...........Aborted (core dumped)

Run the tests in labgraph/runners/tests with verbose output python -m pytest --pyargs -v labgraph.runners.tests. Here's the output on my system:

====================================================== test session starts ======================================================
platform linux -- Python 3.6.8, pytest-3.10.1, py-1.10.0, pluggy-0.13.1 -- /home/snel/miniconda3/envs/labgraph/bin/python
cachedir: .pytest_cache
rootdir: /home/snel, inifile:
plugins: mock-2.0.0
collected 31 items                                                                                                              

miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_aligner.py::test_slow_align_interval SKIPPED [  3%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_aligner.py::test_align_two_streams SKIPPED [  6%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_cpp.py::test_cpp_graph PASSED            [  9%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_exception.py::test_local_throw[MainThrowerNode] PASSED [ 12%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_exception.py::test_local_throw[BackgroundThrowerNode] PASSED [ 16%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_exception.py::test_local_throw[PublisherThrowerNode] PASSED [ 19%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_exception.py::test_local_throw[PublisherSubscriberThrowerNode] PASSED [ 22%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_exception.py::test_parallel_throw[PublisherThrowerGraph] PASSED [ 25%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_exception.py::test_parallel_throw[SubscriberThrowerGraph] PASSED [ 29%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_exception.py::test_parallel_throw[MainThrowerGraph] PASSED [ 32%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_exception.py::test_parallel_throw[BackgroundThrowerGraph] PASSED [ 35%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_exception.py::test_logger_throw PASSED   [ 38%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_launch.py::test_launch PASSED            [ 41%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_process_manager.py::test_normal PASSED   [ 45%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_process_manager.py::test_crash[ProcessPhase.STARTING] PASSED [ 48%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_process_manager.py::test_crash[ProcessPhase.READY] PASSED [ 51%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_process_manager.py::test_crash[ProcessPhase.RUNNING] PASSED [ 54%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_process_manager.py::test_crash[ProcessPhase.STOPPING] PASSED [ 58%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_process_manager.py::test_exception[ProcessPhase.STARTING] PASSED [ 61%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_process_manager.py::test_exception[ProcessPhase.READY] PASSED [ 64%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_process_manager.py::test_exception[ProcessPhase.RUNNING] PASSED [ 67%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_process_manager.py::test_exception[ProcessPhase.STOPPING] PASSED [ 70%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_process_manager.py::test_hang[ProcessPhase.STARTING] PASSED [ 74%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_process_manager.py::test_hang[ProcessPhase.READY] PASSED [ 77%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_process_manager.py::test_hang[ProcessPhase.RUNNING] Aborted (core dumped)

Run only test_process_manager.py: python -m pytest --pyargs -v labgraph.runners.tests.test_process_manager. Here's the output on my system:

====================================================== test session starts ======================================================
platform linux -- Python 3.6.8, pytest-3.10.1, py-1.10.0, pluggy-0.13.1 -- /home/snel/miniconda3/envs/labgraph/bin/python
cachedir: .pytest_cache
rootdir: /home/snel, inifile:
plugins: mock-2.0.0
collected 13 items                                                                                                              

miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_process_manager.py::test_normal PASSED   [  7%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_process_manager.py::test_crash[ProcessPhase.STARTING] PASSED [ 15%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_process_manager.py::test_crash[ProcessPhase.READY] PASSED [ 23%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_process_manager.py::test_crash[ProcessPhase.RUNNING] PASSED [ 30%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_process_manager.py::test_crash[ProcessPhase.STOPPING] PASSED [ 38%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_process_manager.py::test_exception[ProcessPhase.STARTING] PASSED [ 46%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_process_manager.py::test_exception[ProcessPhase.READY] PASSED [ 53%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_process_manager.py::test_exception[ProcessPhase.RUNNING] PASSED [ 61%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_process_manager.py::test_exception[ProcessPhase.STOPPING] PASSED [ 69%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_process_manager.py::test_hang[ProcessPhase.STARTING] PASSED [ 76%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_process_manager.py::test_hang[ProcessPhase.READY] PASSED [ 84%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_process_manager.py::test_hang[ProcessPhase.RUNNING] PASSED [ 92%]
miniconda3/envs/labgraph/lib/python3.6/site-packages/labgraph/runners/tests/test_process_manager.py::test_hang[ProcessPhase.STOPPING] PASSED [100%]

================================================== 13 passed in 37.99 seconds ===================================================

In steps 1 and 2, pytest crashes while running test_process_manager.py. In step 3, we run only the tests in test_process_manager.py, and no crash occurs.

The verbose output in step 2 shows that the crash happens in the test_hang function of test_process_manager.py when the hang_phase is set to ProcessPhase.RUNNING. Here's a link to that function:

labgraph/labgraph/runners/tests/test_process_manager.py

Lines 246 to 287 in e9e98ec

    
           @pytest.mark.parametrize( 
        
               "hang_phase", 
        
               ( 
        
                   ProcessPhase.STARTING, 
        
                   ProcessPhase.READY, 
        
                   ProcessPhase.RUNNING, 
        
                   ProcessPhase.STOPPING, 
        
               ), 
        
           ) 
        
           @local_test 
        
           def test_hang(hang_phase: ProcessPhase) -> None: 
        
               """ 
        
               Tests that we can run multiple processes where one of them hangs. 
        
               """ 
        
               manager = ProcessManager( 
        
                   processes=( 
        
                       ProcessInfo( 
        
                           module=__name__, 
        
                           name="proc1", 
        
                           args=( 
        
                               "--manager-name", 
        
                               "test_manager", 
        
                               "--shutdown", 
        
                               "HANG", 
        
                               "--last-phase", 
        
                               hang_phase.name, 
        
                           ), 
        
                       ), 
        
                       ProcessInfo( 
        
                           module=__name__, 
        
                           name="proc2", 
        
                           args=("--manager-name", "test_manager", "--shutdown", "NORMAL"), 
        
                       ), 
        
                   ), 
        
                   name="test_manager", 
        
                   startup_period=TEST_STARTUP_PERIOD, 
        
                   shutdown_period=TEST_SHUTDOWN_PERIOD, 
        
               ) 
        
               with pytest.raises(ProcessManagerException) as ex: 
        
                   manager.run() 
        
               assert ex.value.failures == {"proc1": ProcessFailureType.HANG, "proc2": None}

Does anyone know what would cause this crash?

Expected behavior

The full test suite should run to completion. Also, the results of running one test script should match the results of running that same script within the full test suite.

Environment

LabGraph Version (e.g., 1.0): 1.0.0 (installed via pip)
OS (e.g., Linux/Mac/Windows): CentOS 8
Python version: 3.6.8

Hi yahiaali, the recent update could help with your reported issue. If this issue is still a blocker for you to use LabGraph, feel free to let us know. I'm going to close this issue for now, feel free to reopen or create a new issue. Thank you for being our top issue reporter as of this moment.

Thanks for the update @jfResearchEng! The issue is resolved now. I was able to run the test suite using the new test_script.sh, and all tests are now passing. The commands I used are below:

export LC_ALL=C.UTF-8
export LANG=en_US.utf-8
sh ./test_script.sh

	@pytest.mark.parametrize(
	"hang_phase",
	(
	ProcessPhase.STARTING,
	ProcessPhase.READY,
	ProcessPhase.RUNNING,
	ProcessPhase.STOPPING,
	),
	)
	@local_test
	def test_hang(hang_phase: ProcessPhase) -> None:
	"""
	Tests that we can run multiple processes where one of them hangs.
	"""
	manager = ProcessManager(
	processes=(
	ProcessInfo(
	module=__name__,
	name="proc1",
	args=(
	"--manager-name",
	"test_manager",
	"--shutdown",
	"HANG",
	"--last-phase",
	hang_phase.name,
	),
	),
	ProcessInfo(
	module=__name__,
	name="proc2",
	args=("--manager-name", "test_manager", "--shutdown", "NORMAL"),
	),
	),
	name="test_manager",
	startup_period=TEST_STARTUP_PERIOD,
	shutdown_period=TEST_SHUTDOWN_PERIOD,
	)

	with pytest.raises(ProcessManagerException) as ex:
	manager.run()
	assert ex.value.failures == {"proc1": ProcessFailureType.HANG, "proc2": None}

Test suite crashes while running test_process_manager.py

🐛 Bug

To Reproduce

Expected behavior

Environment