Allow reporting test suite that recreate driver (for each test) as a one job

[Gil Eliyahu]

Current Behavior

When I run the test suite with multiple tests that before each test I recreate the driver, and after each test, I quit the driver,
The report is reported as 2 jobs (in case there are 2 tests for example).

This is true for unittest and pytest.

Unittest

import unittest

from src.testproject.sdk.drivers import webdriver


class UntitledTestCase(unittest.TestCase):

    def setUp(self):
        self.driver = webdriver.Chrome(jobname="Job with multiple tests")

    def tearDown(self):
        self.driver.quit()

    def test_untitled_test_case1(self):
        self.driver.get("http://google.com")


    def test_untitled_test_case2(self):
        self.driver.get("http://yahoo.com")


if __name__ == "__main__":
    unittest.main()

pytest

@pytest.fixture
def driver():
    # Will be executed before each test
    driver = Chrome(projectname="python sdk project",
                    jobname="before_after_each",
                    disable_reports=False)

    yield driver

    # Will be executed after each test
    driver.quit()


def test_some_test_case_1(driver):
    driver.get("http://google.com/")


def test_some_test_case_2(driver):
    driver.get("http://yahoo.com/")


if __name__ == "__main__":
    pytest.main()

When running one of these codes, the report I get is 2 jobs - first job have only the first test and the second job have the second test.

Wanted Behavior

I want that the code will be reported as one job with 2 tests (in these cases).

@basdijkstra suggested that we will have something like driver.report().start_job("job name") and driver.report().end_job()
So the idea is to call driver.report().start_job("job name") in code that execute only at the very beginning, and call to driver.report().end_job() in code that executes only at the final end of the execution.

Advantages

Currently, without the wanted behavior, the automation developer is forced to keep using the same driver (to be reported as one job) and this is against a very common best practice approach to make the tests independent on each other by NOT using the same driver. The current behavior is encouraging to reuse the same driver and it is considered as a bad practice.

By making this suggestion, we will allow the developer to write code that follows this good practice without having it reported in the wrong way.

[Bas Dijkstra]

@mstrelex do you have an update on this one? Does the Agent / the platform already support this feature? This issue has been open and inactive for a while now... And I think that some work needs to be done on the Agent first..

[Marat Strelets]

Agent has to restart the driver but to reuse the existing development session. It's an enhancement that has not been developed yet. I will update here when it's implemented.

[Marat Strelets]

@basdijkstra Support has been added to Agent 0.64.20+
Please see testproject-io/java-opensdk#32

[Bas Dijkstra]

I've just submitted #83, which provides an implementation for this feature.

[Bas Dijkstra]

@gileli121 can you please retest this with the latest version 0.64.0?

[Tzah Mazuz]

@gileli121 - Waiting for you to confirm/verify this.

[Gil Eliyahu]

@basdijkstra
Sorry but I don't get the expected resultl.
I tested it with the code of the pytest version.
I got this:

I expected to get here 2 tests and not 1 test.
The second test should be callsed "test_some_test_case_2" with step "Navigate To http://yahoo.com/"
but it is missing. I also did not found another job with "test_some_test_case_2". I only got this job with the missing test.

I installed the SDK using command:

pip install --pre --extra-index-url https://test.pypi.org/simple --upgrade --force-reinstall testproject-python-sdk==0.64.4.dev31

I have been told that this version is the latest and should include fix for this bug

[Tzah Mazuz]

I just tested this with Python-SDK 0.65.2 and Agent 0.65.50