Webdrivers crash in multi process parallel test runs
derdualist opened this issue · comments
at first I opened this issue with the Selenium project, but it seems to be related with the webdriver-manager module.
SeleniumHQ/selenium#12751
As far as I can tell there are 2 problems here, that come together:
- The driver cache does not find an existing driver binary, so the manager always downloads it.
- There is no sync logic when multiple processes download the same driver binary, resulting in all but one process that fails with a cryptic OS error.
Maybe after confirmation, this can be split into a bug and a feature ticket.
I created a gist, that can reproduce the issue:
output:
$ python parallel_selenium_tests.py -n2 -m webdriver-manager -b chrome
[WDM LOG] ====== WebDriver manager ======
[WDM LOG] ====== WebDriver manager ======
[WDM LOG] Get LATEST chromedriver version for google-chrome
[WDM LOG] Get LATEST chromedriver version for google-chrome
[WDM LOG] Get LATEST chromedriver version for google-chrome
[WDM LOG] There is no [mac64] chromedriver "117.0.5938.92" for browser google-chrome "117.0.5938.92" in cache
[WDM LOG] Get LATEST chromedriver version for google-chrome
[WDM LOG] There is no [mac64] chromedriver "117.0.5938.92" for browser google-chrome "117.0.5938.92" in cache
[WDM LOG] Get LATEST chromedriver version for google-chrome
[WDM LOG] Get LATEST chromedriver version for google-chrome
[WDM LOG] WebDriver version 117.0.5938.92 selected
[WDM LOG] Modern chrome version https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/117.0.5938.92/mac-arm64/chromedriver-mac-arm64.zip
[WDM LOG] About to download new driver from https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/117.0.5938.92/mac-arm64/chromedriver-mac-arm64.zip
[WDM LOG] WebDriver version 117.0.5938.92 selected
[WDM LOG] Modern chrome version https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/117.0.5938.92/mac-arm64/chromedriver-mac-arm64.zip
[WDM LOG] About to download new driver from https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/117.0.5938.92/mac-arm64/chromedriver-mac-arm64.zip
[WDM LOG] Driver downloading response is 200
[WDM LOG] Driver downloading response is 200
[WDM LOG] Get LATEST chromedriver version for google-chrome
[WDM LOG] Driver has been saved in cache [/Users/tobias.lehmann/.wdm/drivers/chromedriver/mac64/117.0.5938.92]
[WDM LOG] Get LATEST chromedriver version for google-chrome
[WDM LOG] Driver has been saved in cache [/Users/tobias.lehmann/.wdm/drivers/chromedriver/mac64/117.0.5938.92]
Process Process-2:
Traceback (most recent call last):
File "/opt/homebrew/Cellar/python@3.10/3.10.12_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/opt/homebrew/Cellar/python@3.10/3.10.12_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/Users/tobias.lehmann/projects/ta/driver_management/parallel_driver.py", line 88, in test
driver = create_driver(browser, executable_path, options)
File "/Users/tobias.lehmann/projects/ta/driver_management/parallel_driver.py", line 75, in create_driver
return {
File "/Users/tobias.lehmann/projects/ta/driver_management/parallel_driver.py", line 76, in <lambda>
'chrome': lambda: Chrome(service=ChromeService(executable_path=executable_path), options=options),
File "/Users/tobias.lehmann/Library/Python/3.10/lib/python/site-packages/selenium/webdriver/chrome/webdriver.py", line 45, in __init__
super().__init__(
File "/Users/tobias.lehmann/Library/Python/3.10/lib/python/site-packages/selenium/webdriver/chromium/webdriver.py", line 53, in __init__
self.service.start()
File "/Users/tobias.lehmann/Library/Python/3.10/lib/python/site-packages/selenium/webdriver/common/service.py", line 109, in start
self.assert_process_still_running()
File "/Users/tobias.lehmann/Library/Python/3.10/lib/python/site-packages/selenium/webdriver/common/service.py", line 122, in assert_process_still_running
raise WebDriverException(f"Service {self._path} unexpectedly exited. Status code was: {return_code}")
selenium.common.exceptions.WebDriverException: Message: Service /Users/tobias.lehmann/.wdm/drivers/chromedriver/mac64/117.0.5938.92/chromedriver-mac-arm64/chromedriver unexpectedly exited. Status code was: -9
The Internet
Broken Images
The output The Internet Broken Images comes from the test case in the one succeeding process.
Used versions
- webdriver-manager 4.0.0
- selenium 4.12.0
Content of cache
The correct driver key is inside the cache:
{
...
"mac64_chromedriver_117.0.5938.92_for_117.0.5938.92": {
"timestamp": "29/09/2023",
"binary_path": "/Users/USER/.wdm/drivers/chromedriver/mac64/117.0.5938.92/chromedriver-mac-arm64/chromedriver"
}
}
Output of browser version command from os_manager.py:
$ /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --version
Google Chrome 117.0.5938.92
```
In my experience, sharing chrome binaries across threads was a huge pain. I found it easiest to set a maximum amount of chrome binaries to download, based on the expected load and storage limitations. I cache the paths when an install is actually used. when a thread gets locked out of one chrome binary path it just tries the next until it hits one that isn't being used with a thread.
- Check if chrome binary path is set in the cache, if a path is cached, check if its executable. If path isnt set call the typical install() and cache the path returned.
- Try to utilize the chrome binaries in the path, if it fails repeat step 1 with the next cache key until you can create a driver
- Proceed with your stuff
Trying to reproduce this issue with this repository- https://github.com/david-engelmann/selenium_sadness.
i'm using pytest session fixture to download driver before the tests will start so there is no issue with multiple tests writing into the same file, because it's already there when they run
@pytest.fixture(scope="session", autouse=True)
def suite(request):
ChromeDriverManager().install()