Webdriver is being detected
mazshar opened this issue · comments
Seems like one of the bot detectors has detected the behaviour of this script. I've tried running headless but the captchas fail.
Any suggestions?
I haven't seen this before. However, I have found that after running headless it blocks your IP address for a good few hours. After that it seems to work again. You may just need to extend the waiting times between interactions.
driver = webdriver.Chrome(executable_path=chromedriverPath, chrome_options=chrome_options) driver.execute_cdp_cmd('Network.setBlockedURLs', {"urls": ["https://driverpracticaltest.dvsa.gov.uk/nions-to-vnse-the-Bewarfish-so-like-here-hoa-Mon"]}) driver.execute_cdp_cmd('Network.enable', {}) print("Launching queue") driver.get(dvsa_queue_url)
I tried executing this before the main loop so it's the first thing that's executed and still get the same issue. Could you try it, @tp223?
Using undected_chromedriver to create the driver object works - I get the captcha page. E.g.
driver = uc.Chrome(executable_path=chromedriverPath, options=chrome_options)
I tried a similar thing with no luck last night. However, I have found that it is possible to block that script after the first page load which may solve the issue. By the looks of things that script now pings the server with a key so that it knows it's not being blocked. But, after that it doesn't seem to require it. Might be worth looking into.
Are you referring to the '/nions-to-vnse-the-Bewarfish-so-like-here-hoa-Mon' script? If I remove that line, I get an Error 15 Access Denied
Hi, I'm also getting this issue and I've replicated with the following code sample taken from snippets from this repo:
import undetected_chromedriver as uc
uc.install()
from selenium import webdriver
chrome_options = uc.ChromeOptions()
prefs = {"profile.managed_default_content_settings.images": 2}
chrome_options.add_experimental_option("prefs", prefs)
driver = webdriver.Chrome(chrome_options=chrome_options)
driver.execute_cdp_cmd(
"Network.setBlockedURLs",
{
"urls": [
"https://driverpracticaltest.dvsa.gov.uk/nions-to-vnse-the-Bewarfish-so-like-here-hoa-Mon"
]
},
)
driver.execute_cdp_cmd("Network.enable", {})
driver.get("https://driverpracticaltest.dvsa.gov.uk/login")
It's the same result whether I run driver.get on that URL or the queue-it URL. I do believe this is due to some changes the DVSA have made around a day or so ago. I'm guessing they've upped the strictness of their bot detection as undetectable chromedriver is being detected now when it never used to be.
I run my own cancellation driving test software which also begun to fail yesterday so I'm in a similar boat. I'm more than happy to collaborate and see if we can figure a way around this.
It's the same result whether I run driver.get on that URL or the queue-it URL. I do believe this is due to some changes the DVSA have made around a day or so ago. I'm guessing they've upped the strictness of their bot detection as undetectable chromedriver is being detected now when it never used to be.
I run my own cancellation driving test software which also begun to fail yesterday so I'm in a similar boat. I'm more than happy to collaborate and see if we can figure a way around this.
I was a bit confused as to why the selenium webdriver is used to create the driver object but the undetected chromedriver is used for creating options. I did change it to uc.Chrome(options = options) and I get a captcha page which then ends up with the same result.
It's likely to do with the execute_cdp_cmd to block that url as Tom suggest earlier. I tried enabling/disabling various other settings as suggested online but with no luck.
Which cancellation driving test software do you run?
Yes I'm getting similar results. I get the "Pardon our interruption" page if I block their detection script using execute_cdp_cmd. If I don't block that script, I get an "access denied error code 15" page (screenshot attached), which makes sense. It seems like they changed the WAF behaviour to block access if the detection script can't run.
Indeed, it sounds like the driver is being detected based on the execute_cdp_cmd and we can't proceed without blocking the script. There may be a window where we can keep it enabled long enough and then disable it after. Or we need to find another method of disabling this script
Personally I don't think blocking the script will work very well. I think Imperva will have added a good amount of countermeasures against people dropping requests or modifying the behaviour of their detection script. I think the only way forward is to bypass their detection method entirely. It used to be possible with just undetectable chromedriver so maybe modifications to that could help. Additionally, there are other cancellation companies (Testi, cancellations4all), which are still searching even after this update so it seems like there is a way to bypass the detection (assuming they're also using selenium).
For curiosity, you mentioned some other cancellation apps are currently not working. Would you be able to tell me which?
Both the Testi app and website wasn't working for me and neither was Driving Test Cancellations Now the day that this script wasn't working.
Has anyone made progress on this? I've tried various things with no luck. If I understand right, undetected chromedriver is supposed to automatically detect a patch and apply it after you direct it to the blocking url.
I think I have a pretty good solution, if you launch chrome normally using a subproccess with remote debugging enabled the new script doesn't pick up on anything. You can then just connect selenium to the active chrome window and then block the script and refresh. Something like this:
mgmtPort = "8745"
print("Managing chrome on port " + mgmtPort)
portStr = "--remote-debugging-port=" + mgmtPort
chromeProfileDir = "/home/cadt/Desktop/chrome-profiles/chrome-profile-" + mgmtPort
os.mkdir(chromeProfileDir)
dataDirOpt = "--user-data-dir=" + chromeProfileDir
busterExtCmd = "--load-extension=" + busterFolder
launchCmdUbuntu = ["google-chrome", portStr, dataDirOpt, busterExtCmd, "--no-first-run", "--blink-settings=imagesEnabled=false", "https://driverpracticaltest.dvsa.gov.uk/login"]
chrome_options = webdriver.ChromeOptions()
mgmtAddress = "localhost:" + mgmtPort
chrome_options.add_experimental_option("debuggerAddress",mgmtAddress)
subprocess.Popen(launchCmdUbuntu)
time.sleep(20)
driver = webdriver.Chrome(executable_path=chromedriverLocation, chrome_options=chrome_options)
driver.execute_cdp_cmd('Network.setBlockedURLs', {"urls": ["https://driverpracticaltest.dvsa.gov.uk/nions-to-vnse-the-Bewarfish-so-like-here-hoa-Mon"]})
driver.execute_cdp_cmd('Network.enable', {})
driver.refresh()
Just note that this is running on Ubuntu, if you are using windows you will need to replace google-chrome with start chrome otherwise it may break. Let me know if you manage to get this working too as it seems to working well.
Would it be possible to modify the main.py with the updated code?
Thanks
In the light of recent issues and updates does anyone manage to run the code properly?
I have rewritten the underlying logic from this repo as YADC using @tp223's approach. The code has hopefully benefitted from the opportunity to rewrite. I also use tor to connect from a different ip each time, which seems to cure the problem of getting blocked for long periods. Of course, it is only a matter of time before some better way to profile is thought up and this stops working.
Hey Guys, does anyone have the fix for this yet?
The real malicious bots are snatching tickets and reselling at impossible markups, not finding people driving tests.
@2e0byo Might be a stupid question, but how do you know about these bots that are reselling bookings? Wouldn't you need to transfer the booking from one driving licence number to another?
edit: The last sentence is about bots in general not specifically about driving tests.
@Konhub I was talking about ticket bots; I am not aware of any attempts to do this with driving tests, likely for exactly the reason you mention. The prevalence of ticket bots is how companies like imperva and queue-it sell their products.
Note that this is off topic for this issue, and this issue is closed---it's better to open a new issue, even just for questions.