ChromeNotFound when trying to crawl Immo

[robertszooba]

Hi,

I want to crawl Immo but after executing python3 flathunt.py, I received the following error:

File "/flathunter/flathunter/chrome_wrapper.py", line 53, in get_chrome_version
    raise ChromeNotFound()
flathunter.exceptions.ChromeNotFound

I was using flathunter three months ago, and it did work back then. The only thing that has changed, at least on my side, is an update to Sonoma.

I've seen that there was an issue related to this problem, but apparently it was solved by installing chrome. I'm using chroma on a daily basis, so I'm looking for another solution.

[Arthur Taylor]

Yeah - the other issue was #495. If you're seeing ChromeNotFound, it's really usually because the Chrome binary isn't in your path. You can try running google-chrome --version or chromium --version or chrome --version from the terminal where you run python. If all of those fail, it can't find the version of Chrome that it's supposed to use.

[robertszooba]

Do you know how to add Chrome to path?
First I tried with:

sudo ln -s /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome /usr/local/bin/google-chrome

But I kept receiving error messages. So I tried with --headless, i.e. made a script called /usr/local/bin/google-chrome and put the following there:

#!/bin/bash
/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome "$@" --headless

But I'm getting:
ContextResult::kTransientFailure: Failed to send GpuControl.CreateCommandBuffer.

Made a bit of googling, tried a solution with --disable-gpu, but it still does not work.

[Arthur Taylor]

I'm afraid you'll need to wait for someone with a Mac to reply. I'm running on Linux, so that's nothing I can test or reproduce.

[robertszooba]

If I understand correctly, we just want to get the major version of the installed Chrome. According to your knowledge about the program, is there anything that prevents simply returning 118 from function get_chrome_version?

EDIT: It ain't that easy. When I do this, I get Support/undetected_chromedriver/undetected_chromedriver. I guess I must find a way to add Chrome to path.

[Arthur Taylor]

Yeah - undetected_chromedriver is also going to need to be able to find Chrome, though if might have a different technique for doing that.

You can hard-code the version number, but then you just need to be aware that you'll have to change that code every time the chrome version changes (which is roughly every couple of months). The code is there to support flathunter working smoothly over a longer period of time.

[robertszooba]

Ok, resolved by creating a script (as above), but without the --headless option.

[noahskelton]

@robertszooba Got a solution I can use? I'm running into the same problem on MacOS

[friedrichtroescher]

I am running into the same issue. @robertszooba, can you take a look at my version of the same problem you had?

Edit to clarify:
I tried using sudo ln -s as you did without success. Creating the link manually doesn't work, as expected, and produces this error message on starting flat hunter:

sudo pipenv run python flathunt.py
[...]
[2023/11/05 02:17:08|chrome_wrapper.py |INFO ]: Initializing Chrome WebDriver for crawler...
[...]
flathunter.exceptions.ChromeNotFound

Then, trying to create the script manually makes flathunter produce this error:

sudo pipenv run python flathunt.py
[...]
[2023/11/05 02:11:46|chrome_wrapper.py |INFO ]: Initializing Chrome WebDriver for crawler...
Traceback (most recent call last):
[...]
PermissionError: [Errno 13] Permission denied: 'google-chrome'

okay, so I set the permissions with

chmod 755 /usr/local/bin/google-chrome

now, depending on wether I include the --headless in the script, I either get

if headless or options.headless:
AttributeError: 'ChromeOptions' object has no attribute 'headless'

or the original flathunter.exceptions.ChromeNotFound error.

Does anyone have any insight on this?