Cookies accept

[Gianluca Perna]

Your question

Hi, I am trying to do some experiments on browsing and measuring different websites, and I want to complete the experiments by also considering the websites when the cookies are accepted.

To achieve this with Browsertime, I am trying to do the following:

-first, start with the "real Chrome browser", browse the web pages and accept the cookies manually.
This way the cookie's database should be built (which is stored in ~.config/google-chrome/Default if you use Ubuntu).

-second, start browsertime and set --chrome.args user-data-dir to thepath ~.config/google-chrome/.
Unfortunately, this does not work, even if I try to pass the same binary from Google Chrome with --chrome.binaryPath.

Anyway, when I open Chrome using browsertime and browse to chrome://version, I can see that the parameters are correct, the browser actually also loads all the extensions I have in my own Chrome.
When I go into the settings cookies, I can also see the list of cookies, but when I surf on a web page that should be in the list, I get the banner of accept cookie to press.

So I would like to ask you if I missed something or if I am wrong in the methodology, or if there is a way to solve this problem (start the navigation with a known profile and a relative cookie list avoiding the cookie accept banner).

Thanks a lot!

[Peter Hedenskog]

Hi @GianlucaPoliTo I think Chrome starts with a clean profile everytime. I think the way to do it is to setup a new profile directory, you do that with the switch --user-data-dir=/path/to/dir to Chrome, and then when you run Browsertime add --chrome.args "user-data-dir=/path/to/dir" or something. I know that the Firefox team sets up new profiles for their tests but I haven't tested with Chrome. Please let me know if it works. If it do not work, I'll look into, maybe there's something in the code that's causing it that can be fixed.

[Gianluca Perna]

Hi @soulgalore, thanks for the reply. The flag you suggest is exactly what I tried, but it seems that browsertime-chrome somehow "rejects" the option.

I mean, if I create the desired profile using my own Chrome and then use
--user-data-dir to link the correct path to browsertime-chrome, it does not seem to work for some reason.

To clarify, let us always assume that we are looking at the profile in the official directory, then the following command will not work:

browsertime --chrome.args user-data-dir="~/.config/google-chrome" https://website.com


also

browsertime --chrome.binaryPath="/usr/bin/google-chrome" --chrome.args user-data-dir="~/.config/google-chrome" https://website.com


I understand that because of the encryption, it is impossible to copy the cookie database to another machine if you create the profile on another machine.

But if I create the profile on the same machine and test it, it should work, right?

At the moment it seems to work like this instead:

1. Create the profile (e.g. in /tmp/chrome_profile) with npm-browsertime (this way I can navigate and accept the cookies by hand).
2. Start experimenting with npm-browsertime with --no-sandbox and link the profile created above.

But when I try to apply the profile created in 1) in browsertime-docker, again this does not seem to work.

Do you know if this is normal or not?
Thank you very much, unfortunately debugging Chrome seems to be impossible sometimes.

P.S. The chrome://version page is helpful to get some information in case of debugging.

[Peter Hedenskog]

I see, let me try tonight to see how it works. When I dealt with cookies, before I just copy/pasted the cookie from the network tab and added it as a header, but then you need to match the cookie per URL, setting up a profile would be cool.

There's some info here (for setting up another profile with --profile-directory): https://stackoverflow.com/questions/52394408/how-to-use-chrome-profile-in-selenium-webdriver-python-3

Before I try, did anything work for you, starting with NodeJS did it pickup the correct profile with pre made cookies and then switching to Docker failed or? I didn't fully follow.

This would be great to get working, let me try later today and if we can get it to work I'll add it to the documentation.

[Peter Hedenskog]

Also the --debug mode can be very helpful here in Browsertime. Use it and the browser will pause after the navigation, I think that's the easiest way to setup the profile.

[Peter Hedenskog]

I did a quick test and for Node JS this seems to work:

browsertime https://www.1177.se -n 1 --chrome.args user-data-dir="/path/config/google-chrome" --debug

And then I click all cookie banners etc and close the browser and then run:
browsertime https://www.1177.se -n 1 --chrome.args user-data-dir="/path/config/google-chrome"

[Peter Hedenskog]

I'll try with Docker tonight.

[Gianluca Perna]

I did a quick test and for Node JS this seems to work:

browsertime https://www.1177.se -n 1 --chrome.args user-data-dir="/path/config/google-chrome" --debug

And then I click all cookie banners etc and close the browser and then run: browsertime https://www.1177.se -n 1 --chrome.args user-data-dir="/path/config/google-chrome"

Yes, I can confirm that this works, and that is also the only case where I can make it work.

To be as clear as possible I'll continue with some examples.

Small disclaimer: if you are not using Linux (Ubuntu) the path for user-data-dir could be different.

Example 1 - reuse my own original chrome profile with browsertime npm

browsertime --chrome.args user-data-dir="~/.config/google-chrome" https://website.com/

This doesn't work.

Example 2 - build the profile in /tmp/myChromeProfile using browsertime npm and use it with docker

browsertime --chrome.args user-data-dir="/tmp/myChromeProfile" https://website.com/ --debug

docker run -v /opt/google:/opt/google -v /tmp:/tmp sitespeedio/browsertime --chrome.args user-data-dir="/tmp/myChromeProfile -n 1" https://website.com/

I do not know yet if this can work in any way.

The final question is, if we consider always the same machine (otherwise thanks to encryption you cannot do anything), are we able to transfer the chrome profile from docker-browsertime to browsertime-npm to chrome-base and vice-versa?

I'm quite convinced that the answer is yes, but I'm still trying to understand in which way we must provide all the paths/binaries, etc.

Let me know if something is not clear.

Thanks a lot for your help!

[Gianluca Perna]

Ok, I can now confirm that example 2 should work. I tried it out right away!
So it's possibile to move the profile from browsertime-npm to docker-browsertime (at the moment there was an error doing the reverse).

But I am not yet able to do the same from chrome-base to npm-browsertime/docker-browsertime.

[Peter Hedenskog]

Ok, do I understand correctly that if you start Chrome yourself, you haven't been able to use that profile for browsertime? And you you used the same switch to set the data dir for Chrome?

[Gianluca Perna]

Yes, at the moment I cannot use the original profile of my own Chrome in Browsertime-Chrome.
On Ubuntu the original profile is in ~/.config/google-chrome, if you try to pass that path to Browsertime with user-data-dir it does not work.
At this point I am just curious to understand why, because I mean, now is possible to create the profile with browsertime-chrome and use it in docker-browsertime.

[Peter Hedenskog]

I would look at the Chromdriver log and see if you get ant errors when you try to use the profile.

[Peter Hedenskog]

Closing since nothing happend for over a year.