Fails on groups with adult content

Question

Fails on groups with adult content

RuinCakeLie opened this issue 8 years ago · comments

I'm trying to scrape a group (https://groups.google.com/forum/#!topic/3dprintertipstricksreviews/) with the adult content flag turned on. Unfortunately, even using cookies all the escaped_fragment requests only return:

Adult Content Warning

The Group you selected has been identified by its owner as containing adult content.

Ky-Anh Huynh · Answer 1 · Mon Mar 21 2016 07:52:13 GMT+0800 (China Standard Time)

Interesting. I will take a look. Thanks for your reporting.

Ky-Anh Huynh · Answer 2 · Tue Mar 22 2016 08:03:55 GMT+0800 (China Standard Time)

Google yields empty contents when escaped_fragement is specified, e.g.

https://groups.google.com/forum/?_escaped_fragment_=forum/3dprintertipstricksreviews

This is against (?) the standard. We need a different way to receive data from Google. This is a real challenge!

Ky-Anh Huynh · Answer 3 · Tue Mar 22 2016 09:28:29 GMT+0800 (China Standard Time)

Google hides most email headers from the raw message. A raw message isn't actually raw ;)

See also https://groups.google.com/forum/message/raw?msg=3dprintertipstricksreviews/LDFZVHeC8Uk/2D1YhGqGDQAJ

Date: Sun, 20 Mar 2016 06:28:20 -0700 (PDT)
From: Rich Webb <ml...@rawebb.net>
To: 
    "3D Printer Tips, Tricks and Reviews" <3dprintertips...@googlegroups.com>
Message-Id: <d7e58e48-c160-436e-8bdf-10d86a0dc170@googlegroups.com>
Subject: Direction-dependent extrusion volume / track width?
MIME-Version: 1.0
Content-Type: multipart/mixed; 
    boundary="----=_Part_4198_351838098.1458480500604"

------=_Part_4198_351838098.1458480500604
Content-Type: multipart/alternative; 
    boundary="----=_Part_4199_1172380407.1458480500604"

------=_Part_4199_1172380407.1458480500604
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit

Ky-Anh Huynh · Answer 4 · Sat Mar 26 2016 16:44:05 GMT+0800 (China Standard Time)

It's impossible to use traditional method to fetch data from this group. We need to use some higher level tool like phantomjs.

Well, after days of trying scrolling method, I've finally found a way to automate the process. There are two other challenges, but they're definitely solvable.

Stay tuned!

icybin · Answer 5 · Fri Mar 30 2018 11:22:04 GMT+0800 (China Standard Time)

I have some initial works on this issue, but (1) it's slow (2) it's undetermined. Maybe I am not good at selenium.

I'm expecting there's someone can help. I can raise a small fund to support you.

Thanks a lot