Nv7-GitHub / googlesearch

A Python library for scraping the Google search engine.

Home Page:https://pypi.org/project/googlesearch-python/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

i change the user agent and stop getting result

ABaismail opened this issue · comments

Hello, I'm using this beautiful library and I really appreciate your hard work on it.

I have a problem when I make a simple change to the code.

I can't get any result when I make the user agent picked randomly from my file.

this is my query :
`
from googlesearch import search

t=search("google", num_results=10)
print (t)
`

but these are the changes I made in the code :

`
from bs4 import BeautifulSoup
from requests import get
import random
random_pick = open('useragent').read().splitlines()
user_agent =random.choice(random_pick)

def search(term, num_results=10, lang="en", proxy=None):
usr_agent = {
'User-Agent': user_agent}

def fetch_results(search_term, number_results, language_code):
    escaped_search_term = search_term.replace(' ', '+')

    google_url = 'https://www.google.com/search?q={}&num={}&hl={}'.format(escaped_search_term, number_results + 1,
                                                                          language_code)
    proxies = None
    if proxy:
        if proxy[:5] == "https":
            proxies = {"https": proxy}
        else:
            proxies = {"http": proxy}

    response = get(google_url, headers=usr_agent, proxies=proxies)
    response.raise_for_status()

    return response.text

def parse_results(raw_html):
    soup = BeautifulSoup(raw_html, 'html.parser')
    result_block = soup.find_all('div', attrs={'class': 'g'})
    for result in result_block:
        link = result.find('a', href=True)
        title = result.find('h3')
        if link and title:
            yield link['href']

html = fetch_results(term, num_results, lang)
return list(parse_results(html))

`
the result I got is empty list --> []
could you please figure out why? only works when I put fixed user agent