i change the user agent and stop getting result
ABaismail opened this issue · comments
Hello, I'm using this beautiful library and I really appreciate your hard work on it.
I have a problem when I make a simple change to the code.
I can't get any result when I make the user agent picked randomly from my file.
this is my query :
`
from googlesearch import search
t=search("google", num_results=10)
print (t)
`
but these are the changes I made in the code :
`
from bs4 import BeautifulSoup
from requests import get
import random
random_pick = open('useragent').read().splitlines()
user_agent =random.choice(random_pick)
def search(term, num_results=10, lang="en", proxy=None):
usr_agent = {
'User-Agent': user_agent}
def fetch_results(search_term, number_results, language_code):
escaped_search_term = search_term.replace(' ', '+')
google_url = 'https://www.google.com/search?q={}&num={}&hl={}'.format(escaped_search_term, number_results + 1,
language_code)
proxies = None
if proxy:
if proxy[:5] == "https":
proxies = {"https": proxy}
else:
proxies = {"http": proxy}
response = get(google_url, headers=usr_agent, proxies=proxies)
response.raise_for_status()
return response.text
def parse_results(raw_html):
soup = BeautifulSoup(raw_html, 'html.parser')
result_block = soup.find_all('div', attrs={'class': 'g'})
for result in result_block:
link = result.find('a', href=True)
title = result.find('h3')
if link and title:
yield link['href']
html = fetch_results(term, num_results, lang)
return list(parse_results(html))
`
the result I got is empty list --> []
could you please figure out why? only works when I put fixed user agent