deedy5 / duckduckgo_search

Search for words, documents, images, videos, news, maps and text translation using the DuckDuckGo.com search engine. Downloading files and images to a local hard drive.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

automatically get new UserAgents

MadAim123 opened this issue · comments

Is your feature request related to a problem? Please describe.
The UserAgents will be old after some days. An automatically way to update them could be good

Describe the solution you'd like
Crawl the newest 4 needed useragents without manually work

Describe alternatives you've considered
Something like this could be used in "utils":


try:
    # JSON URL containing user agents
    json_url = "https://jnrbsn.github.io/user-agents/user-agents.json"

    # Fetch the JSON data from the URL
    response = requests.get(json_url)
    user_agents_json = response.json()

    # Extract the first 4 user agents
    first_4_user_agents = user_agents_json[:4]

    # Save the user agents into the USERAGENTS list
    USERAGENTS = first_4_user_agents

except Exception as e:
    USERAGENTS = [
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36",
        "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36",
        "Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36",
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36",
    ]

Hello!
Thanks for the suggestion, but there's no point in doing it.

  1. Api requests will work with headers of previous browser versions as well.
  2. User can specify any headers he wants.

At least for the moment.