crawling-python

There are 3 repositories under crawling-python topic.

• Scrapling

  D4Vinci / Scrapling

  🕷️ An undetectable, powerful, flexible, high-performance Python library to make Web Scraping Easy and Effortless as it should be!

  aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
  Language:Python 8114

• lorien / awesome-web-scraping

  List of libraries, tools and APIs for web scraping and data processing.

  web-scrapingcaptcha-bypasscaptcha-recaptchacrawlingcrawling-frameworkcrawling-pythoncrawling-toolscrapingscraping-frameworkscraping-pythonscraping-toolwebscrapingcrawlerspider
  Language:Makefile 7423

• watercrawl / WaterCrawl

  Transform Web Content into LLM-Ready Data

  aicrawlercrawl4aicrawlercrawling-pythonhtml2markdownllm-crawlerllm-scraperscraper
  Language:TypeScript 1517

• scrapfly / scrapfly-scrapers

  Scalable Python web scraping scripts for +40 popular domains

  crawlingpythoncrawlerscrapingweb-scrapingweb-scraping-pythonantibotautomationcaptcha-bypasscrawling-pythondatascrapingproxiespython-scraperscraperscraping-pythonspidertwitter-scraperweb-crawlerwebscraperwebscraping
  Language:Python 749

• shaohua0116 / ICLR2019-OpenReviewData

  Script that crawls meta data from ICLR OpenReview webpage. Tutorials on installing and using Selenium and ChromeDriver on Ubuntu.

  crawlercrawling-pythonopenreviewtutorial
  Language:Jupyter Notebook 387
• telegram-crawler

  MarshalX / telegram-crawler

  🕷 Automatically detect changes made to the official Telegram sites, clients and servers.

  crawlerparsertelegramtelegram-orgtelegram-updatescrawlingcrawling-python
  Language:Python 330

• WwwwwyDev / crawlipt

  The script for selenium in python. Make automated testing easier! 使用json脚本驱动selenium

  crawling-pythonreptileseleniumselenium-pythontest-automationtesting-toolscrawlipt
  Language:Python 155

• WwwwwyDev / crawlist

  A universal solution for web crawling lists. 抓取网页列表的通用解决方案

  crawlcrawlercrawler-pythonpythonreptilecrawling-pythoncrawlist
  Language:Python 110

• thewebscraping / tls-requests

  TLS Requests is a powerful Python library for secure HTTP requests, offering browser-like TLS client, fingerprinting, anti-bot page bypass, and high performance.

  tls-clientcloudflare-scraperweb-scraping-apicloudflare-bypasscf-clearanceanti-botanti-bot-detectionanti-bot-pagepython-tls-clientpython-crawlerpython-spiderweb-scraping-pythonpython-web-scraperpython-web-scrapingpython-web-crawlerweb-spiderscraping-pythonweb-crawler-pythonpython-scrapercrawling-python
  Language:Python 108

• zhouyi207 / WeiBoCrawler

  微博数据采集，微博爬虫，微博网页解析，完整代码（主体内容+评论内容）

  crawling-pythondataweibo-spidervisualizationweibo
  Language:Python 87

• MLArtist / WebScraper

  Python-based web crawling script with randomized intervals, user-agent rotation, and proxy server IP rotation to outsmart website bots and prevent blocking.

  crawling-pythoncrawlerscraperscrapping-pythonscrapingscrapperwebsite-scraperwebsite-crawlerrobots-txtuser-agentiprotationbeautifulsoupbeautifulsoup4beautiful-soup
  Language:Python 84

• fernandod1 / Instagram-downloader

  Instagram user's photos and videos downloader. Download all media files from any username. Working 2022!

  crawlercrawling-pythoninstagraminstagram-downloaderinstagram-feedinstagram-photosinstagram-scraperpythonscrapscraperscrapingscraping-pythonscraping-toolscraping-websites
  Language:Python 74

• xishandong / Android_reverse

  此项目分享安卓逆向的实战案例以及学习笔记，适合新手学习，随着作者逐渐变成大神，这个仓库也会适合大神学习~

  android-applicationcrawling-pythonreverse-engineering
  Language:Python 65

• odaysec / NewsCrap

  NewsCrap adalah alat scraping berita Google berbasis Command Line Interface (CLI) yang dirancang untuk riset, investigasi, dan pengumpulan data OSINT. Dengan fitur canggih seperti rotation proxy, scheduling otomatis, dan multi-format export, alat ini memudahkan pengumpulan data berita secara efisien dan andal.

  osint-toolscraperscraping-websitesscrapping-pythonnewscrapcrawlercrawling-python
  Language:Python 52

• wael-sudo2 / facebook-page-info-scraper

  Free Facebook pages MetaData Scraping Library - Unlimited Calls

  crawlercrawling-pythonfacebookfacebook-page-informationfacebook-scraperfacebook-scrapingmetadatapythonscraping-pythonseleniumdata-analysisdata-miningcrmleadgenerationmarketingscrapingfacebook-apisfacebook-page-scraper
  Language:Python 41

• Galarzaa90 / tibia.py

  API to parse tibia.com content into python objects.

  tibiapythonpython3beautifulsoupwebcrawlingcrawling-python
  Language:Python 40

• mike-gee / webtranspose

  Web scraping API for building AI applications.

  chatbotscrawlingcrawling-pythonpythonscrapingscraping-pythonweb-crawlingweb-scrapingweb-scraping-python
  Language:Python 40

• helviojunior / filecrawler

  File Crawler index files and search hard-coded credentials

  crawlercrawling-pythonelasticsearchleaksleaks-scanner
  Language:Python 35

• samzhangjy / BaiduSpider

  项目已经移动至：https://github.com/BaiduSpider/BaiduSpider ！！ 一个爬取百度搜索结果的爬虫，目前支持百度网页搜索，百度图片搜索，百度知道搜索，百度视频搜索，百度资讯搜索，百度文库搜索，百度经验搜索和百度百科搜索。

  python3spidercrawlingcrawling-pythonpythonbaiduapispidersbaiduspider
  Language:Python 34

• omkarcloud / botasaurus-starter

  🚀 OFFICIAL STARTER TEMPLATE FOR BOTASAURUS SCRAPING FRAMEWORK 🤖

  beautifulsoupcrawlercrawlingcrawling-frameworkcrawling-pythoncrawling-toolheadlessnode-crawlerpython-crawlerscraperscrapingscraping-frameworkscraping-pythonscraping-toolseleniumweb-crawlerweb-crawlingweb-scraperweb-scrapingwebscraping
  Language:TypeScript 29

• M-Taghizadeh / Dollar_Rial_Price_Dataset

  In this dataset, the price of the dollar to the Iranian rial in the years 2011 to 2023 has been collected by our crawler.

  deep-learningfinancehuggingfacekagglelstmpythonrnntime-seriestime-series-predictiontransformercrawling-pythonselenium
  Language:Python 28

• pyladies-brazil / crawler-tutorial

  Tutorial de raspagem de dados realizado em parceria com a JusBrasil

  beautifulsoupcrawlcrawling-pythonpyladiespyladies-brasilpyladies-workshoppythonpython-tutorialrequests-pythonbrazilcrawlercrawler-pythonraspagem-de-dados
  Language:HTML 25
• linkedin-profile-picture

  shashankdeshpande / linkedin-profile-picture

  This package is used to get a profile picture of the LinkedIn user using Google Custom Search API

  linkedinlinkedin-profilelinkedin-scraperlinkedin-crawlerpythonlinkedin-profile-scraperlinkedin-profile-urllinkedin-profile-picturelinked-profile-photoprofile-picturecrawling-pythongoogle-search-using-pythongoogle-custom-search-apipython3linkedin-user
  Language:Python 25

• thaoshibe / crawl-original-google-images

  python scripts for crawling original image from Google Images

  chrome-extensioncrawlercrawlingcrawling-pythongooglegoogle-imagespafyscraperyoutubeyoutube-dlyoutube-search
  Language:Python 23

• LiveCoronaDetector / covid-19-crawler

  코로나 확진자 수/정보 크롤링

  kcdccrawlercrawling-pythoncoronavirus
  Language:Python 20

• t-ega / Terader-Movie-Hub-Telegram-Bot

  A Telegram Bot to help automate movie search and retirevals

  botscrawling-pythonpythontelegram
  Language:Python 17

• serpwings / data-science-for-digital-marketers

  Juypter Notebooks for Lecture Series on Data Science for Digital Marketers

  amazoncrawling-pythondata-sciencedigital-marketingmarketingpython
  Language:Jupyter Notebook 13

• spicyparrot / kafka_scrapy_connect

  A custom library that integrates Scrapy with Kafka.

  crawling-pythonkafkapythonscraping-pythonscrapy
  Language:Python 12

• SMSadegh19 / ResearchGateCrawler

  Python script for crawling ResearchGate.net papers.✨⭐️📎

  seleniumcrawlerpaper-crawlercrawlingcrawling-pythonresearchgateresearch-paperresearch
  Language:Python 11

• deepmancer / advanced-recommender-system

  Advance information retrieval system that combines advanced indexing, machine learning, and personalized search to enhance academic research and document discovery.

  bigram-modelcollaborative-filteringcrawling-pythonfine-tuninginformation-retrievallanguage-modelnatural-language-processingnlppositional-indexingpytorchrecommender-systemseleniumspelling-correctiontokenizationtransformersvectorization
  Language:Jupyter Notebook 9

• Esequiel378 / proxy_randomizer

  This library helps you sfetly crawle apis and web pages

  crawling-pythonproxy-listpython3web-scrapingproxy-randomizer
  Language:HTML 8

• omkarcloud / web-scraping-template

  🚀 THIS WEB SCRAPING TEMPLATE PROVIDES YOU WITH A GREAT STARTING POINT WHEN CREATING WEB SCRAPING BOTS. 🤖

  beautifulsoupcrawlercrawlingcrawling-frameworkcrawling-pythoncrawling-toolheadlessnode-crawlerpython-crawlerscraperscrapingscraping-frameworkscraping-pythonscraping-toolseleniumweb-crawlerweb-crawlingweb-scraperweb-scrapingwebscraping
  Language:Python 8

• 0MeMo07 / Web-Crawler

  Web Crawler with Python

  beautifulsoup4bs4crawlercrawlerscrawlingcrawling-pythonweb-crawlerweb-crawler-pythonweb-crawlingwebcrawler
  Language:Python 6

• ilteriskeskin / football-tracker-crawler

  Generate football player data

  crawlercrawling-pythonfootballfootball-datafootballdata
  Language:Python 6

• JaShakouri / time.ir-crawling

  api getting iran holidays per years or months

  holidays-apicrawling-pythonjalalitimeir
  Language:Python 6

• Anzo52 / osintbeast

  Combining (mostly) Python OSINT tools into a single framework with support for sqlite3 database, currently working on mysql support.

  crawling-pythoninformation-gatheringinformation-retrievalinfosecopen-source-intelligenceopen-source-researchosintosint-frameworkosint-pythonsocmint
  Language:Python 5