crawler-python

There are 3 repositories under crawler-python topic.

• mlscraper

  lorey / mlscraper

  🤖 Scrape data from HTML websites automatically by just providing examples

  crawlercrawler-pythoncrawlingextraction-enginehtmlmachine-learningscraperscraping
  Language:Python 1231
• weixin_crawler

  wonderfulsuccess / weixin_crawler

  稳定工作4年的微信公众号爬虫 Based on python and vuejs 微信公众号采集 Python爬虫 公众号采集 公众号爬虫 公众号备份

  crawler-pythonpythonvuejsweixin-crawler
  Language:Python 259

• WwwwwyDev / crawlist

  A universal solution for web crawling lists

  crawlcrawlercrawler-pythonpythonreptilecrawling-python
  Language:Python 115
• WebScrapper

  nuhmanpk / WebScrapper

  Simple and powerfull all in one Telegram Bot to scrap / crawl webpages using Requests, html5lib and Beautifulsoup

  telegram-botwebscrapingrequestsbeautifulsoup4pyrogram-botpyrogramtelegramwebscrappingwebscrapperwebscrapping-pythoncrawlerscraperscrapingweb-scrapinghacktoberfesthacktoberfest-acceptedhacktoberfest2023seleniumcrawler-enginecrawler-python
  Language:Python 103

• guilatrova / GMaps-Crawler

  Google Maps crawler using Selenium. All extracted data is forwarded to a SQS queue.

  antifragiledevcrawlercrawler-pythongoogle-mapspythonselenium-python
  Language:Python 72

• flulemon / sneakpeek

  Sneakpeek is a framework that helps to quickly and conviniently develop scrapers. It’s the best choice for scrapers that have some specific complex scraping logic that needs to be run on a constant basis

  crawlercrawler-pythoncrawlerscrawlingcrawling-enginecrawling-frameworkpythonpython3scraperscraper-apiscraper-enginescrapersscrapingscraping-frameworkvuewebsite-crawler
  Language:Python 36

• andripwn / crawler-python

  email scraper/crawls using python (Google/Bing)

  email-crawleremail-scrapercrawler-python
  Language:Python 22

• DEENUU1 / meta-spy

  👾 CLI MetaSpy (Facebook, Instagram) scraper and crawler - instagram account, facebook accounts, pages and search

  clifacebookfacebook-loginfastapipythonpython3richscraperseleniumsqlitesqlite3typertyper-cliwebcrawlercrawler-pythongraphcsshtmljinja2
  Language:Python 22

• JimouChen / bing-chat-fxxk

  newbing api by PlayWright

  bing-apicrawlercrawler-pythongpt
  Language:Python 22

• pyladies-brazil / crawler-tutorial

  Tutorial de raspagem de dados realizado em parceria com a JusBrasil

  beautifulsoupcrawlcrawling-pythonpyladiespyladies-brasilpyladies-workshoppythonpython-tutorialrequests-pythonbrazilcrawlercrawler-pythonraspagem-de-dados
  Language:HTML 22

• vlmaier / marvel-snap-scrapr

  Scraper for https://marvelsnapzone.com to retrieve metadata of Marvel SNAP cards.

  marvelmarvel-charactersmarvel-snapcrawlercrawler-pythongamescraperwebsite-crawlerwebsite-scraper
  Language:Python 21

• KSMubasshir / bd-newspaper-crawlers

  A collection of Bangla newspaper and blog crawlers. Can be used to mine bangla text data for Natural Language Processing tasks.

  bangla-datasetbangla-dataset-for-opinion-miningbangla-dataset-machine-translationbangla-language-modelbangla-language-processingbangla-natural-language-processingbangla-nlpcrawlercrawler-pythondata-collectiondataset-generationnewspapernlp
  Language:Python 10

• BaseMax / StackoverflowCrawler

  A web crawler which crawls the stackoverflow website.

  stackoverflow-crawlercrawlercrawlerscrawler-testingcrawler-detectorcrawler-pythonpython-crawlerweb-crawlerweb-crawler-pythontest-crawlerstackoverflowstackoverflow-questionsstackoverflow-apistackoverflow-answerstackoverflow-tagsstackoverflow-getstackoverflow-analysecrawlingtext-processingtext-processor
  Language:Python 9

• Bacdong / web-crawler

  Crawler website with requests library in python

  pythonpython3python-spiderpython-requestspython-librarycrawler-python
  Language:Python 8

• Xunzhuo / AirSpider

  A Fast and Light Python Spider Framework 🕷️

  redisspiderspider-frameworkpython3distributedasynchronouswebcrawlercrawler-python
  Language:Python 8

• changhyeonnam / Google-Full-size-image-crawler

  crawling google full size image

  chrome-crawlercrawlercrawler-pythonfull-size-imagegoogle-images-crawlergoogle-images-downloader
  Language:Python 7

• mcxiaoxiao / xiaohongshuCrawler

  小红书简易爬虫 📕 获取文章title、文章id、文章内容、话题标签

  crawler-pythoncrawling-sites
  Language:JavaScript 7

• michaelradu / web-crawler

  A Web Crawler developed in Python.

  webcrawlercrawlerscrawler-pythonwebcrawlerwebcrawlingwebcrawlweb-crawlerweb-crawlingweb-crawler-pythonweb-crawlerspythonpython3python-3python-scriptscriptscriptsscriptingscripting-language
  Language:Python 7
• Reddit-Crawler

  RaccoonTamer / Reddit-Crawler

  The "Reddit Image Crawler" is a Python script that facilitates the extraction and downloading of image and gifs URLs from a specified subreddit on Reddit. It also includes functionalities to download the images from the fetched URLs, handle duplicate image removal, and rename image files in a directory.

  api-freecrawlercrawler-pythondownloaderimage-analysispythonredditreddit-apireddit-applicationreddit-clientreddit-crawlerreddit-downloaderreddit-theme
  Language:Python 6

• gabfl / sitecrawl

  Simple Python module to crawl a website and extract URLs

  crawlercrawler-pythoncrawling-sitescrawl
  Language:Python 5

• jindada1 / Relaxion

  爬虫练手项目（几个音乐平台）

  aiohttp-servercrawler-pythonmusicsqlite3
  Language:Python 5

• Allama272 / OLX-Mining-and-analyzing-house-prices

  Automatically scraps and collects olx apartment data every 15 days

  beautifulsoupbeautifulsoup4crawlercrawler-pythondata-analysisolxolx-crawlerpandaspythonpython3scraperscraping-websites
  Language:Python 4

• itszeeshan / crawlinit

  A web crawler written in python3

  appsecbugbountybugbounty-toolbugbountytipscrawlercrawler-pythonenumerationinfosecpythonscannerurlwebreconreconnaissance
  Language:Python 4

• 0JCRG0 / Crawler-remote-jobs

  Crawlers to get remote jobs

  bs4crawlercrawler-pythonrss-feedselenium-pythonweb-scrapingpythondata-engineeringjupyter-notebookpandas-dataframe
  Language:Python 3

• a22057916w / StoreManager

  :house: A shop-rental-and-selling platform that ranks the shops for users

  shopping-cartrent-houseandroid-appcrawler-pythonmysql-databasepractical-applications
  Language:Java 3

• alyrezo / digikala-exif-scraper

  A script that collects exif (metadata) photos sent by buyers in DigiKala.com

  big-databigdatacrawlercrawler-pythoncrawlingdigikalaexifexif-metadatageotagged-photosgeotaggingmetadatascraperscraping
  Language:Python 3

• BaseMax / InstagramSeleniumHashtagImagePython

  Instagram Selenium Python: A selenium-based crawler to extract images from special hashtags on Instagram.

  instagrampythonseleniumpython-seleniumselenium-pythoncrawlercrawlerscrawler-python
  Language:Python 3
• csgo-market-crawler

  ew3g / csgo-market-crawler

  CSGO-Market-Crawler is a web crawler that retrieves items from CSGO Steam Market and stores them in a Mongo Database.

  counter-strikecounter-strike-global-offensivecrawlercrawler-pythonmongodbpythonpython3steamsteam-apisteam-market
  Language:Python 3

• chenmozhijin / mediawikiextractor

  一个用于从 MediaWiki 网站中提取数据并保存为json的 Python 脚本。|A Python script for extracting data from a MediaWiki website and saving it as json.

  crawlercrawler-pythoncrawlingextractorjsonmediawikipythonregexweb-crawler
  Language:Python 2

• Idealisten / OneClickToLearnFotonUniversityOnlineStudyPlatform

  study at http://study.foton.com.cn

  crawler-pythononline-courseautomatic-learning
  Language:Python 2

• jasonren0403 / app_crawler

  基于scrapy的应用商店爬虫，包括应用信息本身及其评论

  crawler-pythonscrapyscrapy-crawler
  Language:Python 2

• SH112-kr / APK-Malware-Detector

  악성 APK 탐지 크롤러

  apkcrawler-pythonmalware-detectionpython3
  Language:Python 2

• tony-aptx4869 / microsoft_order_crawler

  Python scripts for crawling Microsoft orders information. / 用来爬虫 Microsoft 订单信息的 Python 脚本。

  crawlercrawler-pythoninformationmicrosoftorderorders-digitalpythonxbox
  Language:Python 2
• fcrawler

  truethari / fcrawler

  Python application that can be used to copy files of a given file type from a folder directory.

  copycopy-filescrawlcrawlercrawler-pythonfilefiles
  Language:Python 2

• jmb-ops / Spyder

  A web crawler named Spyder. a command line tool like (ZAP) Zed Attack Proxy made for spidering/ crawling web pages made using only the python standard library. meaning no dependencies. For windows.

  crawler-pythonscraper-pythonweb-crawlerweb-scrapercrawlingspideringcrawlerpython
  Language:Python 1

• Williams-Media / Exipred-Domain-Finder

  Python script to crawl a website and see if it links to any expired domains.

  crawlercrawler-pythonexpired-domainexpired-domainsseo
  Language:Python 1