scraper

There are 106 repositories under scraper topic.

• huginn / huginn

  Create agents that monitor and act on your behalf. Your agents are standing by!

  agentautomationfeedfeedgeneratorhuginnmonitoringnotificationsrssscrapertwittertwitter-streamingwebscraping
  Language:Ruby 35867

• cheeriojs / cheerio

  Fast, flexible, and lean implementation of core jQuery designed specifically for the server.

  cheeriojqueryhtmlparser2domhtmlparserselectorscraperparserhtml
  Language:TypeScript 25170

• iawia002 / lux

  👾 Fast and simple video download library and CLI tool written in Go

  bilibilicrawlerdownloaddownloadergogolanghacktoberfestiqiyiqqscrapertumblrvideoyoukuyoutube
  Language:Go 18097

• gocolly / colly

  Elegant Scraper and Crawler Framework for Golang

  crawlercrawlingframeworkgogolangscraperscrapingspider
  Language:Go 16841

• codelucas / newspaper

  News, full-text, and article metadata extraction in Python 3. Advanced docs:

  crawlercrawlingnewsnews-aggregatorpythonscraper
  Language:Python 11963

• pwxcoo / chinese-xinhua

  :orange_book: 中华新华字典数据库。包括歇后语，成语，词语，汉字。

  datascraperchinese-traditionalpython3chinesechinese-characterschinese-nlpchinese-languagechinese-simplifiedjson-datasetjsonjson-data
  Language:Python 9489

• guyueyingmu / avbook

  AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

  adultadult-videoavmoocrawlerdatabaseguzzlehttpjavbusjavlibrarylaravelmagnetmagnet-linkscraperspider
  Language:PHP 8387

• arc298 / instagram-scraper

  Scrapes an instagram user's photos and videos

  instagraminstagram-scraperinstagram-user-photospythonscraperinstagram-clientinstagram-api
  Language:Python 6891

• yujiosaka / headless-chrome-crawler

  Distributed crawler powered by Headless Chrome

  headless-chromepuppeteerjquerycrawlercrawlingscraperscrapingchromechromiumpromise
  Language:JavaScript 5239
• ferret

  MontFerret / ferret

  Declarative web scraping

  golangquery-languagedata-miningscrapingscraping-websitesdslcdpcrawlingscrapercrawlergochromeclitoollibraryhacktoberfesthacktoberfest2021
  Language:Go 5023

• BruceDone / awesome-crawler

  A collection of awesome web crawler,spider in different languages

  awesomecrawlernode-crawlerscraperspiderweb-crawlerweb-scraper
  4987
• autoscraper

  alirezamika / autoscraper

  A Smart, Automatic, Fast and Lightweight Web Scraper for Python

  aiartificial-intelligenceautomationcrawlermachine-learningpythonscrapescraperscrapingweb-scrapingwebautomationwebscraping
  Language:Python 4467

• IonicaBizau / scrape-it

  🔮 A Node.js scraper for humans.

  hacktoberfestnode-scraperscraper
  Language:JavaScript 3838

• fent / node-ytdl-core

  YouTube video downloader in javascript.

  youtubevideo-downloaderscraperyoutube-downloadernode
  Language:JavaScript 3258

• DIGITALCRIMINAL / OnlyFans

  Scrape all the media from an OnlyFans account - Updated regularly

  onlyfansdatascrapingscraperarchive
  Language:Python 2971

• niespodd / browser-fingerprinting

  Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️‍♂️ when scraping the web?

  automationbotbot-detectionbrowser-fingerprintingchromedriverchromiumchromium-browsercrawlerdetectionfingerprintingpuppeteerrecaptchascraperspiderstealthwebwebscraping
  Language:JavaScript 2950

• aapatre / Automatic-Udemy-Course-Enroller-GET-PAID-UDEMY-COURSES-for-FREE

  Do you want to LEARN NEW STUFF for FREE? Don't worry, with the power of web-scraping and automation, this script will find the necessary Udemy coupons & enroll you for PAID UDEMY COURSES, ABSOLUTELY FREE!

  pythonscrapingseleniumpython3scraper
  Language:Python 2667

• go-rod / rod

  A Devtools driver for web automation and scraping

  automationcdpchrome-devtoolschrome-devtools-protocolchrome-headlesscrawlingdevtoolsdevtools-protocolgogolanggorodheadlessrodscrapertestingwebweb-scraping
  Language:Go 2486

• jae-jae / QueryList

  :spider: The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

  crawlerquerylistscraperspider
  Language:PHP 2467

• realsirjoe / instagram-scraper

  scrapes medias, likes, followers, tags and all metadata. Inspired by instagram-php-scraper,bot

  botcrawlerigigramscraperinstagrampythonscrapescraper
  Language:Python 2387

• jinfagang / weibo_terminater

  Final Weibo Crawler Scrap Anything From Weibo, comments, weibo contents, followers, anything. The Terminator

  scraperweibosinacorpuschinesechatbot
  Language:Python 2288
• Emby.Plugins.JavScraper

  JavScraper / Emby.Plugins.JavScraper

  Emby/Jellyfin 的一个日本电影刮削器插件，可以从某些网站抓取影片信息。

  embyjav-scraperjavpluginscraperfanart-postersynologyjapaneseadultmetadatajavbusjellyfinfc2jsproxy
  Language:C# 2016

• facundoolano / google-play-scraper

  Node.js scraper to get data from Google Play

  google-playscrapercrawlerapinodejs
  Language:JavaScript 1741

• geziyor / geziyor

  Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.

  goscrapingscrapercrawlerspider
  Language:Go 1726
• saikou

  saikou-app / saikou

  An Android Anilist only client, which lets you stream & download Anime & Manga.

  androidanimekotlinscrapermangaanilist9animegogoanime
  Language:Kotlin 1703
• JobFunnel

  PaulMcInnis / JobFunnel

  Scrape job websites into a single spreadsheet with no duplicates.

  jobssearchindeedmonsterglassdoorscraperjobautomatedcsvbeautifulsoupbeautifulsoup4pythontfidfyamlwaterloointernational
  Language:Python 1568

• fabienvauchelles / scrapoxy

  Scrapoxy hides your scraper behind a cloud. It starts a pool of proxies to send your requests. Now, you can crawl without thinking about blacklisting!

  scraperproxyscrapycrawlercloudnodejsangularjsblacklisting
  Language:JavaScript 1478

• aliparlakci / bulk-downloader-for-reddit

  Downloads and archives content from reddit

  archivedownloadergfycatimgurpythonredditscraper
  Language:Python 1381

• JustAnotherArchivist / snscrape

  A social networking service scraper in Python

  pythonscrapersocial-mediasocial-network
  Language:Python 1349

• felipecsl / wombat

  Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.

  rubyscrapercrawlerdsl
  Language:Ruby 1260

• adyzng / jd-autobuy

  Python爬虫，京东自动登录，在线抢购商品

  crawlerjingdongpythonscraper
  Language:Python 1238

• meetDeveloper / freeDictionaryAPI

  There was no free Dictionary API on the web when I wanted one for my friend, so I created one.

  apidictionary-apidictonaryfree-apigooglegoogle-dictionaryscraper
  Language:JavaScript 1206

• website-scraper / node-website-scraper

  Download website to local directory (including all css, images, js, etc.)

  javascriptwebsite-scraperscrapernodejshacktoberfest
  Language:JavaScript 1180

• holgerd77 / django-dynamic-scraper

  Creating Scrapy scrapers via the Django admin interface

  pythondjangoscraperscrapingscrapyspiderwebscraping
  Language:Python 1087
• informer

  paulpierre / informer

  A Telegram Mass Surveillance Bot in Python

  botcryptocurrencylistenlistenermonitorpythonscraperspysurveillancetelegramtelegram-apitelegram-bottelegram-bot-apitelegrambottelethontelethon-userbot
  Language:Python 1031

• cinemagoer / cinemagoer

  Cinemagoer is a Python package useful to retrieve and manage the data of the IMDb (to which we are not affiliated in any way) movie database about movies, people, characters and companies

  actorscastcharactercinemacinemagoercompanydatabasedbimdbinternet-movie-databasemoviemovie-databasemoviesparserpythonscrapersql
  Language:Python 977