crawling

There are 35 repositories under crawling topic.

• scrapy

  scrapy / scrapy

  Scrapy, a fast high-level web crawling & scraping framework for Python.

  crawlercrawlingframeworkhacktoberfestpythonscraping
  Language:Python 43923

• gocolly / colly

  Elegant Scraper and Crawler Framework for Golang

  crawlercrawlingframeworkgogolangscraperscrapingspider
  Language:Go 16900

• codelucas / newspaper

  News, full-text, and article metadata extraction in Python 3. Advanced docs:

  pythonnewscrawlercrawlingscrapernews-aggregator
  Language:Python 11982

• yujiosaka / headless-chrome-crawler

  Distributed crawler powered by Headless Chrome

  headless-chromepuppeteerjquerycrawlercrawlingscraperscrapingchromechromiumpromise
  Language:JavaScript 5251
• ferret

  MontFerret / ferret

  Declarative web scraping

  golangquery-languagedata-miningscrapingscraping-websitesdslcdpcrawlingscrapercrawlergochromeclitoollibraryhacktoberfesthacktoberfest2021
  Language:Go 5025
• apify-js

  apify / apify-js

  Apify SDK — The scalable web scraping and crawling library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

  web-scrapingweb-crawlingjavascript-librarynpmheadless-chromepuppeteerautomationapifyscrapingcrawlingrpa
  Language:JavaScript 3376
• hakrawler

  hakluke / hakrawler

  Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application

  bugbountycrawlinghackingosintpentestingreconreconnaissance
  Language:Go 2807

• go-rod / rod

  A Devtools driver for web automation and scraping

  cdpchrome-headlesschrome-devtoolschrome-devtools-protocolheadlessweb-scrapingautomationscraperdevtoolsdevtools-protocolrodgogolangtestingwebgorodcrawling
  Language:Go 2555

• apache / nutch

  Apache Nutch is an extensible and scalable web crawler

  apachecrawlinghadoopjavanutchweb-crawler
  Language:Java 2393

• transitive-bullshit / awesome-puppeteer

  A curated list of awesome puppeteer resources.

  automationawesomeawesome-listcrawlingheadless-chromepuppeteerscraping
  1842

• zorlan / skycaiji

  蓝天采集器是一款免费的数据采集发布爬虫软件，采用php+mysql开发，可部署在云服务器，几乎能采集所有类型的网页，无缝对接各类CMS建站程序，免登录实时发布数据，全自动无需人工干预！是网页大数据采集软件中完全跨平台的云端爬虫系统

  crawlercrawlingphpspiderwebcrawler
  Language:PHP 1569

• roach-php / core

  The complete web scraping toolkit for PHP.

  crawlingphpweb-scraping
  Language:PHP 932

• iawia002 / Lulu

  [Unmaintained] A simple and clean video/music/image downloader 👾

  downloadervideopythonpython3crawlerscrapercrawlingscraping
  Language:Python 807

• scrapinghub / scrapyrt

  HTTP API for Scrapy spiders

  crawlercrawlinghacktoberfesthacktoberfest2021pythonscraperscrapytwistedwebcrawlerwebcrawling
  Language:Python 743

• clemfromspace / scrapy-selenium

  Scrapy middleware to handle javascript pages using selenium

  scrapyseleniumcrawling
  Language:Python 741

• MorvanZhou / easy-scraping-tutorial

  Simple but useful Python web scraping tutorial code.

  beautifulsoupscrapyregexurllibscrapingcrawlercrawlingrequestsasynciodistributed-scraper
  Language:Jupyter Notebook 668

• elixir-crawly / crawly

  Crawly, a high-level web crawling & scraping framework for Elixir.

  elixirerlangscraperscrapingscraping-websitesextract-dataspidercrawlercrawling
  Language:Elixir 582

• needleworm / bhban_rpa

  <6개월 치 업무를 하루 만에 끝내는 업무 자동화(생능출판사, 2020)>의 예제 코드입니다. 파이썬을 한 번도 배워본 적 없는 분들을 위한 예제이며, 엑셀부터 디자인, 매크로, 크롤링까지 업무 자동화와 관련된 다양한 분야 예제가 제공됩니다.

  rpaautomationcrawlingdesignpythoneducation
  Language:Python 571
• holiday-cn

  NateScarlet / holiday-cn

  📅🇨🇳中国法定节假日数据 自动每日抓取国务院公告

  chinacrawlingdataholidaynatural-language-processing
  Language:Python 533

• slotix / dataflowkit

  Extract structured data from web sites. Web sites scraping.

  golanggolang-libraryextract-datachrome-fetcherscraping-websitescrawlingscraperscrapingcdpgoheadless
  Language:Go 533

• essandess / isp-data-pollution

  ISP Data Pollution to Protect Private Browsing History with Obfuscation

  webcrawlingdataobfuscationprivacy-enhancing-technologiesdata-analyticsprivacy
  Language:Python 520

• edoardottt / cariddi

  Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

  endpointsendpoint-discoverybugbountyasset-finderendpointsecuritycrawlerendpoint-protectionsecret-keyssecrets-detectioninfosecreconnaissancereconcrawlinggolangpentestinggosecurityosintpenetration-testingsecurity-tools
  Language:Go 498

• zhuyingda / webster

  a reliable high-level web crawling & scraping framework for Node.js.

  scraping-frameworkcrawlercrawlingheadless-chromechromiumspiderautomation-uiautomation-testnodejsnodejs-frameworkjavascriptjavascript-frameworkpuppeteer
  Language:JavaScript 437

• scrapinghub / spidermon

  Scrapy Extension for monitoring spiders execution.

  scrapinghubscrapingmonitoringspiderscrawlingtestingmonitoring-toolhacktoberfest
  Language:Python 419
• mlscraper

  lorey / mlscraper

  🤖 Scrape data from HTML websites automatically by just providing examples

  crawlercrawler-pythoncrawlingextraction-enginehtmlmachine-learningscraperscraping
  Language:HTML 414

• Florents-Tselai / WarcDB

  WarcDB: Web crawl data as SQLite databases.

  clicrawlingdatabasesqlitewarcweb-archivingweb-data
  Language:Python 326

• stopstalk / stopstalk-deployment

  Stop stalking and start StopStalking :wink:

  web2pycompetitive-programmingpythonmaterializecsscrawlingcodechefcodeforcesspojhackerearthhackerrankuvaprogramming-contestsawstimusstopstalkatcoderhacktoberfest
  Language:Python 297

• rivermont / spidy

  The simple, easy to use command line web crawler.

  web-crawlerweb-spiderpythonpython3crawlingcrawler
  Language:Python 293

• infinitbyte / gopa

  GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

  spidercrawlerlightweightelasticsearchweb-crawlercrawlingweb-spiderweb-scrapingscraping
  Language:Go 291

• DarkSand / Sasila

  一个灵活、友好的爬虫框架

  pythonscrapingcrawlingframeworkcrawlerhttprequests
  Language:Python 290
• AdminHack

  mishakorzik / AdminHack

  today we will hack the admin panel of the site.

  termuxlinuxadmin-hackwebsitewebsitehackingadmin-website-hackwebsite-hackingwebsite-hacking-methodsadmin-panelcpanl-finderdirectory-bruteforcecpanelkali-linuxcrawlingadmin-finderhacking-toolallhackingtoolstermux-hackingtermux-tool
  Language:Shell 276

• alephdata / memorious

  Lightweight web scraping toolkit for documents and structured data.

  crawlingscrapingscraping-framework
  Language:Python 273
• linkedin-profile-scraper

  jvandenaardweg / linkedin-profile-scraper

  🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.

  puppeteernodejsscraperscrapingscraping-websiteswebsite-scraperjsonlinkedin-profile-scraperlinkedinscraperscrawlercrawlingspiderexpressjslinkedin-scraperlinkedin-scrapinglinkedin-botlinkedin-crawlerprofile-datalinkedin-profile
  Language:TypeScript 264

• mhmdiaa / second-order

  Second-order subdomain takeover scanner

  securitysecurity-toolswordlistwordlist-generatorpenetration-testingpentestinginfosecreconreconnaissancemappingweb-application-securitypenetration-testing-toolscrawlercrawling
  Language:Go 259

• mustafadalga / Instagram-Bot

  An Instagram bot developed using the Selenium Framework

  automationautomation-seleniumbotbulk-commentsbulk-unfollowcrawlercrawlingdownload-storiesinstagraminstagram-apiinstagram-botinstagram-downloaderinstagram-without-apimass-likingpythonpython3seleniumselenium-frameworkselenium-pythonselenium-webdriver
  Language:Python 253

• antchfx / antch

  Antch, a fast, powerful and extensible web crawling & scraping framework for Go

  golangweb-crawlercrawlingscrapingframeworkcrawlerweb-spider
  Language:Go 233