web-crawling

There are 24 repositories under web-crawling topic.

• crawlee

  apify / crawlee

  Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

  web-scrapingweb-crawlingnpmheadless-chromepuppeteerautomationapifyscrapingcrawlingcrawlerheadlessscraperweb-crawlerjavascriptnodejsplaywrighttypescript
  Language:TypeScript 20477

• apify / crawlee-python

  Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

  apifyautomationbeautifulsoupcrawlercrawlinghacktoberfestheadlessheadless-chromepipplaywrightpythonscraperscrapingweb-crawlerweb-crawlingweb-scraping
  Language:Python 7129
• botasaurus

  omkarcloud / botasaurus

  The All in One Framework to Build Undefeatable Scrapers

  anti-botanti-detectioncloudflare-bypasscloudflare-scrapeanti-detectanti-detect-browserantidetect-browserundetectedundetected-chromedriverbypass-cloudflarepython-web-scraperpython-web-scrapingscraping-frameworkscraping-toolundetectableweb-scraping-pythonbot-detectionscraping-pythonweb-crawlingpython-scraper
  Language:Python 3189

• brightdata / brightdata-mcp

  A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access.

  ai-agentsai-integrationsanti-bot-detectionbrowser-automationdata-collectiondata-extractionllmmcpmcp-servermodelcontextprotocolscrapingscraping-toolsstructured-dataweb-crawlingweb-dataweb-scraping
  Language:JavaScript 1556

• cxcscmu / Craw4LLM

  Official repository for "Craw4LLM: Efficient Web Crawling for LLM Pretraining"

  crawlercrawlinglarge-language-modelsllmpre-trainingpretrainingweb-crawlerweb-crawling
  Language:Python 642

• scrapehero-code / amazon-scraper

  A simple web scraper to extract Product Data and Pricing from Amazon

  amazon-scraperpage-scraperscrape-productsweb-scrapingweb-scraping-tutorialsweb-crawling
  Language:Python 415
• crawler

  crwlrsoft / crawler

  Library for Rapid (Web) Crawler and Scraper Development

  crawlingphpscraperscrapingscraping-websitesweb-crawlerweb-crawlingweb-scrapinghacktoberfestcrawlerweb-scraper
  Language:PHP 366

• spyboy-productions / omnisci3nt

  Omnisci3nt – See What They’ve Tried to Hide Extract deep intelligence from any domain. From subdomains to SSL certs, archived secrets to exposed ports — Omnisci3nt gives you the full picture in seconds.

  dns-enumerationip-lookupport-scanningssl-certificatesubdomain-enumerationtechnology-analysisweb-crawlingweb-reconnaissancewhoisdmarc-record-examinationreconnaissance-toolsocial-media-and-email-discoverywayback-machine-accessdirectory-enumerationosintadmin-login-finderadmin-panel-finderadmin-panel-finder-of-any-websitewebsite-hackingvulnerability-scanner
  Language:Python 306

• godkingjay / selenium-twitter-scraper

  This is a Twitter Scraper which uses Selenium for scraping tweets. It is capable of scraping tweets from home, user profile, hashtag, query or search, and advanced searches.

  scraperselenium-scrapertwittertwitter-scraperweb-crawlinghacktoberfesthacktoberfest-acceptedcollaborateselenium
  Language:Jupyter Notebook 292

• jrbadiabo / Bet-on-Sibyl

  Machine Learning Model for Sport Predictions (Football, Basketball, Baseball, Hockey, Soccer & Tennis)

  machine-learningsportsanalyticssports-statsmachine-learning-algorithmspredictive-analysisalgorithmsseleniumbeautifulsouppythonpython-2scikit-learnweb-scrapingweb-crawlingmachinelearning
  Language:Jupyter Notebook 275
• InfinityCrawler

  TurnerSoftware / InfinityCrawler

  A simple but powerful web crawler library for .NET

  crawlerrobots-txtspiderweb-crawlerweb-crawling
  Language:C# 252
• ayakashi

  ayakashi-io / ayakashi

  :zap: Ayakashi.io - The next generation web scraping framework

  web-scrapingautomationheadless-chromedata-miningweb-crawling
  Language:TypeScript 215
• clauneck

  serpapi / clauneck

  A tool for scraping emails, social media accounts, and much more information from websites using Google Search Results.

  automationcommand-linecommand-line-tooldata-extractiondata-extractoremailemail-extract-with-proxyemail-extractionemail-extractoremail-marketingemail-scraperopen-sourcerubyrubygemserpsocial-media-scraperweb-crawlingwebscraping
  Language:Ruby 187

• scrapinghub / scrapy-training

  Scrapy Training companion code

  scrapypythontrainingweb-scrapingweb-crawling
  Language:Python 173

• brianmadden / krawler

  A web crawling framework written in Kotlin

  webcrawlerkotlinframeworkcrawler4jlink-checkerweb-crawlerweb-crawling
  Language:Kotlin 131

• MaxValue / Terpene-Profile-Parser-for-Cannabis-Strains

  Parser and database to index the terpene profile of different strains of Cannabis from online databases

  cannabisdata-scienceweb-crawler-pythonweb-crawlerweb-crawlingpython-3terpenesplantsbiological-data-analysisbiological-datascrapyhealthcannabis-strainscrawlerpythonbioinformaticsanalysisdatabasearomatherapyterpene-profile
  Language:Python 127

• leogregianin / bancocentralbrasil

  💵 💰 :brazil: Informações sobre taxas oficiais diárias de Inflação, Selic, Poupança, Dólar, Dólar PTAX, Euro e Euro PTAX pelo site do Banco Central do Brasil

  banco-centralmoneyweb-scrapingweb-crawlingbrasilbrazil
  Language:Python 126

• my8100 / scrapyd-cluster-on-heroku

  Set up free and scalable Scrapyd cluster for distributed web-crawling with just a few clicks. DEMO :point_right:

  scrapyscrapydclusterherokupythonscrapydweblogparserweb-crawlingweb-scraping
  Language:Python 122

• maxmindlin / scout-lang

  A web crawling programming language

  dslprogramming-languageweb-crawlingweb-scrapingscraperscrapingscraping-websites
  Language:Rust 116

• SoheilKhodayari / JAW

  JAW: A Graph-based Security Analysis Framework for Client-side JavaScript

  client-sidecsrfjavascriptneo4jproperty-graphstatic-analysisvulnerability-detectionweb-crawling
  Language:JavaScript 112

• ScrapingAnt / amazon_scraper

  Amazon products scraper with using of rotating proxies and headless Chrome from ScrapingAnt

  scrapingscraping-apiscraping-websitesscraping-webscraping-pythonscraping-dataprice-scrapingprice-scraperweb-crawlerweb-crawlingweb-crawlersamazonamazon-scraperamazon-scraping-libraryscraperdata-miningnode-jsjsscrape-products
  Language:JavaScript 87

• jonasjacek / robots.txt

  Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.

  googlebotbingbotrobots-txtrobots-exclusion-standardblocking-botsuser-agentweb-robotsseosearch-enginewhitelistcrawlersweb-crawlingcrawlingsearch-engine-optimizationbaiduspidertwitterbot
  84

• alyakhtar / Katastrophe

  Command Line Tool to download torrents

  screenshotdelugebittorrenttorrentkickass-torrentscommand-linepythonweb-crawling
  Language:Python 83

• spyboy-productions / PhantomCrawler

  Boost website hits by generating requests from multiple proxy IPs.

  ddos-attack-toolsproxyproxy-configurationproxy-rotationweb-crawlingweb-scrappingwebsite-analyticswebsite-hits
  Language:Python 75

• sushantPatrikar / Amazon-Flipkart-Price-Comparison-Engine

  Compares price of the product entered by the user from e-commerce sites Amazon and Flipkart :moneybag: :bar_chart:

  flipkartecommerce-sites-amazoncorresponding-pricesamazonweb-crawlingweb-crawler-pythonpythonpython3python-3tkinter
  Language:Python 68

• GoTrained / Scrapy-Craigslist

  Web Scraping Craigslist's Engineering Jobs in NY with Scrapy

  pythonscrapyweb-scrapingweb-crawlingscrapy-crawlerscrapy-spiderscrapy-tutorialweb-scrapercraigslist
  Language:Python 66

• dongweiming / daenerys

  Scraping and Web Crawling Framework For Zhihu Live

  zhihuzhihuliveweb-crawlingscraping
  Language:Python 63

• jgujerry / python-frameworks

  Another curated list of Python frameworks

  aiai-agentai-agent-frameworkapiartificial-intelligencecmsdata-workflowdeep-learningdevopsdistributed-computingenterprise-integrationsframeworksmachine-learningmessagingparallel-computingpipelinepythontask-queueweb-crawlingwebapp
  Language:HTML 59

• MohamedHmini / tweetsOLAPing

  implementing an end-to-end tweets ETL/Analysis pipeline.

  datawarehousingdatawarehouseetl-pipelinetweetstweets-classificationtweets-scrapertwitter-apigoogle-api-clientapi-clientweb-crawlinganalysiscube-analysismulti-dimensional-analysisssisssas-multidimensionalmultithreadingpowerbi-report
  Language:Python 58

• ScaleUnlimited / flink-crawler

  Continuous scalable web crawler built on top of Flink and crawler-commons

  web-crawlerweb-crawlingcrawlercrawlingspiderflink
  Language:Java 52

• mike-gee / webtranspose

  Web scraping API for building AI applications.

  chatbotscrawlingcrawling-pythonpythonscrapingscraping-pythonweb-crawlingweb-scrapingweb-scraping-python
  Language:Python 40

• ScrapingAnt / zoominfo_scraper

  Zoominfo scraper with using of rotating proxies and headless Chrome from ScrapingAnt

  scrapingscraping-apiscraping-websitesscraping-datascraping-toolpythonweb-harvestingweb-crawlerweb-crawlingweb-crawler-pythonscraperzoominfo-clientdataminingleadgenleadgeneration
  Language:Python 34

• Cheng-Lin-Li / KnowledgeGraph

  This repository for Web Crawling, Information Extraction, and Knowledge Graph build up.

  cdrconditionalconditional-random-fieldscrfsuitefacebook-crawlerfacebook-graph-apiinformation-extractionjsonlinesknowledge-graphpythonpython3web-crawling
  Language:Julia 33

• chrislicodes / udacity-data-analyst-nanodegree

  Repository for the projects needed to complete the Data Analyst Nanodegree.

  udacitydatadata-analysisdata-visualizationdatasetdata-cleaningdata-analyticsdata-analyst-nanodegreestatisticsdata-wranglingdata-gatheringweb-crawlingapitext-miningpandasnumpymatplotlibseaborntweepy
  Language:Jupyter Notebook 33

• HRN-Projects / amazon-captcha-solver

  A TensorFlow (Deep Learning - CNN) based solution for tackling captcha when collecting data from Amazon.

  captchaamazon-captchacaptcha-solvingcaptcha-solverpythonpython3tensorflowkerasopen-cvhrn-projectsweb-scrapingweb-scraping-solutionweb-crawlingamazon-captcha-solveramazon-captcha-solvingapiflask-apicaptcha-solver-apicaptcha-images
  Language:Python 31

• zytedata / spidyquotes

  Example site for web scraping tutorials

  scrapingcrawlingtutorialsweb-scraping-tutorialsweb-scrapingweb-crawlingplayground
  Language:Julia 31