commoncrawl

There are 8 repositories under commoncrawl topic.

• news-please

  fhamborg / news-please

  news-please - an integrated web crawler and information extractor for news that just works

  news-crawlernews-extractorcrawlerextractornewsnews-websiteselasticsearchjsonpythonnlpdata-gatheringnews-archivenews-articlescommoncrawlextract-articlesextract-informationnews-scraperccnewscc-newsroberta
  Language:Python 2342

• commoncrawl / cc-pyspark

  Process Common Crawl data with Python and Spark

  sparkwarc-fileswetcommoncrawlsparksqlpysparkwat-filescommon-crawl
  Language:Python 442

• flairNLP / fundus

  A very simple news crawler with a funny name

  cc-newscommoncrawlcorpuscrawlernews-crawlernews-scrapingnlppythonrssscrapersitemaptext-extractionweb-corpusweb-scrapingcorpus-toolsdatasetsimage-classificationimage-extraction
  Language:Python 414

• commoncrawl / news-crawl

  News crawling with StormCrawler - stores content as WARC

  crawlernewswarcweb-crawlerapache-stormcommon-crawlcommoncrawlstorm-crawler
  Language:Java 358

• commoncrawl / cc-crawl-statistics

  Statistics of Common Crawl monthly archives mined from URL index files

  common-crawlcommoncrawlstatistics
  Language:Python 194

• uhussain / WebCrawlerForInflation

  Price Crawler - Tracking Price Inflation

  aws-athenasparkpython3s3-storagepandas-dataframeparquet-filescommoncrawlpetabytescalculate-inflation-ratesdash
  Language:Python 187

• cocrawler / cdx_toolkit

  A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine

  web-archivingweb-archiveswarccdxcdx-apicommoncrawlpython
  Language:Python 186

• oscar-project / ungoliant

  :spider: The pipeline for the OSCAR corpus

  common-crawlcommoncrawlcorpus-linguisticscrawlerfasttextlanguage-classificationnlposcar
  Language:Rust 171

• commoncrawl / cc-mrjob

  Demonstration of using Python to process the Common Crawl dataset with the mrjob framework

  pythonmap-reducehadoopcommoncrawl
  Language:Python 166

• karust / gogetcrawl

  Extract web archive data using Wayback Machine and Common Crawl

  golangcrawlercommoncrawlwayback-machinewebarchiveconcurrency
  Language:Go 161

• cloudtracer / paskto

  Paskto - Passive Web Scanner

  niktointernet-of-thingsscannerinternetarchivecommoncrawlpassive-vulnerability-scannerosint
  Language:JavaScript 152

• shjwudp / c4-dataset-script

  Inspired by google c4, here is a series of colossal clean data cleaning scripts focused on CommonCrawl data processing. Including Chinese data processing and cleaning methods in MassiveText.

  commoncrawldatasetmassivetextnlppythonspark
  Language:Python 131

• commoncrawl / cc-index-table

  Index Common Crawl archives in tabular format

  columnar-storagesqlsparkaws-athenacommoncrawlapache-parquet
  Language:Java 122

• commoncrawl / cc-webgraph

  Tools to construct and process Common Crawl webgraphs

  webgraph-frameworkcommon-crawlpagerankcentrality-measureswebgraphcommoncrawl
  Language:Java 101

• centic9 / CommonCrawlDocumentDownload

  A small tool which uses the CommonCrawl URL Index to download documents with certain file types or mime-types. This is used for mass-testing of frameworks like Apache POI and Apache Tika

  cdx-filescommoncrawlmime-typeswarcjava
  Language:Java 71

• generals-space / site-mirror-py

  [码云](https://gitee.com/generals-space/site-mirror-py) 通用爬虫, 仿站工具, 整站下载

  commoncrawlcrawlermirrorspider
  Language:Python 69

• commoncrawl / cc-notebooks

  Various Jupyter notebooks about Common Crawl data

  jupyter-notebookcommon-crawlaws-athenawebarchivingcommoncrawlwebgraph-framework
  Language:Jupyter Notebook 59

• CI-Research / KeywordAnalysis

  Word analysis, by domain, on the Common Crawl data set for the purpose of finding industry trends

  commoncrawlwordcountkeyword-extractioncluster-analysis
  57

• commoncrawl / cc-downloader

  A polite and user-friendly downloader for Common Crawl data

  commoncrawldownloaderrust
  Language:Rust 57

• rix4uni / uforall

  uforall is a fast url crawler this tool crawl all URLs number of different sources, alienvault,WayBackMachine,urlscan,commoncrawl

  bugbountycrawlerosintalienvaultcommoncrawlreconreconnaissanceurlscanwayback
  Language:Go 42

• commoncrawl / nutch

  Common Crawl fork of Apache Nutch

  big-datacommoncrawlhadoopjavaweb-crawler
  Language:Java 39

• commoncrawl / cc-warc-examples

  CommonCrawl WARC/WET/WAT examples and processing code for Java + Hadoop

  mapreducejavahadoopcommoncrawl
  Language:Java 37

• ChrisCates / CommonCrawler

  🕸 A simple way to extract data from Common Crawl

  golangcommoncrawl
  Language:Go 34

• Damian89 / commonCrawlParser

  Simple multi threaded tool to extract domain related data from commoncrawl.org

  commoncrawlosintpentesting
  Language:Python 33

• generals-space / site-mirror-go

  来自[码云](https://gitee.com/generals-space/site-mirror-go) 通用爬虫, 仿站工具, 整站下载

  commoncrawlcrawlermirrorspider
  Language:Go 31

• networkdynamics / seldonite

  A News Article Collection Library

  pythoneventsnewsnews-aggregatornews-articlesnlpcommoncrawlspark
  Language:Python 22

• cisnlp / GlotCC

  🕸 GlotCC Dataset and Pipline -- NeurIPS 2024

  glotcccommon-crawlcommoncrawlcorpus-linguisticscrawlerglotlanguage-identificationmultlingualmultilingual-datasetglotlid
  Language:Jupyter Notebook 20

• lxucs / commoncrawl-warc-retrieval

  Python tools to retrieve text from CommonCrawl WARC files based on cdx index.

  cdxcommoncrawltext-retrieval
  Language:Python 18

• imfht / super-Django-CC

  super-Django-CC is a simle web interface for commoncrawl.org

  commoncrawlsecurity-toolssubdomain-scanner
  Language:Python 13

• Tarasa24 / PWA-Store

  The largest collection of publicly accessible Progressive Web Apps*

  commoncrawlcrawleremrgolangmrjobpostgresqlpuppeteerpwalinode
  Language:HTML 13

• ahcm / tantivy_warc_indexer

  builds a tantivy index from common crawl warc.wet files

  commoncrawltantivysearchindex
  Language:Rust 12

• toimik / CommonCrawl

  Common Crawl's processing tools

  common-crawlcommon-crawl-datacommoncrawlwarcwarc-fileswatwat-fileswetwet-files
  Language:C# 11

• code402 / warc-benchmark

  Sample code to grep Common Crawl WARC files in Go, Java, Node and Python.

  commoncrawlwarccommon-crawl
  Language:Shell 8

• thunderpoot / cc-getpage

  Lightweight Python utility for retrieving individual pages from the Common Crawl archives.

  common-crawlcommon-crawl-datacommon-crawl-pythoncommon-crawl-with-pythoncommoncrawl
  Language:Python 6

• astralway / webindex

  Apache Fluo application that creates a web index using Common Crawl data

  fluoaccumulocommoncrawl
  Language:Java 4

• ngc7292 / query_of_cc

  This project is dataset and model checkpoints for the paper "Query of CC: Unearthing Large Scale Domain-Specific Knowledge from Public Corpora".

  commoncrawlknowledgelanguage-modelllmpilepre-training
  4