commoncrawl
There are 6 repositories under commoncrawl topic.
- news-please
fhamborg / news-please
news-please - an integrated web crawler and information extractor for news that just works
Language:Python 1944 commoncrawl / cc-pyspark
Process Common Crawl data with Python and Spark
Language:Python 381News crawling with StormCrawler - stores content as WARC
Language:Java 251michaelharms / comcrawl
A python utility for downloading Common Crawl data
Language:Python 212uhussain / WebCrawlerForOnlineInflation
Price Crawler - Tracking Price Inflation
Language:Python 170Demonstration of using Python to process the Common Crawl dataset with the mrjob framework
Language:Python 166cocrawler / cdx_toolkit
A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine
Language:Python 153cloudtracer / paskto
Paskto - Passive Web Scanner
Language:JavaScript 151oscar-project / ungoliant
:spider: The pipeline for the OSCAR corpus
Language:Rust 150karust / gogetcrawl
Extract web archive data using Wayback Machine and Common Crawl
Language:Go 130Statistics of Common Crawl monthly archives mined from URL index files
Language:Python 117shjwudp / c4-dataset-script
Inspired by google c4, here is a series of colossal clean data cleaning scripts focused on CommonCrawl data processing. Including Chinese data processing and cleaning methods in MassiveText.
Language:Python 103Index Common Crawl archives in tabular format
Language:Java 95Tools to construct and process webgraphs from Common Crawl data
Language:Java 64centic9 / CommonCrawlDocumentDownload
A small tool which uses the CommonCrawl URL Index to download documents with certain file types or mime-types. This is used for mass-testing of frameworks like Apache POI and Apache Tika
Language:Java 58CI-Research / KeywordAnalysis
Word analysis, by domain, on the Common Crawl data set for the purpose of finding industry trends
generals-space / site-mirror-py
[码云](https://gitee.com/generals-space/site-mirror-py) 通用爬虫, 仿站工具, 整站下载
Language:Python 56- Language:Python 45
Various Jupyter notebooks about Common Crawl data
Language:Jupyter Notebook 40CommonCrawl WARC/WET/WAT examples and processing code for Java + Hadoop
Language:Java 38- Language:Go 33
Damian89 / commonCrawlParser
Simple multi threaded tool to extract domain related data from commoncrawl.org
Language:Python 30- Language:Java 25
来自[码云](https://gitee.com/generals-space/site-mirror-go) 通用爬虫, 仿站工具, 整站下载
Language:Go 25networkdynamics / seldonite
A News Article Collection Library
Language:Python 19lxucs / commoncrawl-warc-retrieval
Python tools to retrieve text from CommonCrawl WARC files based on cdx index.
Language:Python 17imfht / super-Django-CC
super-Django-CC is a simle web interface for commoncrawl.org
Language:Python 13- Language:HTML 11
ahcm / tantivy_warc_indexer
builds a tantivy index from common crawl warc.wet files
Language:Rust 8toimik / CommonCrawl
Common Crawl's processing tools
Language:C# 5- Language:Java 4
code402 / warc-benchmark
Sample code to grep Common Crawl WARC files in Go, Java, Node and Python.
Language:Shell 4ArtificialOSS / WebCrawl
Crawls the web to generate a huge dataset for training
Language:Python 3ngc7292 / query_of_cc
This project is dataset and model checkpoints for the paper "Query of CC: Unearthing Large Scale Domain-Specific Knowledge from Public Corpora".
- Language:PHP 3
BhagyashriT / DICLAB2-DataAggregationBigDataAnalysisAndVisualization
Collected data about from three sources, one opinion-based social media in twitter, research data in New York Times, and the third is the common crawl data for the same topic or key phrase, and from similar time periods. Processed the three data sets collected individually using classical big data methods like Map Reduce in Google Dataproc Clusters. And then compared the outcomes using popular visualization methods in tableau.
Language:Python 2
