common-crawl
There are 9 repositories under common-crawl topic.
- StringZilla
ashvardanian / StringZilla
Up to 10x faster strings for C, C++, Python, Rust, and Swift, leveraging SWAR and SIMD on Arm Neon and x86 AVX2 & AVX-512-capable chips to accelerate search, sort, edit distances, alignment scores, etc 🦖
Language:C++ 1827 commoncrawl / cc-pyspark
Process Common Crawl data with Python and Spark
Language:Python 382News crawling with StormCrawler - stores content as WARC
Language:Java 253michaelharms / comcrawl
A python utility for downloading Common Crawl data
Language:Python 215oscar-project / ungoliant
:spider: The pipeline for the OSCAR corpus
Language:Rust 151crissyfield / troll-a
Drill into WARC web archives
Language:Go 130Statistics of Common Crawl monthly archives mined from URL index files
Language:Python 118An asynchronous concurrent pipeline for classifying Common Crawl based on fastText's pipeline.
Language:Go 85Tools to construct and process webgraphs from Common Crawl data
Language:Java 65Various Jupyter notebooks about Common Crawl data
Language:Jupyter Notebook 40- Language:Java 28
bminixhofer / gerpt2
German small and large versions of GPT2.
Language:Python 18The website of the Oscar Project
Language:TeX 10Mgosi / Big-Data-Analysis-using-MapReduce-in-Hadoop
We explore data by using Big Data Analysis and Visualization skills. To obtain this, we perform 3 main operations. i.e. i)Data Aggregation through different sources. ii) Big Data Analysis using MapReduce and iii) Visualization through Tableau. Data Analysis is very critical in understanding the data, and what we can do with the data. For small datasets it is easier to process and obtain the results. But as for big companies, it becomes crucial for them to obtain the trends of the company for any changes need to be made. Hence we introduce Big Data Analysis to solve this problem. In this lab, we collect close to 20000 tweets, 500 articles on New York Times and 500 articles on Common Crawl Data about Entertainment, which is our main topic of discussion. Using this data, we perform preprocessing and feed it to a MapReduce to find the Word Count and Word Co-Occurrence. Using this, we find the trend of the data collected in this topic. We have used Python to perform Data Analysis.Data Analysis is very critical in understanding the data, and what we can do with the data. For small datasets it is easier to process and obtain the results. But as for big companies, it becomes crucial for them to obtain the trends of the company for any changes need to be made. Hence we introduce Big Data Analysis to solve this problem. In this lab, we collect close to 20000 tweets, 500 articles on New York Times and 500 articles on Common Crawl Data about Entertainment, which is our main topic of discussion. Using this data, we perform preprocessing and feed it to a MapReduce to find the Word Count and Word Co-Occurrence. Using this, we find the trend of the data collected in this topic. We have used Python to perform Data Analysis.
Language:Jupyter Notebook 8- Language:R 6
HRN-Projects / common_crawl_with_scrapy
Parsing Huge Web Archive files from Common Crawl data index to fetch any required domain's data concurrently with Python and Scrapy.
Language:Python 6tokenmill / common-crawl-utils
Various Common Crawl utilities in Clojure.
Language:Clojure 6alumik / common-crawl-downloader
Distributed download scripts for Common Crawl data
Language:Python 5toimik / CommonCrawl
Common Crawl's processing tools
Language:C# 5code402 / warc-benchmark
Sample code to grep Common Crawl WARC files in Go, Java, Node and Python.
Language:Shell 4connor-marchand / gau-python
This library gets urls from AlienVault's Open Threat Exchange, the Wayback Machine, and Common Crawl. Inspired by Corbin Leo's gau
Language:Python 2- Language:Python 2
socket-var / nyt-twitter-cc-hadoop
Perform big data analysis on New york times, Twitter and Common Crawl APIs
Language:Jupyter Notebook 2fizerkhan / cdx-index-client
A command-line tool for using CommonCrawl Index API at http://index.commoncrawl.org/
Language:Python 1A small tool which uses the CommonCrawl URL Index to download documents with certain file types or mime-types for mass-testing of frameworks like Apache POI and Apache Tika
Language:Java 1Word analysis, by domain, on the Common Crawl data set for the purpose of finding industry trends
Language:Python 1neil-zt / common-crawl-client
A Common Crawl client example for scraping specific websites.
Language:Jupyter Notebook 1bottomless-archive-project / common-crawl-client
This library is a very lightweight client to Common Crawl's WARC files.
Language:Java 0An application that crawls the Common Crawl corpus for URLs with the specified file extensions.
Language:Java 0Dahouabdelhalim / Discourse-marksers-and-Web-crawling
Discourse Markers identification in French Language
Language:HTML 0hadrianw / abracabra
Eventually a search engine, but currently a filtering pipeline for HTML and soon WARC files.
Language:Rust 0- Language:Java 0
ErikGartner / prometheus-cc-extractor
This repository contains mapreduce extractors to preprocess and extract websites from the common crawl corpus.
Language:Pythonskyler-myers-db / Common-Crawl-Analysis
Parsing the common crawl database using Scala and Spark
Language:Scalasrmocher / fake-science
Analyzing Common Crawl data (specifically) to classify fake/real based on trained deep learning models (LSTM, CNN)
Language:PythonVikasg7 / warc-reader
ES6 Class to read .warc or .warc.gz file member by member in nodejs
Language:TypeScript
