RominYue / warc

Python library for reading and writing warc files (Especially for Clueweb/Trec Datasets)

warc: Python library to work with WARC files (Especially for Clueweb Datasets)

Introduction

ClueWeb09 and ClueWeb12 dataset consists of ~1 billion web pages crawled in January and February 2009 and 2012.

It is used by several tracks of the TREC conference.

Notification

This library is a fork of internetarchive/warc cdegroc/warc-clueweb that can handle both ClueWeb09 and ClueWeb12 dataset.

Why I change the code?

1. there are certain caveats with ClueWeb09's files. it does not use the standard \r\n end-of-line markers. Moreover, some records are ill-formed.
2. cdegroc/warc-clueweb This fork repo could not handle Clueweb12 dataset and another issue is that it could miss 1-2 document when dealiing every .warc.gz file

Documentation

Only minor modifications to the original library were made.

The original documentation of the warc library still holds.

The documentation of the warc library is available at http://warc.readthedocs.org/.

How to use

For simply analyze warc format files such as clueweb09 and clueweb12, you can just use warc/warc.py

For other usage, just reference original document.

License

This software is licensed under GPL v2. See LICENSE file for details.