The Archives Unleashed Toolkit is an open-source platform for analyzing web archives. Tight integration with Hadoop provides powerful tools for analytics and data processing via Apache Spark. Our full documentation can be found at http://archivesunleashed.org/aut/.
The Archives Unleashed Toolkit is part of the Archives Unleashed Project.
You can download the latest release here, build it yourself as per the instructions below, or use Docker.
Clone the repo:
$ git clone http://github.com/archivesunleashed/aut.git
You can then build The Archives Unleashed Toolkit.
$ mvn clean install
For the impatient, to skip tests:
$ mvn clean install -DskipTests
Once built or downloaded, you can follow the basic set of tutorials here.
Licensed under the Apache License, Version 2.0.
This work is primarily supported by the Andrew W. Mellon Foundation. Additional funding for the Toolkit has come from the U.S. National Science Foundation, Columbia University Library's Mellon-funded Web Archiving Incentive Award, the Natural Sciences and Engineering Research Council of Canada, the Social Sciences and Humanities Research Council of Canada, and the Ontario Ministry of Research and Innovation's Early Researcher Award program. Any opinions, findings, and conclusions or recommendations expressed are those of the researchers and do not necessarily reflect the views of the sponsors.