readabilityBUNDLE

Main Content Extraction from html written in Java. It will extract the article text with out the around clutters.

Recent days its really a challenging open issue to extract the main article content from html pages. There are many open source algorithms / implementations available. What i aim in this project is concise some of the best content extraction algorithm implemented in JAVA.

My focus is mainly on the tuning parameters and customization / modifications of these algorithmic features according to my requirements.

readabilityBUNDLE will perform equally what other algorithms does plus below listed extras.

Whats extra in readabilityBUNDLE

Preserve the html tags in the extracted content.
Keep all the possible images in the content instead of finding best image.
Keep all the available videos.
Better extraction of li,ul,ol tags
Content normalization of extracted content.
Incorporated 3 best popular extraction algorithm , you can choose based on your requirement.
Provision to append next pages extracted content and create a consolidated output
Many cleaner / formatter measures added.
Some core changes in algorithms.

The main challenge which i was facing to extract the main content by keeping all the images / videos / html tags / and some realated div tags which are used as content / non content identification by most of the algorithms.

readabilityBUNDLE borrows much code and concepts from Project Goose , Snacktory and Java-Readability. My intension was just fine tune / modify the algorithm to work with my requirements.

Some html pages works very well in a particular algorithm and some not. This is the main reason i put all the available algorithm under a roof . You can choose an algorithm which best suits you.

You can see all author citations in each java file itself.

Dependency Projects

StringHelpers
Network
[NextPageFinder] (https://github.com/srijiths/NextPageFinder)

Usage

You need to say which extraction algorithm to use. The 3 extraction algorithms are ReadabilitySnack,ReadabilityCore and ReadabilityGoose. By default its ReadabilitySnack.

With out next page finding

Sample Usage

Article article = new Article();
ContentExtractor ce = new ContentExtractor();
HtmlFetcher htmlFetcher = new HtmlFetcher();
String html = htmlFetcher.getHtml("http://blogmaverick.com/2012/11/19/what-i-really-think-about-facebook/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+Counterparties+%28Counterparties%29", 0);

article = ce.extractContent(html, "ReadabilitySnack");

System.out.println("Content : "+article.getCleanedArticleText());

With next page html sources

If you need to extract and append content from next pages also then,

You can use [NextPageFinder] (https://github.com/srijiths/NextPageFinder) to find out all the next pages links.
Get the html of each next pages as a List of String using Network
Pass it to the content extractor like

article = ce.extractContent(firstPageHtml,extractionAlgorithm,nextPagesHtmlSources)

Build

Using Maven , mvn clean package

License

Apache License 2 - http://www.apache.org/licenses/LICENSE-2.0.html

masasdani / readabilityBUNDLE