Library does not produce same results as http://boilerpipe-web.appspot.com/
GoogleCodeExporter opened this issue · comments
Google Code Exporter commented
1) Go to http://boilerpipe-web.appspot.com/
2) Type in http://arstechnica.com/ as the URL.
3) Use article extractor and HTML (extract fragment)
4) See a nice list of articles on that page
Compare to:
1) Download latest boilerpipe svn.
2) Use the following code:
final URL url = new URL("http://arstechnica.com/");
final ArticleExtractor articleExtractor = ArticleExtractor.INSTANCE;
final HTMLHighlighter htmlHighlighter = HTMLHighlighter.newExtractingInstance();
final String xhtml = htmlHighlighter.process(url, articleExtractor);
3) xhtml only contains 1 article.
Are there settings that need to be changed? Or is there a code update that
hasn't been checked in?
Original issue reported on code.google.com by paul.jak...@gmail.com
on 30 Mar 2012 at 2:50
Google Code Exporter commented
It looks like the issue is the KeepLargestBlockFilter which rejects every block
except the largest. While taking out this filter in the library should return
results closer to http://boilerpipe-web.appspot.com/, it looks like that will
cause some non-article things that are rejected by
http://boilerpipe-web.appspot.com/ to make it through.
Is there anyplace I can go to see what filters are being by the article
extractor used by http://boilerpipe-web.appspot.com/?
Original comment by paul.jak...@gmail.com
on 30 Mar 2012 at 4:46
Google Code Exporter commented
I'm also unable to get the same results using the HTMLHighlighter in extraction
mode. The web API (http://boilerpipe-web.appspot.com) clearly states that:
"This Web Application probably uses a more recent version than the released
versions in the Boilerpipe Google Code page. You might thus get slightly
different (hopefully better) results."
In the web demo, are you using the HTMLHighligher to extract HTML, or have you
added a different approach to boilerpipe-core? If changes have been made to
core, I'd be happy to contribute to the open source version. Hints would be
appreciated =)
Original comment by matth...@linksmart.com
on 19 Oct 2012 at 5:49
Google Code Exporter commented
I got the similar issue.
When trying the URL "http://www.hokkaido-np.co.jp/news/donai/424760.html"
With ArticleExtractor and "Plain Text" output
Library code did not produce same results as http://boilerpipe-web.appspot.com/
Original comment by jc...@geosemble.com
on 6 Dec 2012 at 12:02
Google Code Exporter commented
Building from source will fix this issue.
Original comment by pettib...@gmail.com
on 24 Jan 2013 at 8:33
Google Code Exporter commented
I have done a build from SVN. Still Japanese URL gives bad results. Same works
gr8 on the Web App. Code snippet below:
URL url = new URL("http://d.hatena.ne.jp/mkusunok/20130817/p1");
String text = ArticleExtractor.INSTANCE.getText(url);
Original comment by navra...@gmail.com
on 7 Oct 2013 at 7:24