[Question] Best practice to cleanup HTML Spaghetti code
JpEncausse opened this issue · comments
Question or comment
I need to cleanup a random HTML page to extract readable content. Modern website use A LOT of Spaghetti HTML. For instance :
<div> <div><div> <a href="/"> <div>Title of the site</div> </a> <div lazyload="event"><!--lazy <div class="headerPageHtml"><a href="/include/news.xml" target="_blank"><img id="socialRss" alt="access to rss" src="/asset/social/rss.png"></a></div></div></div>
In this example I don't want all the <div> and the formating <div><img></div>
Should I strip all the
It's unclear what you're trying to do exactly, but sanitize-html is quite good at keeping only the tags and attributes you approve, as you can see in the documentation. If you want to do more subtle things, there are transformation features. If your needs exceed that, then you might consider using sanitize-html as a first pass and then cheerio for the transformations.
I came here to report a similar issue. An unclosed attribute (missing final double-quote) will cause everything from the start of that tag through to the end of the input to be stripped by sanitize-html.
// ↓ Missing double-quote
sanitize(`Hello, world. <a href="/this>this</a> is a demo of this behavior. <b>I won't be in the output!</b>`)
// => 'Hello, world. 'Angle brackets are not forbidden in quoted HTML attributes, and in fact this document produces the expected title on hover in Chrome:
<h4 title="this is a title<containing><punctuation>">h4 body</h4>
If both the standard and actual browsers permit it then sanitize-html can't reliably detect that it is "wrong" (because it isn't, strictly speaking). Also this behavior is coming from the htmlparse2 module in any case, but keep in mind it is not a bug before reporting anything there.
@boutell Got it. So in the case of invalid HTML (the double quote never closes anywhere), is there any way to get an error back instead of having large portions of the input stripped out?
Okay, I think we can live with that for now. Thank you!