Web2Text

Source code for Web2Text: Deep Structured Boilerplate Removal, full paper at ECIR '18

Introduction

This repository contains

Scala code to parse an (X)HTML document into a DOM tree, convert it to a CDOM tree, interpret tree leaves as a sequence of text blocks and extract features for each of these blocks.
Python code to train and evaluate unary and pairwise CNNs on top of these features. Inference on the hidden Markov model based on the CNN output potentials can be executed using the provided implementation of the Viterbi algorithm.
The CleanEval dataset under src/main/resources/cleaneval/:
- orig: raw pages
- clean: reference clean pages
- aligned: clean content aligned with the corresponding raw page on a per-character basis using the alignment algorithm described in our paper
Output from various other webpage cleaners on CleanEval under other_frameworks/output:
- Body Text Extractor (Finn et al., 2001)
- Boilerpipe (Kohlschütter et al., 2010): default-extractor, article-extractor, largestcontent-extractor
- Unfluff (Geitgey, 2014)
- Victor (Spousta et al., 2008)

Installation

Install Scala and SBT. The code was tested with SBT 0.31. You can also use Docker image hseeberger/scala-sbt:8u222_1.3.3_2.13.1.
Install Python 3 with Tensorflow and NumPy.

Usage

Recipe: extracting text from a web page

Run ch.ethz.dalab.web2text.ExtractPageFeatures through sbt. The arguments are:
- input html file
- the desired output base filename (script produces {filename_base}_edge_feature.csv and {filename_base}_block_features.csv)
Use the python script src/main/python.py with the 'classify' option. The arguments are:
- python3 main.py classify {filename_base} {labels_out_filename}
Use ch.ethz.dalab.web2text.ApplyLabelsToPage through sbt to produce clean text. Arguments:
- input html file
- {labels_out_filename} from step 2
- output destination text file path

HTML to CDOM

In Scala:

import ch.ethz.dalab.web2text.cdom.CDOM
val cdom = CDOM.fromHTML("""
    <body>
        <h1>Header</h1>
        <p>Paragraph with an <i>Italic</i> section.</p>
    </body>
    """)
println(cdom)

Feature extraction

Example:

import ch.ethz.dalab.web2text.features.{FeatureExtractor, PageFeatures}
import ch.ethz.dalab.web2text.features.extractor._

val unaryExtractor = 
    DuplicateCountsExtractor
    + LeafBlockExtractor
    + AncestorExtractor(NodeBlockExtractor + TagExtractor(mode="node"), 1)
    + AncestorExtractor(NodeBlockExtractor, 2)
    + RootExtractor(NodeBlockExtractor)
    + TagExtractor(mode="leaf")

val pairwiseExtractor = 
    TreeDistanceExtractor + 
    BlockBreakExtractor + 
    CommonAncestorExtractor(NodeBlockExtractor)

val extractor = FeatureExtractor(unaryExtractor, pairwiseExtractor)

val features: PageFeatures = extractor(cdom)

println(features)

Aligning cleaned text with original source

import ch.ethz.dalab.web2text.alignment.Alignment
val reference = "keep this"
val source = "You should keep this text"
val alignment: String = Alignment.alignment(source, reference) 
println(alignment) // □□□□□□□□□□□keep this□□□□□

Extracting features for CleanEval

import ch.ethz.dalab.web2text.utilities.Util
import ch.ethz.dalab.web2text.cleaneval.CleanEval
import ch.ethz.dalab.web2text.output.CsvDatasetWriter

val data = Util.time{ CleanEval.dataset(fe) }

// Write block_features.csv and edge_features.csv
// Format of a row: page id, groundtruth label (1/0), features ...
CsvDatasetWriter.write(data, "./src/main/python/data")

// Print the names of the exported features in order
println("# Block features")
fe.blockExtractor.labels.foreach(println)
println("# Edge features")
fe.edgeExtractor.labels.foreach(println)

Training the CNNs

Code related to the CNNs lives in the src/main/python directory.

To train the CNNs:

Set the CHECKPOINT_DIR variable in main.py.
Make sure the files block_features.csv and edge_features.csv are in the src/main/python/data directory. Use the example from the previous section for this.
Convert the CSV files to .npy with data/convert_scala_csv.py.
Train the unary CNN with python3 main.py train_unary.
Train the pairwise CNN with python3 main.py train_edge.

Evaluating the CNN

To evaluate the CNN:

Set the CHECKPOINT_DIR variable in main.py to point to a directory with trained weights. We provide trained weights based on the cleaneval split and a custom web2text split (with more training data.)
Run python3 main.py test_structured to test performance on the CleanEval test set.

The performance of other networks is computed in Scala:

import ch.ethz.dalab.web2text.Main
Main.evaluateOthers()

Raidus / web2text