Hierarchical bert
lvaleriu opened this issue · comments
Hello! Thank you for starting this project.
I have a small question about the hierbert model (HierarchicalBert).
You use it to:
replace flat BERT encoder with hierarchical BERT encoder.
The hierarchy isnt about the labels/classes (classes could belong to a hierarchical tree), right? The hierarchy you mention is related to the text/token segments in a document, i.e you consider that a document is not only a big plaintext but a list of text segments and you give that information to the model?
Thank you for any information.
Hi @lvaleriu,
That's right. Hierarchical BERT (see the implementation of HierarchicalBertat https://github.com/coastalcph/lex-glue/blob/main/models/hierbert.py) considers a list of text segments (e.g., sentences, paragraphs). Each segment is initially parsed (encoded) on its own by the pre-trained model (namedencoder in the code) and then a second-order Transformer (named seg_encoder in the code) fuses segment encodings to produce a final document representation.
In the examined tasks, we have gold-standard factual paragraphs in the case of ECtHR A/B, and silver-standard new-line separated paragraphs in the case of SCOTUS.
Such questions should better be discussed in the Discussions section