Giters

jessevig / bertviz

BertViz: Visualize Attention in NLP Models (BERT, GPT2, BART, etc.)

Home Page:https://towardsdatascience.com/deconstructing-bert-part-2-visualizing-the-inner-workings-of-attention-60a16d86b5c1

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Selecting multiple tokens at once.

giulio98 opened this issue · comments

commented

Hello BERTViz Maintainers,

Firstly, I'd like to express my appreciation for the work you've done with BERTViz.

I am currently exploring attention patterns in BERT and would like to enhance my analysis by visualizing the combined attention that multiple tokens pay to all other tokens in the sequence. Specifically, I'm interested in visualizing the sum of the attention scores for selected tokens to see how they jointly attend to other parts of the input.

From my understanding, the current implementation of BERTViz allows for attention visualization on a per-token basis. I am looking for guidance on how to modify BERTViz to support the following:

  1. Selecting multiple tokens simultaneously.
  2. Summing the attention scores across these selected tokens.
  3. Visualizing the cumulative attention in the UI.

As an example, see the figure attached, the visualization shows the attention score for a single token ("What"), however, I would like to have the possibility to interactively select multiple tokens (e.g "Capital" and "France") and visualize the sum of the attention scores
image

If this feature is not already available, could you provide some pointers on how to implement this with minimal code changes? Any advice on where to start, which parts of the code to focus on, or a high-level approach would be greatly appreciated.

Thank you for your time and assistance.