Giters

jessevig / bertviz

BertViz: Visualize Attention in NLP Models (BERT, GPT2, BART, etc.)

Home Page:https://towardsdatascience.com/deconstructing-bert-part-2-visualizing-the-inner-workings-of-attention-60a16d86b5c1

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Support for BigBird

iamakshay1 opened this issue · comments

commented

Hi, When I am trying to run this for the BigBird model which is trained on Genomic Data, I am getting the following error: ValueError: Attention has 1024 positions, while the number of tokens is 976 for tokens
image

commented

Hi, I'm able to run a BigBird model in the trivial case:

from transformers import AutoTokenizer, AutoModel
from bertviz import model_view

model_name = 'google/bigbird-roberta-base'
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name, output_attentions=True)
inputs = tokenizer.encode("The cat sat on the mat", return_tensors='pt')

outputs = model(inputs)
attention = outputs[-1]  # Output includes attention weights when output_attentions=True
tokens = tokenizer.convert_ids_to_tokens(inputs[0]) 
model_view(attention, tokens)
commented

Is this a publicly available model that you're working with? If so, would you mind sharing the name and associated code so that I can debug on my end? Thanks!