Logits are changing in old adapters-transformer models if used by the new library
Ch-rode opened this issue · comments
Environment info
adapters
version: 0.1.1- Platform: python
- Python version: 3.9.18
- PyTorch version (GPU?):1.10.2 yes gpu
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no
Information
Hello ! I was trying to use an old adapters model i trained last year to do some test using the new adapters library. Unfortunately i noticed that every time i re-open the run, for the same sequence I have different results. This is not happening for models trained last week.
To reproduce
model = BertAdapterModel.from_pretrained(model_name)
model.load_adapter(adapter_path, with_head=True)
model.load_head(adapter_path )
model.set_active_adapters(['x_adapter'])
model.active_head = 'xadapter_head'
model.train_adapter(["xadapter"])
model.delete_head('default')
model.bert.prompt_tuning = nn.Identity()
model = model.to(self.device)
input_texts = [" ".join("".join(sample.split())) for sample in input_texts]
batch_input = ['MERLFPRRFFISQKLTRKKLQKSVGQTANDFVYISEDR']
encoded = tokenizer(batch_input, truncation=True, padding=True, max_length=512, return_tensors="pt").to(self.device)
y_preds =model(**encoded).logits
##run1
[-0.0135214077308774, -0.00513207633048296]
##run2
[0.07941669225692749, 0.0340627059340477]
Expected behavior
I check hundred times and with the same code 'new recent' models are working, old one no. thanks a lot, i really need to make it work !!
Hey @Ch-rode , the inconsistency between adapters trained with the old vs the new library sounds like something we should look into. To reproduce it, can you specify what task you are looking at and what adapters (bottleneck, lora, ...) you are using. If you are using adapters from the hub, please share a link.
Are you setting the model to eval mode with model.eval()
to ensure there is no non-deterministic behaviour in the forward pass?