Logits are changing in old adapters-transformer models if used by the new library

[Chiara Rodella]

Environment info

• adapters version: 0.1.1
• Platform: python
• Python version: 3.9.18
• PyTorch version (GPU?):1.10.2 yes gpu
• Using GPU in script?: yes
• Using distributed or parallel set-up in script?: no

Information

Hello ! I was trying to use an old adapters model i trained last year to do some test using the new adapters library. Unfortunately i noticed that every time i re-open the run, for the same sequence I have different results. This is not happening for models trained last week.

To reproduce

model = BertAdapterModel.from_pretrained(model_name) 
model.load_adapter(adapter_path, with_head=True)
model.load_head(adapter_path )
model.set_active_adapters(['x_adapter'])
model.active_head = 'xadapter_head'  
model.train_adapter(["xadapter"])
model.delete_head('default')
model.bert.prompt_tuning = nn.Identity()

model = model.to(self.device)
input_texts = [" ".join("".join(sample.split())) for sample in input_texts]


batch_input = ['MERLFPRRFFISQKLTRKKLQKSVGQTANDFVYISEDR']
encoded = tokenizer(batch_input, truncation=True, padding=True, max_length=512, return_tensors="pt").to(self.device)
    y_preds  =model(**encoded).logits

##run1
 [-0.0135214077308774, -0.00513207633048296]
##run2
[0.07941669225692749, 0.0340627059340477]

Expected behavior

I check hundred times and with the same code 'new recent' models are working, old one no. thanks a lot, i really need to make it work !!

[Hannah Sterz]

Hey @Ch-rode , the inconsistency between adapters trained with the old vs the new library sounds like something we should look into. To reproduce it, can you specify what task you are looking at and what adapters (bottleneck, lora, ...) you are using. If you are using adapters from the hub, please share a link.
Are you setting the model to eval mode with model.eval() to ensure there is no non-deterministic behaviour in the forward pass?