Error when trying to set up adapter training for CodeLlama

[Michael Mior]

Environment info

• adapters version: 0.1.2
• transformers version: 4.36.2
• Platform: Linux-5.4.0-173-generic-x86_64-with-glibc2.31
• Python version: 3.10.12
• Huggingface_hub version: 0.22.2
• Safetensors version: 0.4.2
• Accelerate version: 0.28.0
• Accelerate config: not found
• PyTorch version (GPU?): 2.2.0+cu121 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: yes
• Using distributed or parallel set-up in script?: no

Information

Model I am using (Bert, XLNet ...): CodeLlama

Language I am using the model on (English, Chinese ...): English

Adapter setup I am using (if any):

The problem arises when using:

• the official example scripts: (give details below)
• my own modified scripts: (give details below)

The tasks I am working on is:

• an official GLUE/SQUaD task: (give the name)
• my own task or dataset: (give details below)

To reproduce

Steps to reproduce the behavior:

1. Run the script below

import adapters
from adapters import AutoAdapterModel, AdapterArguments, setup_adapter_training
from transformers import HfArgumentParser, TrainingArguments
import torch


parser = HfArgumentParser((TrainingArguments, AdapterArguments))
training_args, adapter_args = parser.parse_args_into_dataclasses()

print(training_args, adapter_args)


state_dict = torch.load("/var/data/mmior/finetune-hf/model.pth")
model = AutoAdapterModel.from_pretrained(
        "codellama/CodeLlama-7b-Instruct-hf",
        state_dict=state_dict,
)
adapters.init(model)

model.add_classification_head("mrpc", num_labels=2)
model.set_active_adapters("mrpc")

setup_adapter_training(model, adapter_args, "seq_bn")

Expected behavior

I expect the training to be set up, but instead I get the error below.

/home/mmior/.local/share/virtualenvs/json-descriptions-cLP9Kwl8/lib/python3.10/site-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  _torch_pytree._register_pytree_node(
/home/mmior/.local/share/virtualenvs/json-descriptions-cLP9Kwl8/lib/python3.10/site-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  _torch_pytree._register_pytree_node(
/home/mmior/.local/share/virtualenvs/json-descriptions-cLP9Kwl8/lib/python3.10/site-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  _torch_pytree._register_pytree_node(
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:25<00:00, 12.63s/it]
Traceback (most recent call last):
  File "/home/mmior/apps/json-descriptions/adapter.py", line 15, in <module>
    model.add_classification_head("mrpc", num_labels=2)
  File "/home/mmior/.local/share/virtualenvs/json-descriptions-cLP9Kwl8/lib/python3.10/site-packages/adapters/heads/model_mixin.py", line 70, in wrapper
    return f(self, *args, **kwargs)
  File "/home/mmior/.local/share/virtualenvs/json-descriptions-cLP9Kwl8/lib/python3.10/site-packages/adapters/heads/model_mixin.py", line 325, in add_classification_head
    head = ClassificationHead(self, head_name, num_labels, layers, activation_function, id2label, use_pooler)
  File "/home/mmior/.local/share/virtualenvs/json-descriptions-cLP9Kwl8/lib/python3.10/site-packages/adapters/heads/base.py", line 149, in __init__
    self.build(model)
  File "/home/mmior/.local/share/virtualenvs/json-descriptions-cLP9Kwl8/lib/python3.10/site-packages/adapters/heads/base.py", line 76, in build
    dropout_prob = model_config.hidden_dropout_prob
  File "/home/mmior/.local/share/virtualenvs/json-descriptions-cLP9Kwl8/lib/python3.10/site-packages/transformers/configuration_utils.py", line 265, in __getattribute__
    return super().__getattribute__(key)
AttributeError: 'LlamaConfig' object has no attribute 'hidden_dropout_prob'

[calpt]

Thanks for bringing this up. This issue should be fixed with the merge of #685.