Simple question about llama adapter v1 transformer forward function
yestaehyung opened this issue · comments
Taehyung Noh commented
Hello, first of all, thank you for the excellent work.
From my understanding of the paper, in Llama Adapter v1, the adaption prompt is inserted into the topmost L layers of the transformer.
However, in the code below, if self.adapter_layer is 30, doesn't it insert the adapter from the 3rd to the 32nd layer of the transformer?
Could you please explain why -1 * self.adapter_layer was used here?
for layer in self.layers[: -1 * self.adapter_layer]:
h = layer(h, start_pos, freqs_cis, mask)
for layer in self.layers[-1 * self.adapter_layer :]:
h = layer(h, start_pos, freqs_cis, mask, adapter[adapter_index].half())
adapter_index = adapter_index + 1
I really appreciate any help you can provide.