mosaicml / llm-foundry

LLM training code for Databricks foundation models

https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm

Quick question on finetuning process

palash04 opened this issue a year ago · comments

Palash Kamble commented a year ago

Does the finetuning process in this repo trains only response text and not on prompt text, right?

Daniel King commented a year ago

That is correct!

llm-foundry/llmfoundry/data/finetuning/collator.py

Line 198 in c6cfb42

labels = ([_HF_IGNORE_INDEX] * n_context) + target

Palash Kamble commented a year ago

Thanks