samrawal / llama2_chat_templater

Wrapper to easily generate the chat template for Llama2

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

When having a long conversation, it does not generate new starting <s> tokens from second user - model message pairs

chintan1995 opened this issue · comments

Using the following code:

pt = PromptTemplate(system_prompt="This is a system message")
pt.add_user_message("Hello, how are you?")
pt.add_model_reply("I am doing good! Hope you are doing well too.")
pt.add_user_message("Who is the PM of India?")
pt.add_model_reply("The Prime Minister of India is Narendra Modi.")
pt.add_user_message("Thank you!")

Currently the output looks like this,

'<s>[INST] <<SYS>>\nThis is a system message\n<</SYS>>Hello, how are you? [/INST] I am doing good! Hope you are doing well too. </s>[INST] Who is the PM of India? [/INST] The Prime Minister of India is Narendra Modi. </s>[INST] Thank you! [/INST]'

The actual prompt should be like this,

'<s> [INST] <<SYS>>\nThis is a system message\n<</SYS>> Hello, how are you? [/INST] I am doing good! Hope you are doing well too. </s> <s> [INST] Who is the PM of India? [/INST] The Prime Minister of India is Narendra Modi. </s> <s> [INST] Thank you! [/INST]'

Also, if the system prompt is none, it should still generate the starting [INST] token for the user message, which this is not generating.