format_chat_log formats incorrectly
Nintorac opened this issue · comments
{role}:{role} is used as the fstring, rather than {role}:{content}
What do you think about removing this and delegating that to the model? I guess would have to implement another gRPC endpoint? I'm a bit clueless there !
Hey,
{role}:{role}is used as the fstring, rather than{role}:{content}
Thanks, will fix that and upload a new version for this one and #2 .
What do you think about removing this and delegating that to the model? I guess would have to implement another gRPC endpoint? I'm a bit clueless there !
I've been considering the pros and cons of either doing this, or rather having a .preprocess_input method on the model side, before feeding it to the LM. Passing a formatted string looks more straightforward, and avoid implementing a new interface. String manipulation is easy in most languages and the model will ultimately take a (tokenized) string as input, so even if a dict is cleaner, information has to be lost at some point anyway.
format_chat_log overall shall be more robust: what happens if content="user:" for instance? I guess the assumption there was that the end user isn't malicious and trying to break things. Perhaps we should have a "more unique" format here.
I think we can extend this discussion to the add_instructions function: it works as expected on the Alpaca example, but lacks flexibility.
Happy to read your suggestions :)
In the long run I can see architectures that integrate multiple messages in ways other than just concatenation. i.e control messages may want to be inserted via a separate channel as a security measure, or to incorporate summary vectors of previous messages to artificially increase context. (I even wonder if OpenAI is already doing this as I've observed ChatGPT recalling parts of conversations that I thought would be well over the models context).
Apart from that I don't even imagine that there will be a standard conversation formatting so format_chat_log or add_instructions will have to be quite flexible and at that point why not just hand off to the consumer so they can get as complex as they like.
And just from a pedantic standpoint it makes sense to have a gRPC service for each endpoint :P
Very good points, especially the last one, I think I'm sold and will add something for chat. Current solution was a relatively lazy one that seemed to work decently for a PoC, but moving towards this seems to be the better direction for the next iteration.😃
Both format_chat_log and add_instructions will still be available as imports on model side if needed, so it should remain convenient to use with "concatenation based" models in Python (as I would like to keep new model integration rather straightforward).