[Bug] When loading a model by using transformers and using stream chat, it seems no whitespace character in English response.

[zhulinJulia24]

Describe the bug

When loading a model by using transformers, it seems to be an issue with English answers not having spaces

Environment

It canbe reproduced on transformers 4.34.0 、 4.37.1 and 4.38.1 version

When prompt is "what's your name"
the answer is
MynameisInternLM(书生·浦语).I'maconversationallanguagemodeldevelopedbyShanghaiAILaboratory.It'snicetomeetyou!
with out whitespace

Other information

script:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

prompts = ['你好', "what's your name"]


tokenizer = AutoTokenizer.from_pretrained("internlm/internlm2-chat-7b", trust_remote_code=True)
# Set `torch_dtype=torch.float16` to load model in float16, otherwise it will be loaded as float32 and cause OOM Error.
model = AutoModelForCausalLM.from_pretrained("internlm/internlm2-chat-7b", torch_dtype=torch.float16, trust_remote_code=True).cuda()
model = model.eval()

history = []
for prompt in prompts:
    length = 0
    for response, history in model.stream_chat(tokenizer, prompt,history=[]):
        print(response[length:], flush=True, end='')
        length = len(response)
    print(response)

[zhulinJulia24]

fixed