AttributeError: 'NoneType' object has no attribute 'replace'

[MrBun]

I encountered the following error while using the ChatGPT evaluation script.

Suggestion: Add a null check operation on line 48 of the src/chatgpt.py script to prevent errors caused by an empty pred.

[Haonan Li]

Hi, thank you for bringing this to our attention. Our code does not accept the model returning "None" because frequent occurrences of this due to network errors could lead to unfair evaluations. To address this, we have set a timeout_counter threshold of 30.

Could you please identify why the "NoneType" might be returned? Is it related to the value of completion.choices[0].message['content'], or does it exceed the 30-time timeout limit? (I didn't see any timeout message) Your insights will be valuable in better resolving this issue.

[MrBun]

Thank you for your work. I am not sure about the reason that caused pred to be empty, but I made some modifications to the code, and as a result, my evaluation completed successfully. I changed line 48 in the original src/chatgpt.py script to:

all_preds.append(pred.replace("\n", "") if pred is not None else "")

I observed the while loop condition in the source code and speculated that network issues might be causing the number of failures to exceed 30 attempts.

[Haonan Li]

Thank you, we've update the code.