wired output from "cocon_output"

[Yufei Wang]

Hi,

Thanks for your great work. I tried to run your generation script, but I got very disfluent "cocon_output".

For example, Given original_input_text "<|endoftext|>Once upon a time" and context_text "is perfect",

cocon_output is "<|endoftext|>Once upon a time each compuls gone 20 J-t and like to been and sa p I less millions the three is ( other remaining more we party in few V the only other end one inf only really more inf have S t and here super so Jp Lor now"

while the prependgpt2_ar_gen "is perfect<|endoftext|>Once upon a time I worked with a bike hacker who used specialised software in simple ways to automate various processes and monitor game flow. A few months ago, at the start of the second-year cycle (which sadly involved a lot of fake (and now living)"

I think the prependgpt2_ar_gen output is much better than the cocon output. I am wondering if this is correct output. Can you check to see if this is the real output from your cocon model or there is something wrong with the script? I basically follow everything in this repo.

Thanks

[alvinchangw]

Hi,

Thanks for your interest in COCON and for pointing this out.

This turns out to be the param name formatting bug during the COCON model loading. During the final open-source preparation, the code was refactored but this was not reflected in the saved weights.

This is now fixed with the fix_state_dict_naming function in the traininfer_cocon.py.

Cheers