Fails with UnicodeDecodeError on Japanese text in path
cyaoeu opened this issue · comments
pathname = f.readline() fails with UnicodeDecodeError if the text constains Japanese (or any other non-ascii character). I fixed it locally with with open(f"{assetEntryDir}/pathname", encoding='utf8') as f: but not sure if it's the best solution.
Huh, can you provide your Python version, the full error, and maybe an example of one of the paths it was trying to extract or just the unitypackage it failed on?
I added a test and was not able to reproduct. pathname with Assets/テスト.txt and asset with テスト, but with katakana!. Seems to work in Python 3.6-3.8: #10
Oh, my guess is that your system encoding is just set differently from mine/GitHub Actions CI. The output of the following is probably different for your system.
>>> import locale
>>> locale.getpreferredencoding()
'UTF-8'I can't imagine Unity would be expecting to use an encoding other than UTF8 for these, I will go ahead and make that the default
Okay, it should now default to utf-8 on systems who are configured differently, and there's now an encoding= kwarg just in case
If you can, pull down the PR and test and let me know if it works. I'll merge in a week regardless
Merged, should be fixed in main branch. Also want to track #11 if you're using the built .exe version (haven't pushed a new build yet, changed OSs)