UnicodeDecodeError: 'ascii' codec can't decode byte... when using dss.run_command()
pedrotorres08 opened this issue · comments
Versions
- Python version: 3.8.5
- Python architecture: x64
- Operating system and version: Windows 10
- OpenDSSDirect.py version number: 0.6.0
- IDE: Spyder
Bug
Hi... maybe this is a noob question but I cannot get the dss.run_command() to work.
I am following the example from the notebook, and I receive the following error:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc7 in position 78: ordinal not in range(128)
The error traces to this line:
File "C:\Users\pferr\anaconda3\lib\site-packages\dss_cffi_api_util.py", line 150, in get_string
return self.ffi.string(b).decode(self.codec)
Any thoughts on that?
The problem is that we are using ascii here:
@PMeira I think we can use utf8 always, no? Since ascii is a strict subset. Was there an issue that resulted in adding ascii as the codec?
I think we can use utf8 always, no? Since ascii is a strict subset. Was there an issue that resulted in adding ascii as the codec?
@kdheepak It's complicated, the root issue is not in Python. Depending on the characters and Windows codepage, you get different results, hence the explicit ascii there (on Windows, for most western codepages, it might be fine). Depending on the string, you can try using UTF8 on the Python side but I don't guarantee it would work in every environment; something like this should do it:
import opendssdirect
opendssdirect._utils.codec = 'utf8'In Free Pascal, UTF8 strings not are the default ones and cross-platform support is a bit weird. From what I read, we'll have to handle encoding/decoding manually in a lot of places in the Pascal code.
I always recommend using only ASCII on both file contents and paths due to that. I also hit issues with non-ANSI chars in the official OpenDSS, for example.
See also dss-extensions/dss_capi#6
For me, this is not a high priority right now but I'd prefer if we could handle it in the next few months. Next week I should be able to continue merging some other changes.
I always recommend using only ASCII on both file contents and paths due to that. I also hit issues with non-ANSI chars in the official OpenDSS, for example.
Oh, and if this is the case but you still get an error, please try to provide some more details so we can reproduce it.
I should have realized that in free pascal utf8 wasn't the default. I guess I've been spoiled by modern languages :)
On Linux, I believe filesystems store filenames as just bytes and don't have to be valid in an encoding, so technically someone would create a filename with open(b'\xf0\x28\x8c\x28.dss', 'w').write('New Circuit.circuitname') and on Linux I think that would be a valid file. On a mac however, this filename is illegal though since on a mac filenames are utf8 encoded. I'm not sure how this works exactly on Windows. I guess using strict ascii is going to be restrictive in some cases. Maybe we should just document this somewhere?
I tested using the filename 3Φmodel.dss just now. Using this monkey patching techniques works for now (at least on a mac).
import opendssdirect
opendssdirect._utils.codec = 'utf8'I believe that'll resolve @pedrotorres08's issue as well.
Dear all, thanks for the fast reply. Turns out that the file path contained a folder with cedilla (ç) and o with tilde chars (õ). When I renamed the folder it worked well just like in the notebook.