UTF-8 characters are not properly displayed due to key mapping in the screen.py

[Beniamin Kalinowski]

Describe the bug
While using TextBox I cannot insert utf-8 characters 'ą' 'ć' while I can do it for other ones 'ż', 'ź' etc.

To Reproduce

• Create a TextBox
• Press Alt+a to get a 'ą'
• Press Alt+c to get a 'ć'

Nothing is displayed.

Expected behavior
PL utf-8 characters should be displayed.

Screenshots / Logs

PRESSING Alt+a
DEBUG:asciimatics.screen:Processing key: c4
DEBUG:asciimatics.screen:Byte stream: 1 bytes left
DEBUG:asciimatics.screen:Processing key: 85
DEBUG:asciimatics.screen:Returning key: 105

PRESSING Alt+c
DEBUG:asciimatics.screen:Processing key: c4
DEBUG:asciimatics.screen:Byte stream: 1 bytes left
DEBUG:asciimatics.screen:Processing key: 87
DEBUG:asciimatics.screen:Returning key: 107

PRESSING Alt+z
DEBUG:asciimatics.screen:Processing key: c5
DEBUG:asciimatics.screen:Byte stream: 1 bytes left
DEBUG:asciimatics.screen:Processing key: bc
DEBUG:asciimatics.screen:Returning key: 17c

System details (please complete the following information):

• OS and version: Ubuntu 20.04.3 LTS
• Python version: Python 3.6.10
• Python distribution: Anaconda, Inc.
• Asciimatics version 1.3.0

Additional context
The problem is because the 0x105 (261) key is being mapped in the Screen._KEY_MAP to Screen.KEY_RIGHT and 0x107 key is being mapped to Screen.KEY_BACK. While during the build time we know it's the utf-8 character because in screen.py file we're decoding it to utf-8. So there should be some flag to indicate that it's not the mapping, but a normal utf-8 character that should be displayed.

[pivtom]

Hi, have the same problem as BeniaminK - some Czech utf-8 characters - ě š č - are not displayed in the TextBox (although they are displayed in Text), other Czech characters are displayed correctly in both TextBox and Text (žřý etc.). The last commit didn't solve the problem. (The Bug is identical to the above one, just with different characters). Thanks for any help!

[peterbrittain]

That's surprising given the fix. Can you get a similar trace as BeniaminK sis (by enabling debug logging on your application)?

[pivtom]

Sure - using sample/contact_list.py:
In the program itself, I add a contact, go to Notes and start writing:

I press 'á' (Czech keyboard):

DEBUG:asciimatics.screen:Processing key: c3
DEBUG:asciimatics.screen:Byte stream: 1 bytes left
DEBUG:asciimatics.screen:Processing key: a1
DEBUG:asciimatics.screen:Returning key: e1
DEBUG:asciimatics.widgets.utilities:New event: KeyboardEvent: 225
DEBUG:asciimatics.widgets.utilities:Cursor: 0,4

---> á is displayed.

I press 'ě':

DEBUG:asciimatics.screen:Processing key: c4
DEBUG:asciimatics.screen:Byte stream: 1 bytes left
DEBUG:asciimatics.screen:Processing key: 9b
DEBUG:asciimatics.screen:Returning key: 11b
DEBUG:asciimatics.widgets.utilities:New event: KeyboardEvent: -20
DEBUG:asciimatics.widgets.utilities:Cursor: 0,4

--> nothing is displayed.

I press 'š':

DEBUG:asciimatics.screen:Processing key: c5
DEBUG:asciimatics.screen:Byte stream: 1 bytes left
DEBUG:asciimatics.screen:Processing key: a1
DEBUG:asciimatics.screen:Returning key: 161
DEBUG:asciimatics.widgets.utilities:New event: KeyboardEvent: -302

--> nothing is displayed and the program reacts as if I pressed ARROW-UP (=goes up in the menu)

Thanks!
Tomas

[peterbrittain]

You are still hitting the translation issue, which should be impossible in the new code. How did you pick up the fix?

[pivtom]

Pulled it from git and also cloned it, to be sure. I am on MacOS (Big Sur), python 3.9.0, if that's relevant... Thanks.

[peterbrittain]

Shouldn't be... Can you add something to the code to verify your installation? For example, add the RuntimeError at line 2557 in screen.py (if you're using master) as shown below:

                if self._bytes_to_return:
                    # UTF-8 character - resetting _bytes_to_return
                    self._bytes_to_return = b""
                    raise RuntimeError(key)
                    # return KeyboardEvent(key)

If you're using the new code, it will now hit the exception whenever you enter one of your keys that injects unicode characters.

[pivtom]

You were right, I didn't realize I had asciimatics installed through pip in the system. My bad :-/. It works now. Sorry for the "much ado about nothing" and thanks again for your help! Much appreciated! T.

…

On Thu, Jan 20, 2022 at 6:10 PM peterbrittain ***@***.***> wrote: Shouldn't be... Can you add something to the code to verify your installation? For example, add the RuntimeError at line 2557 in screen.py (if you're using master) as shown below: if self._bytes_to_return: # UTF-8 character - resetting _bytes_to_return self._bytes_to_return = b"" raise RuntimeError(key) # return KeyboardEvent(key) If you're using the new code, it will now hit the exception whenever you enter one of your keys that injects unicode characters. — Reply to this email directly, view it on GitHub <#340 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACGNCJITGFCUXIL2KKCSJTUXA6YFANCNFSM5MBCZPCQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

[peterbrittain]

NP. Good luck with your project!