Charset seem to be ISO-8859 even when system is set to UTF-8

Question

Charset seem to be ISO-8859 even when system is set to UTF-8

kuttkutt opened this issue 4 years ago · comments

Hello,
I set the Raspberry to UTF-8 and the character encoding of the terminal (screenshot is Terminus, but it's the same, when I try Putty) to UTF-8 aswell.

The borders seem to use the wrong encoding. MC for example works fine without any crumbled screen.

If I set the character encoding to ISO-8859 on putty/terminus, wavemon looks pretty again, but mc is crumbled. Actually this would be right, because the Raspberry rUns still on UTF-8

So my guess is that wavemon does not regotnize the charset of the system and falls back to ISO-8859...
Is there a way to tell wavemon to use UTF-8 e.g. in the config file?

I forgot: These are my locales settings:
root@raspberrypi:/home/pi# locale LANG=en_GB.UTF-8
LANGUAGE=
LC_CTYPE="en_GB.UTF-8"
LC_NUMERIC="en_GB.UTF-8"
LC_TIME="en_GB.UTF-8"
LC_COLLATE="en_GB.UTF-8"
LC_MONETARY="en_GB.UTF-8"
LC_MESSAGES="en_GB.UTF-8"
LC_PAPER="en_GB.UTF-8"
LC_NAME="en_GB.UTF-8"
LC_ADDRESS="en_GB.UTF-8"
LC_TELEPHONE="en_GB.UTF-8"
LC_MEASUREMENT="en_GB.UTF-8"
LC_IDENTIFICATION="en_GB.UTF-8"
LC_ALL=

The problem persist aswell, when I set LC_ALL and LANGUAGE

Gerrit Renker · Answer 1 · Mon Jun 08 2020 02:33:20 GMT+0800 (China Standard Time)

I am not sure that this an encoding issue. My terminal uses en_CA.UTF-8 without problems.
Can you check what the value of the $TERM variable is? Mine is xterm-256color.

kuttkutt · Answer 2 · Mon Jun 08 2020 21:00:10 GMT+0800 (China Standard Time)

hmm same here:

pi@raspberrypi:~ $ echo $TERM
xterm-256color

I mean it's really odd. I am used to have this issue with my, when I set the wrong encoding in my terminal tool. Is there a way (command line parameter) on wavemon to check which character encoding is currently in use? Or maybe force a different encoding (may be by recompiling it)?

Gerrit Renker · Answer 3 · Tue Jun 09 2020 20:48:10 GMT+0800 (China Standard Time)

I even generated en_GB.UTF-8, unable to reproduce this on my Linux system.

You said that you use both putty and terminus (this terminus?). Both are terminal emulators, so the problem could be sitting there.

Does the problem persist when you ssh into the pi from a Linux terminal and run mc/wavemon?

kuttkutt · Answer 4 · Tue Jun 09 2020 22:24:58 GMT+0800 (China Standard Time)

Well, to ssh in from an other linux machine is a bit difficult ;)

Just let me briefly describe my situation :)
I'm on a sailboat anchoring at the island Dominica. It's a few meters to my WiFi Hotspot (a Bar) and so I build a small WiFi Repeater out of a HighGain Netgear A6210 USB WLAN Stick and build a small WiFi onboard for alle the Tablets and Mobile Phones.

I use Putty on my Laptop and Terminus on my Android Tablet. I guess the "Terminus" I use is a port of the one you linked in from GitHub.

I got wavemon via the apt reposity. Could it be that thats the problem? Maybe I should try to compile it from source ...

Gerrit Renker · Answer 5 · Wed Jun 10 2020 02:53:16 GMT+0800 (China Standard Time)

wavemon already seems to be relatively new, though you did not include a version. It could be ncurses, which is where mc comes in handy.
htop is another strawman to use, would be interesting for comparison.

Gerrit Renker · Answer 6 · Wed Jun 10 2020 07:36:07 GMT+0800 (China Standard Time)

I was able to reproduce the issue with Android Termius. A fix is to set the terminal type (Settings -> Terminal Type) to vt100.
This is still not ideal (black/white only), but the box lines are rendered better.

kuttkutt · Answer 7 · Fri Jun 12 2020 20:55:23 GMT+0800 (China Standard Time)

Thanks for the info. Yea, with vt100 it looks alright.

Actually I don't care much about the colors - as long the output is readable, it's ok with me.

PS: I compiled it from source, but it didn't made any change

Disassembler0 · Answer 8 · Wed Sep 23 2020 02:39:05 GMT+0800 (China Standard Time)

Encountered the same, also on RPi. In PuTTY the fix is to check "Enable VT100 line drawing even in UTF-8 mode" checkbox under Window > Translation

I also confirm that other apps (mc, htop etc.) work without any issues even without this setting. PuTTY usually sets $TERM to plain xterm.

Gerrit Renker · Answer 9 · Wed Sep 23 2020 21:51:17 GMT+0800 (China Standard Time)

I will look into this eventually when I get time.

Busy with other things, so this sinks to the bottom of my priority list.

kuttkutt · Answer 10 · Sat Sep 26 2020 06:38:18 GMT+0800 (China Standard Time)

No prob, it's not such a big hassle. If you find time to look / fix that, then it will be fixed ;)

harris · Answer 11 · Sat Oct 03 2020 15:05:28 GMT+0800 (China Standard Time)

I'm also facing this issue on an RPi, connecting from Bash on Windows Subsystem for Linux (Windows 10). Level histogram "curve" font does not look right + in the scan window some SSIDs which contain unicode umlaut characters are not showing up correctly. They show in literal form, like "\xc3\xa4".

harris · Answer 12 · Sun Oct 04 2020 15:07:41 GMT+0800 (China Standard Time)

I'm also facing this issue on an RPi, connecting from Bash on Windows Subsystem for Linux (Windows 10). Level histogram "curve" font does not look right + in the scan window some SSIDs which contain unicode umlaut characters are not showing up correctly. They show in literal form, like "\xc3\xa4".

Just FYI: the same thing with local terminal directly, on two different RPis. Bash itself prints out unicode characters just fine (at least the ones I've tested).

Gerrit Renker · Answer 13 · Mon Oct 12 2020 06:28:47 GMT+0800 (China Standard Time)

I just added ncursesw support in master and I believe the problem is fixed:

Clone from current master,
make sure that libncursesw{6,5} and libncursesw{6,5}-dev are installed (6 is the more recent version).
Rebuild (./config/bootstrap; ./configure && make && sudo make install).
Export the environment variable NCURSES_NO_UTF8_ACS:

> export NCURSES_NO_UTF8_ACS=1

Start wavemon.

Try if (4) is needed by skipping the step. If it is needed, you can add this to your ~/.bashrc:

> echo -e "\n# ncurses(3) initialization for vt100 line drawing\nexport NCURSES_NO_UTF8_ACS=1" >> ~/.bashrc

I confirmed this to work by logging in from a chromebook and from Android termius.

Disassembler0 · Answer 14 · Mon Oct 12 2020 13:55:38 GMT+0800 (China Standard Time)

Works for me on Raspbian 10 + PuTTY, but NCURSES_NO_UTF8_ACS=1 is needed. Without it, it looks still the same as on OP's screenshot.

As a side note, on Raspbian 10 the package is called just libncurses-dev. The packages libncurses5-dev and libncursesw5-dev exist, but they are just transitional packages for libncurses-dev in version 6.1 anyway.

Gerrit Renker · Answer 15 · Tue Oct 13 2020 10:21:24 GMT+0800 (China Standard Time)

Thanks for testing. Some other tricks (like changing the terminal settings) or putty-specific settings might help in corner cases.

I will add this to the documentation and wrap up the fix soon.

Gerrit Renker · Answer 16 · Mon Oct 19 2020 06:23:06 GMT+0800 (China Standard Time)

I added another improvement - when linking against libncursesw, the NCURSES_NO_UTF8_ACS is not needed, it will use the wide-character variants for drawing the box outlines. Tested to work on Android with putty/termius and on chromebook with the ssh extension.

Disassembler0 · Answer 17 · Mon Oct 19 2020 13:28:48 GMT+0800 (China Standard Time)

Yesss! I confirm that on Raspbian 10 + PuTTY, NCURSES_NO_UTF8_ACS=1 is no longer needed either.

Gerrit Renker · Answer 18 · Mon Oct 19 2020 19:37:16 GMT+0800 (China Standard Time)

Thank you for testing. I will update the documentation to close the issue.