Colors for pdksh removed

Question

Colors for pdksh removed

rene-s opened this issue 5 years ago · comments

Hello agkozak,

with 550fc20 colors for pdksh were removed.

I do not challenge this decision. Care to explain why colors were are unreliable in pdksh though? I would like to have a look into it but I do not know what the actual issue was. And I want to reintroduce some colors into the drab OpenBSD world :)

Thanks!
edit: typo
edit: thanks for polyglot btw

Alexandros Kozák · Answer 1 · Wed Feb 06 2019 01:32:14 GMT+0800 (China Standard Time)

Thanks so much for your interest in seeing that feature revived!

I did not intend to remove it permanently (unless there is no workaround). I was keen, though, not to keep it in the master branch while I could see a problem whose solution was not obvious to me.

If I log into a FreeBSD server while using the Blink Shell terminal on iOS and run pdksh, the terminal does not position the cursor correctly. It starts out several places to the right of where it should be and continues to stay far ahead of where I'm typing. For some reason the terminal must be including the color escape sequences in its count when it positions the cursor.

It's likely that with enough experimentation we can get pdksh colors working again. It's even possible that there's some bug in Blink Shell, although that app is generally quite reliable. Let me know what you think.

Alexandros Kozák · Answer 2 · Wed Feb 06 2019 04:30:55 GMT+0800 (China Standard Time)

I may have a fix. Give me a little while to test it out.

Rene Schmidt · Answer 3 · Wed Feb 06 2019 04:57:34 GMT+0800 (China Standard Time)

Ah I see. I do not have an iOS device to test the specific case you mention though :/

I don't know if you have seen it, I am experimenting with reintroducing the colors for pdksh:

master...rene-s:try_to_reintroduce_colors_for_pdksh

It obviously does not address the "pdksh on FreeBSD via Blink Shell on iOS" issue though. It is working for me though -- so far. But if you have a fix that's even better I guess :)

Alexandros Kozák · Answer 4 · Wed Feb 06 2019 05:19:11 GMT+0800 (China Standard Time)

I have confirmed that a related problem occurs with other terminals, namely that pdksh mistakes the escape codes for displayable characters and puts the < in the wrong place when you get near the end of a line and the line editor should start scrolling. This is with FreeBSD.

On DragonFlyBSD, that also happens, plus there is another problem where the escape characters are actually displayed as glyphs, but that may have something with the default cons25 terminal.

No trouble at all that I can see on OpenBSD or NetBSD running in VirtualBox, though. If I can confirm that everything works over SSH, I might just enable color selectively for those platforms. But it would be nice to know what the underlying difference is, anyway.

The basic issue is discussed in the pdksh man page:

Note that since the command line editors try to figure out how long the prompt is (so they know how far it is to edge of the screen), escape codes in the prompt tend to mess things up. You can tell the shell not to count certain sequences (such as escape codes) by prefixing your prompt with a non-printing character (such as control-A) followed by a carriage return and then delimiting the escape codes with this non-printing character. If you don't have any non-printing characters, you're out of luck... BTW, don't blame me for this hack; it's in the original ksh.

Thanks so much for your experimentation, and let me know if you find anything that will work for FreeBSD or DragonFlyBSD. And on OpenBSD, I'd suggest running git checkout da93302 (the last commit with pdksh color codes) and trying anything that might complicate things: log in locally, over SSH with different terminals, etc. etc. Let me know everything you try that works, and be sure to test for the problem of the < symbol appearing in the right place before the line editor wraps.

Rene Schmidt · Answer 5 · Wed Feb 06 2019 06:41:20 GMT+0800 (China Standard Time)

I have dabbled around a bit after reading your comment.

Have a look at this screenshot: https://files.chew.hostsys.de/f/1432127d50244dd290ef/

The terminal on top shows Polyglot on zsh. I have entered an umlaut (2 bytes character AFAIK) -- no issue there.

The terminal on the bottom shows my "hacked" Polyglot on pdksh (on OpenBSD@Apple PowerMac G4 Cube BTW). There is an issue when I hit the border with my "ä" umlaut. There are some umlauts appended to "<", with a trailing garbled character.

At first I assumed that this may be caused by my "hack" (which is not a hack, I have just partially reverted the removal of the colors for pdksh for non-root users) so I checked out your master and tried again. Unfortunately, the issue remains. It even remains when Polyglot has not been sourced at all, so this is not Polyglot's fault.

My assumption is that pdksh over ssh seems to have an issue not only with escape codes for colors but a general issue with multibyte characters.

I could not replicate this error locally though. It seems to be a "pdksh over ssh and multibyte characters" issue.

In my opinion the colors for pdksh could be reintroduced again because

pdksh over ssh seems to have a general issue with multibyte characters and removing the color escape codes for pdksh in Polyglot does not change that.
OpenBSD without color prompt looks drab -- OK this is just my opinion :)

Of course it's up to you to draw a conclusion. I'm tapping out for today, cu.

Alexandros Kozák · Answer 6 · Wed Feb 06 2019 07:37:17 GMT+0800 (China Standard Time)

@rene-s Thanks. Your assessment is a good one.

Two questions:

When you're using colors with Polyglot and pdksh and you type lots of ASCII characters (i.e. no umlauts), does it wrap correctly and does the < character appear in the right place?
Could you tell me your terminal type and pdksh version?

echo $TERM $KSH_VERSION

Thanks!

Rene Schmidt · Answer 7 · Wed Feb 06 2019 18:50:32 GMT+0800 (China Standard Time)

Hi there:

yes and yes -- it even happens locally contrary to my perception yesterday, so it does not even seem to be related to SSH and it's also not related to Polyglot. It's a bug in pdksh in my opinion.
xterm-256color @(#)PD KSH v5.2.14 99/07/13.2 -- it's the one that comes with Open BSD 6.4.

Alexandros Kozák · Answer 8 · Thu Feb 07 2019 04:03:53 GMT+0800 (China Standard Time)

OK. I've got something for you to try. Go to the Polyglot directory and run

git pull
git checkout pdksh

The pdksh branch now has an experimental pdksh-specific prompt again. See if it helps and tell me what you find. There's still more work to do, of course -- I have to make sure that the new code doesn't cause problems with the other shells, etc. etc.

The background is this: pdksh requires a non-printable ASCII character to delimit the escape sequences in the prompt so that they don't get counted. The documentation suggests \001 (Ctrl-A), but that doesn't really seem to work anymore. Of the 32 technically non-printable ASCII characters, only four don't immediate cause problems with the terminals I use: \016 (Shift In), \017 (Shift Out), \021 (Ctrl-Q), \022 (Ctrl-S). Shift In and Shift Out were used long ago to switch between character sets, so I'd like to avoid those. Ctrl-Q and Ctrl-S are still used in flow control, but I've tested Ctrl-Q (\021), and outputting it in the terminals I use doesn't actually seem to affect flow control (sending it does). So I'm using Ctrl-Q; let's test it in lots of terminals. If I feel comfortable with it, I'll finish rewriting that part of the prompt and we'll be done.

Alexandros Kozák · Answer 9 · Thu Feb 07 2019 23:45:45 GMT+0800 (China Standard Time)

I feel increasingly confident with how I've got the prompt working in pdksh in OpenBSD, NetBSD, and FreeBSD. The outlier is DragonFly BSD, whose default console always displays the delimiter character.

Rene Schmidt · Answer 10 · Thu Feb 07 2019 23:51:13 GMT+0800 (China Standard Time)

Hello Alexandros ok great, I will have a look at it when I am home. Regards

Alexandros Kozák · Answer 11 · Fri Feb 08 2019 02:46:22 GMT+0800 (China Standard Time)

OK. Now that it's more stable, I've moved development over to the develop branch. I think it should be ready to release soon.

Rene Schmidt · Answer 12 · Fri Feb 08 2019 03:39:46 GMT+0800 (China Standard Time)

I'm not sure if I'm doing it wrong...

https://files.chew.hostsys.de/f/a86218a849db43c995a0/

Die delimiter character still looks not quite right in my VM. The terminal is st but the result is the same in xterm.

Alexandros Kozák · Answer 13 · Fri Feb 08 2019 04:43:09 GMT+0800 (China Standard Time)

I think I know what's going on. Try typing set -o. It will show that emacs mode is on. If you set -o vi, vi mode will be on -- the scrolling works differently. What you're showing is correct behavior for emacs mode.

I'm not sure the umlauts will ever work correctly. This is a very archaic shell.

Rene Schmidt · Answer 14 · Fri Feb 08 2019 05:05:02 GMT+0800 (China Standard Time)

Aha ok...

https://files.chew.hostsys.de/f/3f1943f03a4e41da833b/

Looks good enough for me if you ask me. :) It's not perfect but when I am using a BSD I am grateful for everything that works OK without too much effort.

Rene Schmidt · Answer 15 · Fri Feb 08 2019 05:07:01 GMT+0800 (China Standard Time)

Thanks for the update :) Let me know when I can test something.

Alexandros Kozák · Answer 16 · Fri Feb 08 2019 07:29:32 GMT+0800 (China Standard Time)

All right! I've released v1.7.0 with color support for pdksh. You may switch back to using the master branch. Thanks for your encouragement and your help, @rene-s!