sd replaces non-existent newline at the beginning of STDIN in powershell

[desk7]

• sd 0.7.5
• Windows 10
• Powershell 7.3.4

In powershell the command

"Hello
everyone!" |  sd  '\n' '\\n'

wrongly gives the following result:

\nllo\neveryone!

Please note that the problem does not happen when processing a file in powershell.
I don't know if the problem depends on powershell or sd.

[peterlits zo]

This will help:

"Hello
everyone!" | sd '\r\n|\n' '\\n'

If you just use '\n', the token '\r' will not be replaced. It only happens in Windows. I think it maybe is a bug...

[desk7]

@PeterlitsZo Thanks for the reply. Now I can see the source of the problem. When \r is left alone, without \n, the carriage return (`r) character moves the output cursor to the beginning of the current line and continues writing. Any characters on the current line are overwritten. However, there is something unexplained. I use hexyl to see the hex of the strings.
So, let's use this simple example

"Hello
everyone!" | sd '\n' 'A'

The string which is actually passed to sd is ¹

Hello`nHello`r`n

48 65 6c 6c 6f 0a 48 65 6c 6c 6f 0d 0a
and the output string should be

HelloAHello!`rA

When the string is printed, the last A should be actually moved back to the beginning of the line because of the carriage return.
However, the result is bizzare. The printed string is:

AelloAHello

but, looking at the hex:
48 65 6c 6c 6f 41 48 65 6c 6c 6f 0d 0a 41 0d 0a, that is

HelloAHello`r`nA`r`n

That result is the same if we do the command

"Hello
Hello" | sd '\n' 'A'> example.txt

To conclude, the solution you proposed works very well, but I wonder why, asking sd to replace only \n, I actually get `r`nA instead of `rA (obviously, I don't consider the appended `r`n probably produced by the shell writing command).

Footnotes

1. Why does powershell inconsistently use \n between Hello and everyone while using \r\n at the end of the line? I can't understand 😅 ↩

[peterlits zo]

Thanks, it is interesting... I will come back...

[CosmicHorror]

I can reproduce the original issue (although I think we're all in agreement that it's expected behavior at this point): however, I can't reproduce the issue of \r\n remaining in the output string from sd

❯ echo -n "Hello\nHello\r\n" | sd '\n' 'A' | hexyl
┌────────┬─────────────────────────┬─────────────────────────┬────────┬────────┐
│00000000│ 48 65 6c 6c 6f 41 48 65 ┊ 6c 6c 6f 0d 41          │HelloAHe┊llo_A   │
└────────┴─────────────────────────┴─────────────────────────┴────────┴────────┘

The above also matches your observed output of

AelloAHello

since \r\n remaining in the output would display the initial A on the next line instead of it being on the same line like above

I don't have a windows machine setup that I can use for easy testing unfortunately, and I'm a bit pressed for time lately

[peterlits zo]

but, looking at the hex:
48 65 6c 6c 6f 41 48 65 6c 6c 6f 0d 0a 41 0d 0a, that is

HelloAHello`r`nA`r`n

I build a rust application to reproduce this error:

use std::io::{stdin, Read};

fn main() {
    let mut s = Vec::new();
    stdin().read_to_end(&mut s).unwrap();
    for c in &s {
        print!("{:?} ", (*c as char));
    }
}

And this is the command run in the PowerShell:

> "Hello        
>> Hello" | sd '\n' 'A' | ./tmp.exe
'H' 'e' 'l' 'l' 'o' 'A' 'H' 'e' 'l' 'l' 'o' '\r' '\n' 'A' '\r' '\n' 
> "Hello
>> Hello" | ./tmp.exe
'H' 'e' 'l' 'l' 'o' '\n' 'H' 'e' 'l' 'l' 'o' '\r' '\n' 

The reason for this question is now clear:

• sd give us 'H' 'e' 'l' 'l' 'o' 'A' 'H' 'e' 'l' 'l' 'o' '\r' 'A'
• PowerShell turns the \r into \r\n.
• PowerShell also appends a \r\n at the end of the text if it doesn't find a \r\n at the end.

I didn't find a document about this behavior. But I think it is right after some tests.

[CosmicHorror]

Seems good. Thanks for digging into this more @PeterlitsZo!

[peterlits zo]

But I think we can add a flag or something to use \n to catch \n and \r\n. Do you think it is a good idea? That can make us use the same command in both Windows and UNIX-like OS.

[CosmicHorror]

Hmmm, I'll have to think about that more for now