sh.which doesn't return an object and doesn't throw exceptions

[StPanning]

Hi,
I experience this unexpected behavior:
Is this a bug, or am I doing something wrong?

Python Version:

$ python --version
Python 3.9.5

sh Version: 1.14.2

Code snippet:

 try:
      # i expect behavior as described here:
      # https://amoffat.github.io/sh/sections/exit_codes.html
      output = sh.which('xxxxx')
      # output returns a string or None , not an object
      # print(output.exit_code) fails because neither 'str' nor 'None'  has the attribute 'exit_code' 
 except ErrorReturnCode:
      # this exception is not thrown 
      logger.error("program not found")

[Andrew]

sh.which is not actually a binary on your system, it is a helper function, in the same way that the real which command is a shell helper command, and not an actual binary on your system. So the normal process execution rules do not apply to sh.which.

[StPanning]

it is a binary on my system.

 which which
/usr/bin/which

the helper function does the same thing I tried to accomplish with the system binary, hence the confusion.
I was not aware of the helper function.

Thanks for the fast answer.

[Andrew]

Oh interesting, you're right, it appears to be a binary as well. Ok, this is a rare case of sh overriding normal behavior then. Thanks for pointing it out, I will make a note in the docs.

[Andrew]

It looks like I might be misremembering which ever being a shell builtin. I'm having trouble finding documentation about it. It could be that this behavior is not correct and should be changed, if which is only ever a regular binary command.

[StPanning]

I had to to check this myself because I wasn't sure either, but at least bash -c help doesn't list which as a builtin command

[Erik Cederstrand]

@amoffat I guess an easy fix would be to rename the internal which() to _which() and let from sh import which go through the normal path for resolving binaries? Is there a real end-user need for the helper or could we just deprecate it for 2.0?

Alternatively, have sh.which support the full API of the which binary, but that varies between operating systems.

[Andrew]

@ecederstrand I like your first suggestion 👍 . I don't think there is a real end-user need for which() to be something custom like it is now.

@StPanning the work around currently, if you need to run the real which on your system, should be:

real_which = sh.Command("which")
real_which(whatever)

[StPanning]

@amoffat I am using already the helper function.
I was not aware of it before.

[Andrew]

So it turns out in zsh, which is a builtin, and I wasn't losing my mind for remembering that 😄 :

root@f413aec5dfbb:/# echo $SHELL
/bin/bash
root@f413aec5dfbb:/# which which
/usr/bin/which
root@f413aec5dfbb:/# zsh
f413aec5dfbb# which which
which: shell built-in command

@ecederstrand I think the MR might need to account for both cases. I'll add comments to it.

[mentalisttraceur]

which is an absolute mess of a command-line tool.

Some shells have it built-in, but it is often provided as an external program too (but the external one can't know about commands that are in your shell like builtins, functions, and aliases), and that external program might be a csh script (that's right, CSH script - that's what what it was originally, and this still has some historical significance on some *nix systems).

You can learn a lot more about this if you dig around enough on StackOverflow, searching for stuff like "command vs type vs which"

If you're using a normal shell script (not a weird outlier like csh or tcsh or fish - a Bourne-like shell, which includes bash and zsh) you shouldn't ever be using which, unless you want to have habits that are wrong in some situations (like if you find yourself on a system where which is still a csh script which missed your shell's aliases+functions+builtins but does find commands defined in some csh config somewhere) - you should be using command -v.

If you're using which from Python with sh.... well first of all why? What are you actually using which for, and is imitating what you'd do in the shell into Python the best way to achieve that? Depends. If you're just trying to do a PATH search to learn where something is, ideally there would be a library that just solves the problem of searching all directories in a PATH-like variable for a given named file - because all sorts of programs use PATH-style variables for conceptually similar things (in a better world we would just have per-process inheritable directory overlays provided by the OS over the file system, but alas we can't have such nice things yet, so countless programs reuse the pattern of path variables).

Anyway, I think this all this informs whether or not sh should provide its own which:

1. How valuable is it for sh to give users consistently available means to look shorthand for looking up where commands are found? (Seems like a nice-to-have I guess.)

2. Is the best interface for that in Python through a library like sh an imitation of the which command? (I think no - I think a more Pythonic and "sh-ic" way would be an interface that lets you get a str or pathlib.Path object from an sh.Command object - something like sh.get_path_of(sh.foo) or sh.foo._path, or maybe you can even somehow make it so that pathlib.Path(sh.foo) Just Works.)

[mentalisttraceur]

Check out os.PathLike. I would give sh.Command an __fspath__ method which does the path lookup for that command, and then let sh.which be treated like any other command instead of giving it special treatment.

(I would make commands with bound arguments and subcommands give the path of their command - because it can be useful to ask "what is the location of the program that implements this command?" in those cases - kinda like how it can be useful in Python to know what module a function was defined in, even if that function is nested in a class definition or whatever.)

[Erik Cederstrand]

As I see it, sh just needs to provide functionality to resolve command paths via PATH. That's not hard and shouldn't require access to an external implementation. While sh aims to replace many shell scripts, I don't think we should venture into users' dotfiles or try to access aliases or built-ins.

I like the idea of putting the logic in __fspath__. This also ties into the discussion going on in #602. The only issue is that sh still supports Python 2.x and Python 3.5 which does not have os.PathLike.

[mentalisttraceur]

The only issue is that sh still supports Python 2.x which does not have os.PathLike.

Well, the nice thing about magic methods like __fspath__ is that people who want to write code for older Pythons can just use that method directly.

So if someone wanted to write code using sh which is compatible with Python 2, I think it's just a few lines of code to backport/polyfill the os.PathLike ABC, and so on.

So it turns into a nice progressive enhancement / graceful degradation thing.